Music classification method, music classification apparatus, electronic device, and storage medium

Abstract

This application provides a music classification method, a music classification device, an electronic device, and a storage medium, belonging to the field of artificial intelligence technology. The method includes: acquiring sample audio data and sample lyrics data of sample music; extracting audio features from the sample audio data to obtain sample audio features; extracting lyric features from the sample lyrics data to obtain sample lyric features; constructing positive music sample pairs and negative music sample pairs based on the sample audio features and sample lyric features; training a neural network model based on the positive and negative music sample pairs to obtain a music classification model; acquiring target data for target music; extracting features from the target data to obtain target music features; scoring the target music according to its genre based on the music classification model and target music features to obtain genre score data; and determining the genre category of the target music based on the genre score data. This application can improve the accuracy of music classification.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a music classification method, a music classification device, an electronic device, and a storage medium. Background Technology

[0002] Currently, music classification tasks often rely on extracted music features to determine the genre. However, feature extraction often results in missing features, leading to low accuracy in music classification. Therefore, improving the accuracy of music classification has become an urgent technical problem to be solved. Summary of the Invention

[0003] The main objective of this application is to provide a music classification method, a music classification device, an electronic device, and a storage medium, with the aim of improving the accuracy of music classification.

[0004] To achieve the above objectives, a first aspect of this application proposes a music classification method, the method comprising:

[0005] Obtain sample audio data and sample lyrics data of the sample music;

[0006] Audio features are extracted from the sample audio data to obtain sample audio features;

[0007] Lyrics features are extracted from the sample lyrics data to obtain sample lyrics features;

[0008] Based on the sample audio features and the sample lyrics features, construct positive music sample pairs and negative music sample pairs;

[0009] Based on the positive music sample pairs and the negative music sample pairs, a preset neural network model is trained to obtain a music classification model;

[0010] Obtain the target data for the target music;

[0011] Feature extraction is performed on the target data to obtain the target music features of the target music;

[0012] Based on the music classification model and the target music features, the target music is scored by genre to obtain genre score data, and the genre category of the target music is obtained based on the genre score data.

[0013] In some embodiments, the step of extracting audio features from the sample audio data to obtain sample audio features includes:

[0014] The sample audio data is converted to a different format to obtain sample spectrum data;

[0015] The sample spectral data is input into a preset feature extraction model, wherein the feature extraction model includes convolutional layers and pooling layers;

[0016] Based on the convolutional layer, the sample spectral data is subjected to two-dimensional convolution processing to obtain preliminary spectral features;

[0017] The initial spectral features are subjected to max pooling based on the pooling layer to obtain the sample audio features.

[0018] In some embodiments, the step of extracting lyric features from the sample lyric data to obtain sample lyric features includes:

[0019] The sample lyrics data is input into a preset text encoding model, wherein the text encoding model includes an embedding layer and a transformer encoder;

[0020] The sample lyrics data are embedded based on the embedding layer to obtain the sample lyrics embedded representation.

[0021] The sample lyrics features are obtained by performing feature encoding on the sample lyrics embedding representation based on the transformer encoder.

[0022] In some embodiments, both the positive music sample pair and the negative music sample pair include a sample audio feature and a sample lyrics feature. Constructing the positive music sample pair and the negative music sample pair based on the sample audio feature and the sample lyrics feature includes:

[0023] Based on the sample audio features and sample lyric features derived from the same sample music, the positive music sample pair is constructed;

[0024] The negative music sample pairs are constructed based on the sample audio features and sample lyric features derived from different sample music.

[0025] In some embodiments, training a preset neural network model based on the positive music sample pairs and the negative music sample pairs to obtain a music classification model includes:

[0026] The positive music sample pairs are subjected to feature dimensionality reduction to obtain positive music feature pairs, and the negative music sample pairs are subjected to feature dimensionality reduction to obtain negative music feature pairs, wherein the negative music feature pairs include a first negative feature pair and a second negative feature pair;

[0027] The positive music feature pairs are scored for feature similarity to obtain first similarity data; the first negative feature pairs are scored for feature similarity to obtain second similarity data; and the second negative feature pairs are scored for feature similarity to obtain third similarity data.

[0028] Loss calculation is performed based on the first similar data and the second similar data to obtain the first loss data;

[0029] Loss calculation is performed based on the first similar data and the third similar data to obtain the second loss data;

[0030] The parameters of the neural network model are optimized based on the first loss data and the second loss data to obtain the music classification model.

[0031] In some embodiments, the target data includes target audio data and target lyrics data of the target music, and the step of extracting features from the target data to obtain target music features of the target music includes:

[0032] Audio features are extracted from the target audio data to obtain the target audio features;

[0033] Lyrics features are extracted from the target lyrics data to obtain target lyrics features;

[0034] The target audio features and the target lyrics features are concatenated to obtain the target music features.

[0035] In some embodiments, the step of scoring the target music by genre based on the music classification model and the target music features to obtain genre score data, and obtaining the genre category of the target music based on the genre score data, includes:

[0036] Based on the preset function and candidate genre categories of the music classification model, the target music features are scored by genre to obtain the genre score data, wherein the genre score data is used to represent the probability that the target music belongs to each of the candidate genre categories;

[0037] The candidate genre category with the highest genre score is selected as the genre category of the target music.

[0038] To achieve the above objectives, a second aspect of this application provides a music classification device, the device comprising:

[0039] The sample data acquisition module is used to acquire sample audio data and sample lyrics data of sample music.

[0040] The audio feature extraction module is used to extract audio features from the sample audio data to obtain sample audio features;

[0041] The lyrics feature extraction module is used to extract lyrics features from the sample lyrics data to obtain sample lyrics features;

[0042] The sample pair construction module is used to construct positive music sample pairs and negative music sample pairs based on the sample audio features and the sample lyrics features;

[0043] The model training module is used to train a preset neural network model based on the positive music sample pairs and the negative music sample pairs to obtain a music classification model.

[0044] The target data acquisition module is used to acquire the target data of the target music.

[0045] The target feature extraction module is used to extract features from the target data to obtain the target music features of the target music.

[0046] The genre classification module is used to score the target music by genre based on the music classification model and the characteristics of the target music, obtain genre score data, and obtain the genre category of the target music based on the genre score data.

[0047] To achieve the above objectives, a third aspect of the present application provides an electronic device, the electronic device including a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the method described in the first aspect.

[0048] To achieve the above objectives, a fourth aspect of the present application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method described in the first aspect.

[0049] The music classification method, device, electronic device, and storage medium proposed in this application acquire sample audio data and sample lyrics data of sample music; extract audio features from the sample audio data to obtain sample audio features; and extract lyrics features from the sample lyrics data to obtain sample lyrics features. This method can simultaneously extract both lyric and audio information from the sample music, improving the richness of feature information. Furthermore, based on the sample audio and lyric features, positive and negative music sample pairs are constructed; a pre-defined neural network model is trained using these positive and negative music sample pairs to obtain a music classification model. This model can fuse features from different modalities using contrastive learning, constructing multiple sample pairs. Using positive and negative music sample pairs for model training facilitates the model's learning of the correlation and differences between sample pairs, significantly improving the model's accuracy in music classification. Finally, target data for the target music is obtained; features are extracted from the target data to obtain the target music features; the target music is scored by genre based on the music classification model and the target music features to obtain genre score data, and the genre category of the target music is obtained based on the genre score data. This can better realize the model's classification of music in the genre dimension and improve the accuracy of music classification. Attached Figure Description

[0050] Figure 1 This is a flowchart of the music classification method provided in the embodiments of this application;

[0051] Figure 2 yes Figure 1 The flowchart of step S102 in the document;

[0052] Figure 3 yes Figure 1 The flowchart of step S103 in the process;

[0053] Figure 4 yes Figure 1 The flowchart of step S104 in the process;

[0054] Figure 5 yes Figure 1 The flowchart of step S105 in the process;

[0055] Figure 6 yes Figure 1 The flowchart of step S107 in the process;

[0056] Figure 7 yes Figure 1 The flowchart of step S108 in the process;

[0057] Figure 8 This is a schematic diagram of the structure of the music classification device provided in the embodiments of this application;

[0058] Figure 9 This is a schematic diagram of the hardware structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0059] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0060] It should be noted that although functional modules are divided in the device schematic diagram and a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first," "second," etc., in the specification, claims, and the aforementioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0061] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0062] First, let's analyze some of the terms used in this application:

[0063] Artificial intelligence (AI) is a new branch of computer science that studies, develops, and applies theories, methods, technologies, and systems to simulate, extend, and expand human intelligence. It aims to understand the essence of intelligence and produce intelligent machines that can react in a way similar to human intelligence. Research in this field includes robotics, speech recognition, image recognition, natural language processing, and expert systems. AI can simulate the information processes of human consciousness and thought. Furthermore, AI utilizes digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceiving the environment, acquiring knowledge, and using that knowledge to achieve optimal results.

[0064] Natural Language Processing (NLP): NLP uses computers to process, understand, and utilize human language (such as Chinese and English). NLP is a branch of artificial intelligence and an interdisciplinary field of computer science and linguistics, often referred to as computational linguistics. NLP includes syntactic analysis, semantic analysis, and discourse understanding. It is commonly used in machine translation, handwritten and printed character recognition, speech recognition and text-to-speech conversion, intent recognition, information extraction and filtering, text classification and clustering, sentiment analysis, and opinion mining. It involves data mining, machine learning, knowledge acquisition, knowledge engineering, artificial intelligence research, and linguistic research related to language computation.

[0065] Information Extraction (NER) is a text processing technique that extracts factual information such as entities, relationships, and events from natural language text and outputs it as structured data. Information extraction is a technique for extracting specific information from text data. Text data is composed of specific units, such as sentences, paragraphs, and chapters. Text information is composed of smaller, specific units, such as characters, words, phrases, sentences, paragraphs, or combinations of these units. Extracting noun phrases, names of people, and place names from text data is an example of text information extraction. Of course, text information extraction techniques can extract information of various types.

[0066] Fourier transform: Represents a function that satisfies certain conditions as a linear combination of trigonometric functions (sine and / or cosine functions) or their integrals. In different research fields, the Fourier transform has various variants, such as the continuous Fourier transform and the discrete Fourier transform.

[0067] Mel-Frequency Cipstal Coefficients (MFCCs) are a set of key coefficients used to construct a Mel-Frequency Cipstal spectrum. From a segment of a music signal, a set of cepstrum values ​​can be obtained that is sufficient to represent the music signal. The Mel-Frequency Cipstal Coefficients are the cepstrum values ​​derived from this cepstrum (i.e., the spectrum of the spectrum). Unlike a regular cepstrum, the most distinctive feature of the Mel-Frequency Cipstrum is that its frequency bands are uniformly distributed across the Mel scale. In other words, compared to the linear cepstrum representations commonly seen, this frequency band is closer to the non-linear human auditory system. For example, Mel-Frequency Cipstals are frequently used in audio compression techniques.

[0068] Encoding: This involves transforming an input sequence into a vector of fixed length.

[0069] Decoding: This involves transforming a previously generated fixed vector into an output sequence; the input sequence can be text, speech, image, or video; the output sequence can be text or image.

[0070] Softmax function: The Softmax function is a normalized exponential function.

[0071] Currently, music classification tasks often rely on extracted music features to determine the genre. However, feature extraction often results in missing features, leading to low accuracy in music classification. Therefore, improving the accuracy of music classification has become an urgent technical problem to be solved.

[0072] Based on this, embodiments of this application provide a music classification method, a music classification device, an electronic device, and a storage medium, aiming to improve the accuracy of music classification.

[0073] The music classification method, apparatus, electronic device, and storage medium provided in this application are specifically described through the following embodiments. First, the music classification method in this application is described.

[0074] The embodiments of this application can acquire and process relevant data based on artificial intelligence technology. Artificial intelligence (AI) refers to the theories, methods, technologies, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results.

[0075] Foundational technologies in artificial intelligence generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies mainly encompass computer vision, robotics, biometrics, speech processing, natural language processing, and machine learning / deep learning.

[0076] The music classification method provided in this application relates to the field of artificial intelligence technology. The music classification method provided in this application can be applied to a terminal, a server, or software running on either a terminal or a server. In some embodiments, the terminal can be a smartphone, tablet, laptop, desktop computer, etc.; the server can be configured as an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application implementing the music classification method, but is not limited to the above forms.

[0077] It should be noted that in all specific embodiments of this application, when processing data related to user identity or characteristics, such as user information, user behavior data, user historical data, and user location information, user permission or consent is obtained first. Furthermore, the collection, use, and processing of this data comply with relevant laws, regulations, and standards of the relevant countries and regions. In addition, when embodiments of this application require access to sensitive personal information of users, separate permission or consent from the user is obtained through pop-ups or redirects to confirmation pages. Only after obtaining the user's separate permission or consent is the necessary user-related data for the proper functioning of the embodiments of this application obtained.

[0078] This application can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0079] Figure 1 This is an optional flowchart of the music classification method provided in the embodiments of this application. Figure 1 The method may include, but is not limited to, steps S101 to S108.

[0080] Step S101: Obtain sample audio data and sample lyrics data of the sample music;

[0081] Step S102: Extract audio features from the sample audio data to obtain the sample audio features;

[0082] Step S103: Extract lyrics features from the sample lyrics data to obtain sample lyrics features;

[0083] Step S104: Based on the sample audio features and sample lyrics features, construct positive music sample pairs and negative music sample pairs;

[0084] Step S105: Train the preset neural network model based on positive music sample pairs and negative music sample pairs to obtain a music classification model;

[0085] Step S106: Obtain the target data for the target music;

[0086] Step S107: Extract features from the target data to obtain the target music features;

[0087] Step S108: Based on the music classification model and the characteristics of the target music, the genre score of the target music is obtained to obtain genre score data, and the genre category of the target music is obtained based on the genre score data.

[0088] Steps S101 to S108 of this embodiment involve acquiring sample audio data and sample lyrics data of sample music; extracting audio features from the sample audio data to obtain sample audio features; and extracting lyric features from the sample lyrics data to obtain sample lyric features. This method can simultaneously extract lyric and audio information of the sample music, improving the richness of feature information. Furthermore, based on the sample audio and lyric features, positive and negative music sample pairs are constructed; a preset neural network model is trained using these positive and negative music sample pairs to obtain a music classification model. This model can fuse features from different modalities based on contrastive learning, constructing multiple sample pairs. Using positive and negative music sample pairs for model training helps the model learn the correlation and differences between sample pairs, significantly improving the model's accuracy in music classification. Finally, target data for the target music is obtained; features are extracted from the target data to obtain the target music features; the target music is scored by genre based on the music classification model and the target music features to obtain genre score data, and the genre category of the target music is obtained based on the genre score data. This can better realize the model's classification of music in the genre dimension and improve the accuracy of music classification.

[0089] In step S101 of some embodiments, the music data of the sample music can be extracted from a preset music database, or downloaded through a network platform or other channels. Other methods can also be used to obtain the music data of the sample music, and these are not limited to these. The music data of the sample music includes sample audio data and sample lyrics data. The sample music can include music of different languages, scenes, and styles. For example, the sample music includes Chinese songs, English songs, or Japanese songs, etc. The sample audio data can be a WAV file in the form of a one-dimensional time-series signal.

[0090] Please see Figure 2 In some embodiments, step S102 may include, but is not limited to, steps S201 to S204:

[0091] Step S201: Convert the format of the sample audio data to obtain sample spectrum data;

[0092] Step S202: Input the sample spectrum data into a preset feature extraction model, wherein the feature extraction model includes convolutional layers and pooling layers;

[0093] Step S203: Perform two-dimensional convolution processing on the sample spectral data based on the convolutional layer to obtain preliminary spectral features;

[0094] Step S204: Perform max pooling on the preliminary spectral features based on the pooling layer to obtain the sample audio features.

[0095] In step S201 of some embodiments, a preset function can be called from a commonly used speech processing database. For example, the speech processing database can be the librosa library, and the preset function can be the librosa.feature.melspectrogram() function in the librosa library. The librosa.feature.melspectrogram() function is directly used to extract spectral features from the sample audio data and perform format conversion. First, the sample audio data is converted into a Fourier transform spectrum, and then the Fourier transform spectrum is converted into a Mel-frequency spectrum to obtain the sample spectral data, which is a Mel-frequency cepstral feature. This method can convert one-dimensional, difficult-to-process time-series signals into two-dimensional frequency domain data that is easy to process and rich in audio information.

[0096] In step S202 of some embodiments, sample spectrum data can be input into a preset feature extraction model using a preset computer program. The feature extraction network can be constructed based on a two-dimensional convolutional neural network model, and the feature extraction network includes two-dimensional convolutional layers and pooling layers.

[0097] In step S203 of some embodiments, the sample spectrum data is convolved based on a two-dimensional convolutional layer to extract the spectrum information in the sample spectrum data and obtain preliminary spectrum features.

[0098] In step S204 of some embodiments, max pooling is performed on the preliminary spectral features based on the pooling layer, which can maximize the acquisition of all spectral feature content in the sample spectral data to obtain sample audio features. The sample audio features include the rhythmic characteristics, pitch characteristics, and music content information of the sample music, etc.

[0099] Furthermore, the feature extraction network in this embodiment of the application also includes a dropout layer and a batch normalization layer. The dropout layer is used to prevent the feature extraction network from overfitting during the model training stage, and the batch normalization layer is used to calculate the mean and standard deviation of the sample data during the model training stage, and adjust the parameters of the feature extraction network based on the calculation results to improve the output stability of the feature extraction network.

[0100] Through the above steps S201 to S204, the sample audio data can be easily converted from time domain signal to frequency domain signal, and the feature extraction network can be used to extract features from the frequency domain signal (i.e. sample spectrum data). This can improve the feature quality of the obtained sample audio features, which is beneficial for model training based on sample audio features. It can also improve the model's ability to learn the spectrum information of sample music and improve the accuracy of music classification.

[0101] Please see Figure 3 In some embodiments, step S103 may include, but is not limited to, steps S301 to S303:

[0102] Step S301: Input the sample lyrics data into a preset text encoding model, wherein the text encoding model includes an embedding layer and a transformer encoder;

[0103] Step S302: The sample lyrics data is embedded based on the embedding layer to obtain the embedded representation of the sample lyrics;

[0104] Step S303: Based on the transformer encoder, feature encoding is performed on the embedded representation of the sample lyrics to obtain the sample lyrics features.

[0105] In step S301 of some embodiments, sample lyrics data can be input into a preset text encoding model using a preset computer program. The text encoding model can be constructed based on the BERT model, etc., and includes an embedding layer and a transformer encoder.

[0106] In step S302 of some embodiments, when embedding the sample lyrics data based on the embedding layer, the text segments in the sample lyrics data can be embedded from a high-dimensional space into a preset low-dimensional continuous vector space, and the text segments in the sample lyrics data can be converted into vector features in the real number field to obtain the sample lyrics embedding representation. The sample lyrics embedding representation is the sum of word embedding, position embedding and type embedding of the text segments. The word embedding represents the word information of the text segments, the position embedding represents the position information of the text segments, the type embedding represents the part-of-speech category information of the text segments, and so on.

[0107] In step S303 of some embodiments, the transformer encoder is a bidirectional encoder composed of multiple stacked coding units, each of which includes a self-attention layer, a residual layer, a normalization layer, and a dropout layer. When the transformer encoder performs feature encoding on the sample lyrics embedding representation, the self-attention layer performs global self-attention processing on the sample lyrics embedding representation, and the residual layer performs a residual concatenation between the result of the self-attention processing and the sample lyrics embedding representation. Then, the normalization layer normalizes the mean and standard deviation of the residual concatenation result, and uses the normalized result as the sample lyrics feature. This process can conveniently extract semantic information from the sample lyrics data, which can contain the textual content characteristics of the sample music.

[0108] Through the above steps S301 to S303, the sample lyrics data can be easily embedded into a preset low-dimensional continuous vector space to obtain the sample lyrics embedding representation. Based on the sample lyrics embedding representation, the text content information of the sample music can be extracted to obtain the sample lyrics features. The genre information contained in the sample lyrics features can be used to train the music classification ability of the model, which can improve the accuracy of the model in music classification.

[0109] Please see Figure 4 In some embodiments, both the positive music sample pair and the negative music sample pair include a sample audio feature and a sample lyrics feature. Step S104 may include, but is not limited to, steps S401 to S402:

[0110] Step S401: Construct positive music sample pairs based on the sample audio features and sample lyric features from the same sample music.

[0111] Step S402: Based on the sample audio features and sample lyric features from different sample music, construct negative music sample pairs.

[0112] In step S401 of some embodiments, since the musical information correlation between sample audio features and sample lyric features from the same sample music is high, positive music sample pairs are constructed based on the sample audio features and sample lyric features from the same sample music. A positive music sample pair includes both sample audio features and sample lyric features from the same sample music. For example, a positive music sample pair includes... and in, The sample audio features of the i-th sample music and sample lyrics features constitute, The sample audio features of the j-th sample music and sample lyrics features constitute.

[0113] In step S402 of some embodiments, since the differences in musical information between the sample audio features and sample lyric features from different sample music are high, negative music sample pairs are constructed based on the sample audio features and sample lyric features from different sample music. A negative music sample pair includes sample audio features from one sample music and sample lyric features from another sample music. For example, a negative music sample pair includes... and in, The sample audio features of the i-th sample music The sample lyrics features of the j-th sample music constitute, The sample audio features of the j-th sample music The sample lyrics features of the i-th sample music constitute.

[0114] Through the above steps S401 to S402, multiple positive and negative music sample pairs can be constructed, reducing the difference in musical information between the sample audio features and sample lyric features of the positive music sample pairs, and increasing the difference in musical information between the sample audio features and sample lyric features of the negative music sample pairs. Based on the positive and negative music sample pairs, the neural network model is trained, improving the model's ability to learn the differences in musical information between the sample audio features and sample lyric features within the positive and negative music sample pairs, thereby improving the model's ability to identify musical information and improving the accuracy of the model's music classification.

[0115] Please see Figure 5 In some embodiments, step S105 may include, but is not limited to, steps S501 to S505:

[0116] Step S501: Perform feature dimensionality reduction on the positive music sample pairs to obtain positive music feature pairs, and perform feature dimensionality reduction on the negative music sample pairs to obtain negative music feature pairs, wherein the negative music feature pairs include a first negative feature pair and a second negative feature pair;

[0117] Step S502: Perform feature similarity scoring on positive music feature pairs to obtain first similarity data; perform feature similarity scoring on first negative feature pairs to obtain second similarity data; and perform feature similarity scoring on second negative feature pairs to obtain third similarity data.

[0118] Step S503: Calculate the loss based on the first similar data and the second similar data to obtain the first loss data;

[0119] Step S504: Calculate the loss based on the first similar data and the third similar data to obtain the second loss data;

[0120] Step S505: Optimize the parameters of the neural network model based on the first loss data and the second loss data to obtain the music classification model.

[0121] In step S501 of some embodiments, when performing feature dimensionality reduction on positive and negative music sample pairs, a linear mapping layer g can be used. a Feature dimensionality reduction is performed on the sample audio features using a linear mapping layer g. l Perform dimensionality reduction on the sample lyrics features to obtain the corresponding positive music feature pairs for each positive music sample pair. For example, for each positive music sample pair... Perform feature reduction to obtain positive music feature pairs For example, negative music sample pairs Perform feature dimensionality reduction to obtain the first negative feature pair. negative music sample pairs Perform feature dimensionality reduction to obtain the second negative feature pair.

[0122] In step S502 of some embodiments, when performing feature similarity scoring on positive music feature pairs, first negative feature pairs, and second negative feature pairs, cosine similarity algorithm or Euclidean distance method can be used to calculate similarity, obtaining the first similarity data corresponding to the positive music feature pairs, the second similarity data corresponding to the first negative music feature pairs, and the third similarity data corresponding to the second negative music feature pairs. The formula for calculating the feature similarity score can be as shown in formula (1):

[0123] s(A, L) = g a (E a )·g l (E l ) Formula (1)

[0124] Where s(A,L) represents the calculated similarity data, and g a (E a ) represents the sample audio features after feature reduction, g l (E l () represents the lyric features of the sample after feature reduction.

[0125] In step S503 of some embodiments, the process of calculating the loss based on the first similarity data and the second similarity data to obtain the first loss data can be represented as shown in formula (2):

[0126]

[0127] Among them, L A→L The first loss term is the audio-to-lyrics loss term; B is the total number of sample music tracks; τ1 is the model parameter; j is the j-th sample music track; s(A i L i Let s(A) be the first similarity data between the sample audio features and sample lyric features of the i-th sample music. i L i ) represents the second similarity data between the sample audio features of the i-th sample music and the sample lyrics features of the j-th sample music.

[0128] In step S504 of some embodiments, the process of calculating the loss based on the first similarity data and the third similarity data to obtain the second loss data can be represented as shown in formula (3):

[0129]

[0130] Among them, L L→A The second loss data is the loss term from lyrics to audio; B is the total number of sample music tracks, τ2 is the model parameter, and s(A) i L i Let s(A) be the first similarity data between the sample audio features and sample lyric features of the i-th sample music. j L i ) represents the third similarity data between the sample audio features of the j-th sample music and the sample lyrics features of the i-th sample music.

[0131] In step S505 of some embodiments, the first loss data and the second loss data are summed to obtain the target loss data. Based on the relationship between the target loss data and the preset loss threshold, the model parameters of the neural network model are continuously adjusted, and the neural network model is iteratively trained so that the neural network model can better learn and mine the correlation of music genre information in the audio features and lyric features of the same sample music, as well as the differences in music genre information in the audio features and lyric features of different sample music. The model parameters that ultimately make the target loss data less than the loss threshold are used as the final model parameters, thereby obtaining the music classification model.

[0132] Through the above steps S501 to S505, comparative learning can be performed based on positive music sample pairs and negative music sample pairs. The first loss data and the second loss data obtained from the comparative learning are used to train the neural network model. This comparative learning method can effectively eliminate the differences between features of different modalities, enhance the complementarity between different modalities, and improve the model's feature fusion ability for features of different modalities. At the same time, the music classification method of this application is based on the feature information of the lyrics features and audio features of the sample music (i.e., features of two different modalities) for model training. This enables the model to learn the genre information in the audio features and the genre information in the lyrics features of the sample music at the same time, which can effectively improve the accuracy of the model in classifying music genres.

[0133] In step S106 of some embodiments, target data of the target music can be extracted from a preset music database, or downloaded through a network platform or other channels. Other methods can also be used to obtain the target data, and this is not limited to these. The target data of the target music includes target audio data and target lyrics data. The target audio data can be a WAV file in the form of a one-dimensional time-series signal.

[0134] Please see Figure 6 In some embodiments, the target data includes target audio data and target lyrics data of the target music, and step S107 includes, but is not limited to, steps S601 to S603:

[0135] Step S601: Extract audio features from the target audio data to obtain the target audio features;

[0136] Step S602: Extract lyrics features from the target lyrics data to obtain target lyrics features;

[0137] Step S603: Perform feature concatenation on the target audio features and target lyrics features to obtain the target music features.

[0138] In some embodiments, step S601 involves extracting audio features from the target audio data to obtain the target audio features. The specific process is basically the same as the feature extraction process in step S102 described above, and will not be repeated here.

[0139] In step S602 of some embodiments, the specific process of extracting lyrics features from the target lyrics data to obtain the target lyrics features is basically the same as the feature extraction process in step S103 above, and will not be repeated here.

[0140] In step S603 of some embodiments, when performing feature concatenation on the target audio features and target lyrics features, the target audio features and target lyrics features can be vector concatenated or vector added to obtain the target music features.

[0141] Through the above steps S601 to S603, the audio feature information and lyrics text feature information of the target music can be extracted, and the target music features can be generated based on the audio feature information and lyrics text feature information. This allows the target music features to simultaneously possess the audio information and lyrics information of the target music, and can also perform good feature fusion of features from different modalities, thereby improving the feature quality of the target music features. The target music features can then be used in the subsequent music classification process, which is beneficial to improving the accuracy of music classification.

[0142] Please see Figure 7 In some embodiments, step S108 may include, but is not limited to, steps S701 to S702:

[0143] Step S701: Based on the preset function of the music classification model and the candidate genre categories, the target music features are scored by genre to obtain genre score data, wherein the genre score data is used to represent the probability that the target music belongs to each candidate genre category.

[0144] Step S702: Select the candidate genre category with the largest genre score data as the genre category of the target music.

[0145] In step S701 of some embodiments, the preset function can be a softmax classifier, etc., without limitation. Taking a softmax classifier as an example, a probability distribution of the target music feature on each candidate genre category is created based on the softmax classifier, thereby achieving a music genre score for the target music feature, and using the probability distribution vector of each candidate genre category as the genre score data of the target music on the candidate genre category.

[0146] In step S702 of some embodiments, since the size of the genre score data can directly reflect the probability that the target music belongs to each candidate genre category, that is, the larger the genre score data, the more the genre characteristics of the target music are biased towards the candidate genre category corresponding to that genre score data. Therefore, the largest genre score data can be selected from the genre score data as the target score data.

[0147] Furthermore, after determining the target score data, the candidate genre categories are filtered based on the target score data, and the candidate genre category corresponding to the target score data is selected as the target genre label, and the target genre label is used as the genre category of the target music.

[0148] It should be noted that the candidate music genre tags include pop, rock, folk, electronic, light music, blues, etc., and can be set according to actual needs, and are not limited to these.

[0149] Through the above steps S701 to S702, the genre category of the target music can be clearly determined based on the target music characteristics. It is possible to predict the probability distribution of the target music in each candidate genre category based on the target music characteristics, and to quantify the probability that the genre information represented in the target music belongs to each candidate genre category using genre score data. This can significantly improve the accuracy of genre classification of the target music.

[0150] The music classification method of this application acquires sample audio data and sample lyrics data of sample music; extracts audio features from the sample audio data to obtain sample audio features; and extracts lyrics features from the sample lyrics data to obtain sample lyrics features. This method can simultaneously extract lyrics and audio information of the sample music, improving the richness of feature information. Furthermore, based on the sample audio features and sample lyrics features, positive music sample pairs and negative music sample pairs are constructed; a preset neural network model is trained based on the positive and negative music sample pairs to obtain a music classification model. This method can fuse features from different modalities based on contrastive learning, constructing multiple sample pairs. Using positive and negative music sample pairs for model training helps the model learn the correlation and differences between sample pairs, significantly improving the model's accuracy in music classification. Finally, target data for the target music is obtained; features are extracted from the target data to obtain the target music features; the target music is scored by genre based on the music classification model and the target music features to obtain genre score data, and the genre category of the target music is obtained based on the genre score data. This can better realize the model's classification of music in the genre dimension and improve the accuracy of music classification.

[0151] Please see Figure 8This application also provides a music classification device that can implement the above-described music classification method. The device includes:

[0152] The sample data acquisition module 801 is used to acquire sample audio data and sample lyrics data of the sample music;

[0153] The audio feature extraction module 802 is used to extract audio features from the sample audio data to obtain the sample audio features.

[0154] The lyrics feature extraction module 803 is used to extract lyrics features from the sample lyrics data to obtain sample lyrics features;

[0155] The sample pair construction module 804 is used to construct positive music sample pairs and negative music sample pairs based on sample audio features and sample lyric features;

[0156] The model training module 805 is used to train a preset neural network model based on positive music sample pairs and negative music sample pairs to obtain a music classification model.

[0157] Target data acquisition module 806 is used to acquire target data of the target music;

[0158] The target feature extraction module 807 is used to extract features from the target data to obtain the target music features of the target music.

[0159] The genre classification module 808 is used to score the target music by genre based on the music classification model and the characteristics of the target music, obtain genre score data, and obtain the genre category of the target music based on the genre score data.

[0160] The specific implementation of this music classification device is basically the same as the specific implementation of the music classification method described above, and will not be repeated here.

[0161] This application also provides an electronic device, which includes: a memory, a processor, a program stored in the memory and executable on the processor, and a data bus for communication between the processor and the memory. When the program is executed by the processor, it implements the aforementioned music classification method. This electronic device can be any smart terminal, including tablet computers, in-vehicle computers, etc.

[0162] Please see Figure 9 , Figure 9 The hardware structure of an electronic device according to another embodiment is illustrated. The electronic device includes:

[0163] The processor 901 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this application.

[0164] The memory 902 can be implemented as a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 902 can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 902 and is called and executed by the processor 901 using the music classification method of the embodiments of this application.

[0165] The input / output interface 903 is used to implement information input and output;

[0166] The communication interface 904 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0167] Bus 905 transmits information between various components of the device (e.g., processor 901, memory 902, input / output interface 903, and communication interface 904);

[0168] The processor 901, memory 902, input / output interface 903, and communication interface 904 are connected to each other within the device via bus 905.

[0169] This application also provides a computer-readable storage medium storing one or more programs that can be executed by one or more processors to implement the above-described music classification method.

[0170] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0171] The music classification method, music classification device, electronic device, and computer-readable storage medium provided in this application acquire sample audio data and sample lyrics data of sample music; extract audio features from the sample audio data to obtain sample audio features; and extract lyrics features from the sample lyrics data to obtain sample lyrics features. This method can simultaneously extract lyrics and audio information of the sample music, improving the richness of feature information. Furthermore, based on the sample audio features and sample lyrics features, positive music sample pairs and negative music sample pairs are constructed; a preset neural network model is trained based on the positive and negative music sample pairs to obtain a music classification model. This model can fuse features from different modalities based on contrastive learning, constructing multiple sample pairs. Using positive and negative music sample pairs for model training helps the model learn the correlation and differences between sample pairs, significantly improving the model's accuracy in music classification. Finally, target data for the target music is obtained; features are extracted from the target data to obtain target music features; the target music is scored by genre based on the music classification model and the target music features to obtain genre score data, and the genre category of the target music is obtained based on the genre score data. This can effectively classify music in the genre dimension, improving the accuracy of music classification. Furthermore, the idea of ​​using contrastive learning for model training in this embodiment can be extended to other tasks and scenarios, exhibiting good generalization and versatility.

[0172] The embodiments described in this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided by the embodiments of this application. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.

[0173] It will be understood by those skilled in the art that Figure 1-7 The technical solutions shown do not constitute a limitation on the embodiments of this application, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0174] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0175] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or appropriate combinations thereof.

[0176] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0177] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0178] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0179] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0180] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0181] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0182] The preferred embodiments of the present application have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present application. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and substance of the embodiments of the present application shall be within the scope of the claims of the present application.

Claims

1. A music classification method, characterized in that, The method includes: Obtain sample audio data and sample lyrics data of the sample music; Audio features are extracted from the sample audio data to obtain sample audio features; Lyrics features are extracted from the sample lyrics data to obtain sample lyrics features; Based on the sample audio features and the sample lyrics features, construct positive music sample pairs and negative music sample pairs; Based on the positive music sample pairs and the negative music sample pairs, a preset neural network model is trained to obtain a music classification model; Obtain target data for the target music; extract features from the target data to obtain the target music features of the target music; Based on the music classification model and the target music features, the target music is scored by genre to obtain genre score data, and the genre category of the target music is obtained based on the genre score data. Both the positive music sample pair and the negative music sample pair include a sample audio feature and a sample lyric feature. The construction of the positive music sample pair and the negative music sample pair based on the sample audio feature and the sample lyric feature includes: Based on the sample audio features and sample lyric features from the same sample music, the positive music sample pair is constructed; based on the sample audio features and sample lyric features from different sample music, the negative music sample pair is constructed. The process of training a preset neural network model based on the positive and negative music sample pairs to obtain a music classification model includes: The positive music sample pairs are subjected to feature dimensionality reduction to obtain positive music feature pairs, and the negative music sample pairs are subjected to feature dimensionality reduction to obtain negative music feature pairs, wherein the negative music feature pairs include a first negative feature pair and a second negative feature pair; The positive music feature pairs are scored for feature similarity to obtain first similar data; the first negative feature pairs are scored for feature similarity to obtain second similar data; and the second negative feature pairs are scored for feature similarity to obtain third similar data. A loss is calculated based on the first and second similar data to obtain first loss data; a loss is calculated based on the first and third similar data to obtain second loss data; and the parameters of the neural network model are optimized based on the first and second loss data to obtain the music classification model.

2. The music classification method according to claim 1, characterized in that, The step of extracting audio features from the sample audio data to obtain sample audio features includes: The sample audio data is converted to a different format to obtain sample spectrum data; The sample spectral data is input into a preset feature extraction model, wherein the feature extraction model includes convolutional layers and pooling layers; Based on the convolutional layer, the sample spectral data is subjected to two-dimensional convolution processing to obtain preliminary spectral features; The initial spectral features are subjected to max pooling based on the pooling layer to obtain the sample audio features.

3. The music classification method according to claim 1, characterized in that, The step of extracting lyric features from the sample lyric data to obtain sample lyric features includes: The sample lyrics data is input into a preset text encoding model, wherein the text encoding model includes an embedding layer and a transformer encoder; The sample lyrics data are embedded based on the embedding layer to obtain the sample lyrics embedded representation. The sample lyrics features are obtained by performing feature encoding on the sample lyrics embedding representation based on the transformer encoder.

4. The music classification method according to claim 1, characterized in that, The target data includes the target audio data and target lyrics data of the target music. The step of extracting features from the target data to obtain the target music features includes: Audio features are extracted from the target audio data to obtain the target audio features; Lyrics features are extracted from the target lyrics data to obtain target lyrics features; The target audio features and the target lyrics features are concatenated to obtain the target music features.

5. The music classification method according to any one of claims 1 to 4, characterized in that, The process of scoring the target music by genre based on the music classification model and the target music features to obtain genre score data, and obtaining the genre category of the target music based on the genre score data, includes: Based on the preset function and candidate genre categories of the music classification model, the target music features are scored by genre to obtain the genre score data, wherein the genre score data is used to represent the probability that the target music belongs to each of the candidate genre categories; The candidate genre category with the highest genre score is selected as the genre category of the target music.

6. A music classification device, characterized in that, The device includes: The sample data acquisition module is used to acquire sample audio data and sample lyrics data of sample music. The audio feature extraction module is used to extract audio features from the sample audio data to obtain sample audio features; The lyrics feature extraction module is used to extract lyrics features from the sample lyrics data to obtain sample lyrics features; The sample pair construction module is used to construct positive music sample pairs and negative music sample pairs based on the sample audio features and the sample lyrics features; The model training module is used to train a preset neural network model based on the positive music sample pairs and the negative music sample pairs to obtain a music classification model. The target data acquisition module is used to acquire the target data of the target music. The target feature extraction module is used to extract features from the target data to obtain the target music features of the target music. The genre classification module is used to score the target music by genre based on the music classification model and the characteristics of the target music, obtain genre score data, and obtain the genre category of the target music based on the genre score data. Both the positive music sample pair and the negative music sample pair include a sample audio feature and a sample lyric feature. The construction of the positive music sample pair and the negative music sample pair based on the sample audio feature and the sample lyric feature includes: Based on the sample audio features and sample lyric features from the same sample music, the positive music sample pair is constructed; based on the sample audio features and sample lyric features from different sample music, the negative music sample pair is constructed. The process of training a preset neural network model based on the positive and negative music sample pairs to obtain a music classification model includes: The positive music sample pairs are subjected to feature dimensionality reduction to obtain positive music feature pairs, and the negative music sample pairs are subjected to feature dimensionality reduction to obtain negative music feature pairs, wherein the negative music feature pairs include a first negative feature pair and a second negative feature pair; The positive music feature pairs are scored for feature similarity to obtain first similar data; the first negative feature pairs are scored for feature similarity to obtain second similar data; and the second negative feature pairs are scored for feature similarity to obtain third similar data. A loss is calculated based on the first and second similar data to obtain first loss data; a loss is calculated based on the first and third similar data to obtain second loss data; and the parameters of the neural network model are optimized based on the first and second loss data to obtain the music classification model.

7. An electronic device, characterized in that, The electronic device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the music classification method as described in any one of claims 1 to 5.

8. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the music classification method according to any one of claims 1 to 5.

Images