A music genre classification method and system fusing a knowledge graph

Abstract

The application discloses a music genre classification method and system fusing a knowledge graph, extracts initial audio features and calculates corresponding genre prediction scores, then learns the relationship between the genres in the knowledge graph by using a graph neural network, fuses the knowledge graph and the audio features to obtain enhanced audio features, and finally is used for automatically identifying music genres. The application first proposes to use a knowledge graph to guide audio representation learning and apply it to genre classification. Meanwhile, the application uses the metadata of a public music dataset to construct a knowledge graph, learns audio features fused with the relationship knowledge between the genres, and achieves better genre classification performance, so the application has wide application prospects.

Description

Technical Field

[0001] This invention belongs to the field of music signal analysis and processing technology, specifically relating to a music genre classification method and system that integrates knowledge graphs. Background Technology

[0002] Music genre classification has applications in many real-world scenarios, such as music streaming platforms creating more suitable recommended playlists for specific users and users discovering other music similar to their preferred genres. However, the boundaries between different music genres remain blurred, making automatic music genre recognition (MGR) from audio samples an important task.

[0003] Experts in related fields have proposed several methods to address this problem. Early approaches explored using different inputs (i.e., waveforms or time-spectrum graphs) or different classifiers for music classification; for example, using self-supervised learning models reconstructed from multiple audio features (such as MFCC, Chroma, Tempogram, etc.) to improve classification performance. Recent research has proposed leveraging relevant tasks (such as artist labels) to obtain multi-level and multi-scale music representations and utilizing transfer learning to enhance genre classifiers. All of these solutions use only audio samples as input. Other methods utilize additional information (such as lyrics, comments, etc.) for genre classification; for example, using natural language descriptions of the music content to supervise the learning of audio representations, or combining lyrics and audio to extract fused features for genre classification. However, it is worth noting that many open-source datasets or real-world tasks do not provide such detailed information. Obtaining accurate lyrics or descriptions for each audio segment requires the assistance of music APIs or search engines, which is a very labor-intensive and time-consuming process. Summary of the Invention

[0004] The technical problem to be solved by this invention is to address the shortcomings of the prior art by providing a music genre classification method and system that integrates knowledge graphs. It proposes to use knowledge graphs to guide audio representation learning without needing to obtain additional information for each audio segment, thus effectively improving the performance of music genre classification and solving the technical problem of automatic music genre classification using side information-assisted neural networks.

[0005] The present invention adopts the following technical solution:

[0006] A music genre classification method integrating knowledge graphs includes the following steps:

[0007] S1. Convert the audio data into a Mel spectrogram, input the Mel spectrogram into the audio feature extraction network to learn the audio representation, and add a linear layer at the end of the audio feature extraction network to obtain the prediction score for each genre.

[0008] S2. Construct a knowledge graph related to different schools of thought;

[0009] S3. Use the prediction scores of each school of thought obtained in step S1 to initialize the school of thought nodes in the knowledge graph constructed in step S2. Then, use a graph neural network to learn the feature vector of each school of thought node and concatenate all feature vectors to obtain the final knowledge representation.

[0010] S4. Using the SE module, different attention weights are assigned to the audio representation obtained in step S1 and the knowledge representation obtained in step S3. Then, the weighted representations are concatenated to obtain the enhanced audio representation. The enhanced audio representation is input into the fully connected layer to form a music genre classification model to achieve music genre classification.

[0011] Specifically, step S1 is as follows:

[0012] S101. Cut the audio χ into multiple non-overlapping segments with a duration of 1 second, and use the librosa library to convert the cut audio segments into a 128-dimensional Mel spectrogram to obtain the time-frequency representation S.

[0013] S102. Input the time-frequency representation S obtained in step S101 into the backbone network f(.) to learn the audio representation Z. a ;

[0014] S103. Add a linear layer g(.) after the backbone network f(.) in step S102, and obtain a C-dimensional vector Z by pre-training the network g○f. s , where represents the network's prediction score for each genre, and C represents the number of genres.

[0015] Furthermore, in step S102, the backbone network f(.) uses the Inception-ResNet-V2 architecture.

[0016] Specifically, in step S2, the knowledge graph It includes an entity set and an edge set; the entity set V contains G+A+I elements, where G is the number of music genres, A is the number of artists, and I is the number of instruments; the edge set E is in the knowledge graph. The middle part is the set of edges connecting the various entities.

[0017] Furthermore, knowledge graphs The edge set E in the equation is:

[0018]

[0019] Among them, 0 G×G It is a zero matrix of size G×G, P G×A It is a probability matrix of size G×A relating music genres and artists, P G×I It is a probability matrix of size G×I relating music genres and instruments, P A×G It is a probability matrix of size A×G relating artists and music genres, where 0 A×A It is a zero matrix of size A×A, 0 A×I It is a zero matrix of size A×I, P I×G It is a probability matrix of size I×G relating musical instruments to musical genres, where 0 I×A Let A be a zero matrix of size I×A, where 0 I×I It is a zero matrix of size I×I.

[0020] Specifically, step S3 is as follows:

[0021] S301. Initialize artist node A and instrument node I using the zero vector, and use the Z obtained in step S1. s Initialize the corresponding genre node G, and obtain the input feature x of each node after initialization. v ;

[0022] S302. At iteration t, the hidden state h of node i is... i Determined by the previous state and the messages propagated from its neighbors, after T iterations, the message is in the entire graph. The knowledge graph is propagated through a series of layers to obtain the final hidden states of all nodes. The final linear layer outputs the final features of each node, and these features are concatenated to obtain the entire knowledge graph. The characterization of Z KG .

[0023] Furthermore, in step S302, at iteration t, the hidden state h of node i is... i Its final state is determined by the messages propagated from its neighbors, specifically:

[0024]

[0025]

[0026] in, This is the initial hidden state of node i, x i These are the input features of node i. It represents the hidden state of node i at the t-th iteration. This is the hidden state of node K at the (t-1)th iteration, where K is the total number of nodes in the knowledge graph, and E is the hidden state of node K. iIt is a matrix representing the connection relationships between node i and its neighboring nodes.

[0027] Specifically, step S4 is as follows:

[0028] S401, Representing Knowledge Z KG With the initial audio representation Z a The knowledge representation Z is input into the SE module. KG With the initial audio representation Z a Different attention weights are assigned to adaptively determine features; then the weighted features are concatenated to obtain the audio representation F;

[0029] S402. Input the audio representation F obtained in step S401 into the music genre classification model for genre classification.

[0030] Furthermore, in step S402, the music genre classification model is trained using cross-entropy loss, where the cross-entropy loss L is:

[0031]

[0032] in, It is a prediction of genre tags. is the real label, and N is the total number of input audio samples.

[0033] Secondly, embodiments of the present invention provide a music genre classification system that integrates knowledge graphs, including:

[0034] The learning module converts audio data into Mel spectrograms, inputs the Mel spectrograms into the audio feature extraction network to learn audio representations, and adds a linear layer at the end of the audio feature extraction network to obtain prediction scores for each genre.

[0035] Build modules to construct knowledge graphs related to different genres;

[0036] The representation module initializes the faction nodes in the knowledge graph constructed by the building module with the prediction scores of each faction obtained by the learning module. Then, it uses a graph neural network to learn the feature vector of each faction node and concatenates all feature vectors to obtain the final knowledge representation.

[0037] The classification module uses the SE module to assign different attention weights to the audio representations obtained by the learning module and the knowledge representations obtained by the representation module. Then, the weighted representations are concatenated to obtain the enhanced audio representations. The enhanced audio representations are then input into the fully connected layer to form a music genre classification model, thereby achieving music genre classification.

[0038] Compared with the prior art, the present invention has at least the following beneficial effects:

[0039] This invention provides a music genre classification method that integrates knowledge graphs. It constructs a knowledge graph related to music genres using metadata (i.e., genres, artists, and instruments) provided in the FMA-medium dataset, utilizes GGNN to learn the correlations between different genres from the graph, and integrates the learned knowledge with audio representations to enhance the audio representations.

[0040] Furthermore, in order to obtain the initial audio representation, the audio representation learning network needs to be pre-trained. In order for the knowledge graph to guide the learning of audio representation in a targeted manner, a linear layer needs to be added after the audio representation learning network to obtain the prediction score of each audio segment for each genre. These prediction scores are then used to initialize the knowledge graph.

[0041] Furthermore, the backbone network f(.) adopts the architecture of the relatively mature classification network Inception-ResNet-V2. This network combines the Inception block with the residual network (ResNet), which has both low computational cost and fast training speed, and can effectively improve the performance of music genre classification.

[0042] Furthermore, to better enhance audio representation through the knowledge graph, it is necessary to construct a knowledge graph containing music genres and related information. Entity set V contains G+A+I elements related to identifying music genres, where G represents the number of music genres, A represents the number of artists, and I represents the number of instruments; edge set E is in the knowledge graph. The set of edges connecting entities represents the relationships between them.

[0043] Furthermore, to more easily input the knowledge graph into the GGNN network, it is necessary to... The edge set E in the matrix is ​​represented as a matrix. Relationships between entities that are not directly connected are represented by a zero matrix. The value of the edge connecting two entities at the corresponding position in the matrix is ​​the correlation probability between the two entities.

[0044] Furthermore, to enable the knowledge graph to specifically guide the learning of audio representations, it is necessary to initialize the genre nodes G with the prediction scores of each audio segment for each genre, so that each node has input features x. v The remaining artist node A and instrument node I need to be initialized with zero vectors.

[0045] Furthermore, the GGNN network is used to iteratively update node features to learn the features of the knowledge graph.

[0046] Furthermore, the SE module assigns different attention weights to the initial audio representation and knowledge representation, adaptively determining which feature is more beneficial to the entire model. The fused audio representation is then input into the fully connected layer for music genre classification, thereby improving the accuracy of genre classification.

[0047] Furthermore, in order to better control the learning rate, cross-entropy loss L was chosen to train the music genre classification model, which can also avoid gradient vanishing during the training process.

[0048] It is understood that the beneficial effects of the second and third aspects mentioned above can be found in the relevant descriptions in the first aspect mentioned above, and will not be repeated here.

[0049] In summary, this invention constructs a knowledge graph related to music genres, inputs the information in it into a GGNN network in a structured form to learn the correlation between different genres, and initializes the nodes of the knowledge graph by obtaining the prediction scores of each audio for each genre through pre-training, thereby specifically enhancing the audio representation. Finally, the enhanced audio representation is used to obtain higher music genre classification performance.

[0050] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0051] Figure 1 This is a system framework diagram of the present invention;

[0052] Figure 2 This is a schematic diagram of the knowledge graph of the present invention;

[0053] Figure 3 A comparison chart with other classification methods;

[0054] Figure 4 This is a comparison chart showing the effects of having and not having a knowledge graph. Detailed Implementation

[0055] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0056] In the description of this invention, it should be understood that the terms "comprising" and "including" indicate the presence of the described features, integrals, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.

[0057] It should also be understood that the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the invention. As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.

[0058] It should also be further understood that the term "and / or" as used in this specification and the appended claims refers to any combination and all possible combinations of one or more of the associated listed items, and includes such combinations. For example, A and / or B can represent three cases: A alone, A and B simultaneously, and B alone. Additionally, the character " / " in this document generally indicates that the preceding and following objects have an "or" relationship.

[0059] It should be understood that although terms such as first, second, third, etc., may be used in the embodiments of the present invention to describe the preset range, these preset ranges should not be limited to these terms. These terms are only used to distinguish the preset ranges from one another. For example, without departing from the scope of the embodiments of the present invention, the first preset range may also be referred to as the second preset range, and similarly, the second preset range may also be referred to as the first preset range.

[0060] Depending on the context, the word "if" as used here can be interpreted as "when," "when," "in response to determination," or "in response to detection." Similarly, depending on the context, the phrase "if determination" or "if detection (of the stated condition or event)" can be interpreted as "when determination," "in response to determination," "when detection (of the stated condition or event)," or "in response to detection (of the stated condition or event)."

[0061] The accompanying drawings illustrate various structural schematic diagrams according to embodiments disclosed in this invention. These drawings are not to scale, and some details have been enlarged for clarity, and some details may have been omitted. The shapes of the various regions and layers shown in the drawings, as well as their relative sizes and positional relationships, are merely exemplary and may deviate from reality due to manufacturing tolerances or technical limitations. Furthermore, those skilled in the art can design regions / layers with different shapes, sizes, and relative positions as needed.

[0062] This invention provides a music genre classification method that integrates knowledge graphs. First, audio data is converted into a Mel spectrogram, which is then input into an audio feature extraction network to learn audio representations. Simultaneously, a linear layer is added at the end of the audio feature extraction network to obtain a prediction score for each genre, used to initialize genre nodes in the knowledge graph. To obtain the relationships between genres, this invention constructs a genre-related music knowledge graph using metadata from a publicly available audio dataset and uses a graph neural network to learn the knowledge representation of the graph. This knowledge representation is then used to further enhance the initial audio representation and applied to the genre classification task. This invention fully utilizes the correlations between different genres learned from the knowledge graph, enhancing audio representations and thus improving the accuracy of music genre classification, showing broad application prospects.

[0063] Please see Figure 1 This invention provides a method for classifying music genres using knowledge graphs, comprising the following steps:

[0064] S1. Obtain the audio data of the music from the dataset, crop the audio and convert it into a Mel spectrogram; then input the obtained Mel spectrogram into the audio feature extraction network to learn the audio representation; at the same time, add a linear layer at the end of the audio feature extraction network to obtain the prediction score for each genre, which is used to initialize the genre nodes of the knowledge graph.

[0065] Please see Figure 1 The audio representation module, located in the lower left corner, is as follows:

[0066] S101. Audio data is selected from the FMA-medium subset of the open-source music dataset FMA, which contains 25,000 tracks with a duration of 30 seconds, belonging to 16 imbalanced genres. Each input audio segment χ is cropped into 30 non-overlapping segments with a duration of 1 second, and the cropped audio samples are converted into 128-dimensional Mel spectrograms using the librosa library to obtain their time-frequency representation S.

[0067] S102. Input the preprocessed audio data into an Inception-ResNet-V2 network with reduced layers f(.) to learn the audio representation Z. a Multi-scale features of audio were obtained while reducing computational complexity;

[0068] S103. Add a linear layer g(.) after Inception-ResNet-V2, and obtain the vector Z by pre-training the network g○f. s It represents the network's prediction score for each genre, Z. s The dimension is equal to the number of schools of thought, i.e., 16 dimensions; the model is trained using the Adam optimizer with a learning rate of 0.001.

[0069] S2. Use the metadata in the dataset to build a knowledge graph related to different schools of thought, which is used to represent the relationships between different schools of thought;

[0070] Please see Figure 2 The image shows a partial visualization of the knowledge graph. (Constructing a knowledge graph of music genres) This is used to represent the relationships between schools of thought, including entity sets and edge sets.

[0071] Entity set V contains G+A+I elements, where G represents the number of music genres, A represents the number of artists, and I represents the number of instruments. Edge set E is located in the knowledge graph. The term "edge" refers to the set of edges connecting various entities. There are two types of edges in the knowledge graph constructed in this invention.

[0072] One type of edge connects an artist and a music genre, representing the probability that an artist has songs from a specific music genre. This probability can be calculated statistically and is denoted as P. G×A .

[0073] The specific calculation formula is as follows:

[0074]

[0075] Where i represents an artist, j represents a genre, and N i It is the total number of songs owned by artist i. It is the number of songs belonging to genre j among these songs.

[0076] Another type of edge connects an instrument and a music genre, representing the probability that a song played on a particular instrument belongs to a certain genre. This probability can be obtained from the openMIC-2018 dataset and is denoted as P. G×I .

[0077] knowledge graph The edge set E in the equation is represented as:

[0078]

[0079] When there is no connection between two nodes, the edge between them is represented as a zero matrix, for example, 0. G×G or 0 A×I .

[0080] S3. Use the predicted scores of each school of thought obtained in step S1 to initialize the school of thought nodes in the knowledge graph constructed in step S2. Then use a graph neural network to learn the feature vector of each node and concatenate all the feature vectors together to obtain the final knowledge representation.

[0081] First, the prediction scores of each genre obtained from the pre-trained network are used to analyze the knowledge graph. The system initializes the knowledge representation nodes, then uses a GGNN network to learn the feature vectors of each node, and concatenates all the feature vectors together to obtain the final knowledge representation Z. KG The final output knowledge representation has a dimension of 1536; please refer to [link / reference]. Figure 1 The knowledge representation module, located in the lower right corner, is as follows:

[0082] S301. Initialize artist node A and instrument node I using the zero vector, and initialize the corresponding genre node G using the predicted score Zs. The score Zs is obtained through a pre-trained network. What is obtained can be represented here as:

[0083] Zs = {s1, s2, ..., s} G}

[0084] Furthermore, after initialization, the input features of each node can be represented as:

[0085]

[0086] Among them, 0 G-1 and 0 A+I These represent the zero vectors of dimensions G-1 and A+I, respectively;

[0087] S302. Using GGNN network to learn knowledge graphs The characterization of Z KG .

[0088] GGNN is a recurrent neural network architecture that learns features of arbitrary graph structure data by iteratively updating node features.

[0089] Furthermore, at iteration t, the hidden state h of node i is... i Determined by its final state and the messages propagated from its neighbors, it is represented as:

[0090]

[0091]

[0092] Among them, E i It is a matrix representing the connection relationships between node i and its neighboring nodes.

[0093] After T iterations, where the iteration count T is set to 5, the message will be displayed throughout the entire graph. The knowledge graph is propagated through a series of layers to obtain the final hidden states of all nodes. The final linear layer outputs the final features of each node, and concatenating these features yields the entire knowledge graph. The characterization of Z KG .

[0094] S4. Using the SE (Squeeze and Excitation) module, different attention weights are assigned to the audio representations obtained in step S1 and the knowledge representations obtained in step S3. Then, the weighted representations are concatenated to obtain the enhanced audio representations. The enhanced audio representations are then input into the fully connected layer for genre classification.

[0095] Please see Figure 1 The first part, the genre classification and fusion module, is as follows:

[0096] S401, Representing Knowledge Z KG With the initial audio representation Z a Both features are input into the SE (Squeeze and Excitation) module, which assigns different attention weights to the two representations, adaptively determining which feature is beneficial to the overall model. The weighted features are then concatenated to obtain the final enhanced audio representation F, denoted as:

[0097]

[0098] Among them, W a and W KG The SE block represents the initial audio representation Z. a and knowledge representation Z KG The attention weights assigned;

[0099] S402. Input the enhanced audio representation F into a fully connected (FC) layer for genre classification. Use cross-entropy loss to train the model.

[0100] The loss function is defined as:

[0101]

[0102] in, It is a prediction of genre tags. It's a real label.

[0103] In another embodiment of the present invention, a music genre classification system integrating knowledge graphs is provided. This system can be used to implement the above-mentioned music genre classification method integrating knowledge graphs. Specifically, the music genre classification system integrating knowledge graphs includes a learning module, a construction module, a representation module, and a classification module.

[0104] The learning module converts audio data into Mel spectrograms, inputs the Mel spectrograms into the audio feature extraction network to learn audio representations, and adds a linear layer at the end of the audio feature extraction network to obtain a prediction score for each genre.

[0105] Build modules to construct knowledge graphs related to different genres;

[0106] The representation module initializes the faction nodes in the knowledge graph constructed by the building module with the prediction scores of each faction obtained by the learning module. Then, it uses a graph neural network to learn the feature vector of each faction node and concatenates all feature vectors to obtain the final knowledge representation.

[0107] The classification module uses the SE module to assign different attention weights to the audio representations obtained by the learning module and the knowledge representations obtained by the representation module. Then, the weighted representations are concatenated to obtain the enhanced audio representations. The enhanced audio representations are then input into the fully connected layer to form a music genre classification model, thereby achieving music genre classification.

[0108] In another embodiment of the present invention, a terminal device is provided, comprising a processor and a memory. The memory stores a computer program, the computer program including program instructions, and the processor executes the program instructions stored in the computer storage medium. The processor may be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. It is the computing and control core of the terminal, suitable for implementing one or more instructions, specifically suitable for loading and executing one or more instructions to achieve a corresponding method flow or corresponding function. The processor described in this embodiment of the present invention can be used for the operation of a music genre classification method that integrates knowledge graphs, including:

[0109] Audio data is converted into Mel spectrograms, which are then input into an audio feature extraction network to learn audio representations. A linear layer is added at the end of the audio feature extraction network to obtain prediction scores for each genre. A knowledge graph related to each genre is constructed. Genre nodes in the knowledge graph are initialized using the prediction scores for each genre, and then a graph neural network is used to learn the feature vectors of each genre node. All feature vectors are concatenated to obtain the final knowledge representation. A SE module is used to assign different attention weights to the audio representation and the knowledge representation, and then the weighted representations are concatenated to obtain an enhanced audio representation. The enhanced audio representation is then input into a fully connected layer to construct a music genre classification model, achieving music genre classification.

[0110] In another embodiment of the present invention, a storage medium is also provided, specifically a computer-readable storage medium (memory). This computer-readable storage medium is a memory device in a terminal device used to store programs and data. It is understood that the computer-readable storage medium here can include both the built-in storage medium in the terminal device and extended storage media supported by the terminal device. The computer-readable storage medium provides storage space that stores the terminal's operating system. Furthermore, this storage space also stores one or more instructions suitable for loading and execution by a processor. These instructions can be one or more computer programs (including program code). It should be noted that the computer-readable storage medium here can be high-speed RAM or non-volatile memory, such as at least one disk storage device.

[0111] One or more instructions stored in a computer-readable storage medium can be loaded and executed by a processor to implement the corresponding steps of the music genre classification method related to the fusion knowledge graph in the above embodiments; one or more instructions in the computer-readable storage medium are loaded by the processor and executed as follows:

[0112] Audio data is converted into Mel spectrograms, which are then input into an audio feature extraction network to learn audio representations. A linear layer is added at the end of the audio feature extraction network to obtain prediction scores for each genre. A knowledge graph related to each genre is constructed. Genre nodes in the knowledge graph are initialized using the prediction scores for each genre, and then a graph neural network is used to learn the feature vectors of each genre node. All feature vectors are concatenated to obtain the final knowledge representation. A SE module is used to assign different attention weights to the audio representation and the knowledge representation, and then the weighted representations are concatenated to obtain an enhanced audio representation. The enhanced audio representation is then input into a fully connected layer to construct a music genre classification model, achieving music genre classification.

[0113] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0114] This invention follows the setup of the FMA-medium dataset, using 80% of the data for model training, 10% for validation, and the remaining 10% for testing. To obtain better prediction results, during testing, this invention averages the softmax results of 30 one-second clips of an audio file to obtain the final prediction result.

[0115] Please see Figure 3 To demonstrate the improved accuracy of this invention in genre classification, it was compared with other related works, achieving an accuracy of 68.07%, surpassing all previous state-of-the-art methods; the ROC-AUC and PR-AUC were 0.883 and 0.471, respectively, which are 0.5% and 14.6% higher than the CLMR system. These results well demonstrate the effectiveness of this invention compared to existing methods.

[0116] This invention also verifies the contribution of knowledge graphs to audio feature representation learning:

[0117] Please see Figure 4 Within the framework of this invention, the genre classification accuracy obtained by audio feature learning guided by knowledge graphs is improved by 3%, while the genre classification accuracy obtained by using the knowledge graphs in this invention to enhance the audio representations in the other two baseline networks is improved by 2.3% and 9.34%, respectively.

[0118] The results above demonstrate that the framework of this invention supports the use of knowledge graphs to guide audio representation learning, which can facilitate fine-grained genre classification.

[0119] In summary, this invention provides a music genre classification method and system that integrates knowledge graphs. It utilizes knowledge graphs to guide audio representation learning, eliminating the need to acquire additional information for each audio segment, thus saving significant manpower and time costs associated with data acquisition and processing. Furthermore, this invention classifies music genres using enhanced audio representations, fully leveraging the correlations between different genres learned from the knowledge graph, thereby enhancing audio representations and improving the accuracy of music genre classification. This invention has broad application prospects.

[0120] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0121] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0122] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0123] In the embodiments provided by this invention, it should be understood that the disclosed devices / terminals and methods can be implemented in other ways. For example, the device / terminal embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0124] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0125] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0126] If the integrated module / unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium can be appropriately added or removed according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electrical carrier signals and telecommunication signals.

[0127] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0128] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0129] These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable apparatus for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0130] The above content is only for illustrating the technical concept of the present invention and should not be construed as limiting the scope of protection of the present invention. Any modifications made to the technical solution based on the technical concept proposed in this invention shall fall within the scope of protection of the claims of this invention.

Claims

1. A music genre classification method integrating knowledge graphs, characterized in that, Includes the following steps: S1. Convert the audio data into a Mel spectrogram, input the Mel spectrogram into the audio feature extraction network to learn the audio representation, and add a linear layer at the end of the audio feature extraction network to obtain the prediction score for each genre. S2. Construct a knowledge graph related to different genres. It includes an entity set and an edge set; the entity set Include One element, It refers to the number of music genres. It is the number of artists. It is the number of musical instruments; edge set In knowledge graph The middle part is the set of edges connecting various entities; a knowledge graph is a collection of edges. edge set in for: in, It is the size of The zero matrix, It is the size of The probability matrix relating music genres to artists It is the size of The probability matrix relating music genres to musical instruments. It is the size of The probability matrix relating artists to music genres. It is the size of The zero matrix, It is the size of The zero matrix, It is the size of The probability matrix relating musical instruments to musical genres. For size The zero matrix, It is the size of The zero matrix; S3. Initialize the category nodes in the knowledge graph constructed in step S2 using the prediction scores of each category obtained in step S1. Then, use a graph neural network to learn the feature vector of each category node, and concatenate all feature vectors to obtain the final knowledge representation, specifically: S301. Initialize the artist node using a zero vector. and instrument nodes Using the results obtained in step S1 Initialize the corresponding genre node After initialization, the input features of each node are obtained. ; S302, in one iteration At that time, node Hidden state Determined by the previous state and the messages transmitted from its neighbors, after... After the next iteration, the message is in the entire graph. The knowledge graph is propagated through a series of layers to obtain the final hidden states of all nodes. The final linear layer outputs the final features of each node, and these features are concatenated to obtain the entire knowledge graph. The representation ; S4. Using the SE module, different attention weights are assigned to the audio representation obtained in step S1 and the knowledge representation obtained in step S3. Then, the weighted representations are concatenated to obtain the enhanced audio representation. The enhanced audio representation is input into the fully connected layer to form a music genre classification model to achieve music genre classification.

2. The music genre classification method integrating knowledge graphs according to claim 1, characterized in that, Step S1 is as follows: S101, Audio The audio clips were trimmed into multiple non-overlapping segments, each 1 second in length, and then converted into 128-dimensional Mel spectrograms using the librosa library to obtain a time-frequency representation. ; S102, The time-frequency representation obtained in step S101 is... Input backbone network Learning audio representations ; S103, the backbone network in step S102 Add a linear layer later Through the network Pre-training to obtain a dimensional vector , representing the network's prediction score for each genre. The number of schools of thought.

3. The music genre classification method integrating knowledge graphs according to claim 2, characterized in that, In step S102, the backbone network The architecture used is Inception-ResNet-V2.

4. The music genre classification method integrating knowledge graphs according to claim 1, characterized in that, In step S302, during one iteration At that time, node Hidden state Its final state is determined by the messages propagated from its neighbors, specifically: in, It is a node The initial hidden state, It is a node Input features, It is the first Node at the next iteration The hidden state, It is the first Node at the next iteration The hidden state, where K is the total number of nodes in the knowledge graph. It is a representation of a node A matrix showing the connection relationships between its adjacent nodes.

5. The music genre classification method integrating knowledge graphs according to claim 1, characterized in that, Step S4 is as follows: S401. Representing Knowledge With initial audio representation The knowledge is represented by the SE module. With initial audio representation Different attention weights are assigned to adaptively determine features; The weighted features are then concatenated to obtain the audio representation. ; S402, The audio representation obtained in step S401 is... Input the music genre classification model to classify genres.

6. The music genre classification method integrating knowledge graphs according to claim 5, characterized in that, In step S402, the music genre classification model is trained using cross-entropy loss. for: in, It is a prediction of genre tags. It's a real label. It represents the total number of input audio samples.

7. A music genre classification system integrating knowledge graphs, characterized in that, include: The learning module converts audio data into Mel spectrograms, inputs the Mel spectrograms into the audio feature extraction network to learn audio representations, and adds a linear layer at the end of the audio feature extraction network to obtain prediction scores for each genre. The building module constructs a knowledge graph related to different genres. It includes an entity set and an edge set; the entity set Include One element, It refers to the number of music genres. It is the number of artists. It is the number of musical instruments; edge set In knowledge graph The middle part is the set of edges connecting various entities; a knowledge graph is a collection of edges. edge set in for: in, It is the size of The zero matrix, It is the size of The probability matrix relating music genres to artists It is the size of The probability matrix relating music genres to musical instruments. It is the size of The probability matrix relating artists to music genres. It is the size of The zero matrix, It is the size of The zero matrix, It is the size of The probability matrix relating musical instruments to musical genres. For size The zero matrix, It is the size of The zero matrix; The representation module initializes the category nodes in the knowledge graph constructed by the building module using the prediction scores of each category obtained from the learning module. Then, it uses a graph neural network to learn the feature vector of each category node and concatenates all feature vectors to obtain the final knowledge representation. Specifically: Initialize the artist node using a zero vector. and instrument nodes , using the obtained Initialize the corresponding genre node After initialization, the input features of each node are obtained. In one iteration At that time, node Hidden state Determined by the previous state and the messages transmitted from its neighbors, after... After the next iteration, the message is in the entire graph. The knowledge graph is propagated through a series of layers to obtain the final hidden states of all nodes. The final linear layer outputs the final features of each node, and these features are concatenated to obtain the entire knowledge graph. The representation ; The classification module uses the SE module to assign different attention weights to the audio representations obtained by the learning module and the knowledge representations obtained by the representation module. Then, the weighted representations are concatenated to obtain the enhanced audio representations. The enhanced audio representations are then input into the fully connected layer to form a music genre classification model, thereby achieving music genre classification.

Images