Model training method, emotion recognition method, device, server and storage medium

Abstract

The model training method provided in the embodiments of the present application utilizes a plurality of different preset unlabeled audio data sets to perform self-supervised learning training on a music emotion recognition model, updates the music emotion recognition model, so that the music emotion recognition model can learn more potential feature representations, then further trains the updated music emotion recognition model according to a preset labeled training sample set, and comprehensively considers shallow emotion features and deep emotion features in the training process, so that the target music emotion recognition model obtained through further training can more accurately recognize emotion information of music audio, effectively improving the accuracy of the music emotion recognition model.

Description

Technical Field

[0001] This application relates to the field of music emotion recognition technology, and in particular to a model training method, emotion recognition method, device, server and storage medium. Background Technology

[0002] Music Emotion Recognition (MER) refers to the identification of emotional information contained in a given musical piece. Its input is an audio file, and its output is the category of emotion. MER is widely used in emotion-based music retrieval and recommendation, and has become an important topic and key task in the field of music information retrieval. Current MER methods mainly use labeled music sample data to train deep learning models, resulting in MER models capable of recognizing musical emotions.

[0003] However, training a high-accuracy music emotion recognition model requires a large amount of high-quality labeled music sample data. Emotion labeling of music sample data is usually done manually by labelers, which is costly. Furthermore, emotion is a very complex psychological activity; different people have different emotional thresholds, making emotion labeling of music tracks even more difficult and compromising the quality of the training data, thus affecting the accuracy of the music emotion recognition model. Therefore, improving the accuracy of music emotion recognition models is a pressing issue that needs to be addressed. Summary of the Invention

[0004] This application provides a model training method, an emotion recognition method, an apparatus, a server, and a storage medium, aiming to improve the accuracy of music emotion recognition models.

[0005] In a first aspect, embodiments of this application provide a model training method for training a music emotion recognition model, the music emotion recognition model comprising a first Transformer encoding layer, a second Transformer encoding layer, a first multi-head attention layer, a second multi-head attention layer, and an emotion classification layer, the method comprising:

[0006] The music emotion recognition model is trained using self-supervised learning based on multiple different preset audio datasets to update the music emotion recognition model. The audio data in the same preset audio dataset has the same attributes, while the audio data in different preset audio datasets has different attributes.

[0007] Select a training sample from a preset training sample set as the target training sample. The training sample includes music audio and labeled emotional information.

[0008] The music audio in the target training sample is encoded through the first Transformer encoding layer to obtain a first vector;

[0009] The first vector is encoded by the second Transformer encoding layer to obtain the second vector, and the first vector is processed by the first multi-head attention layer to obtain the third vector;

[0010] The second vector and the third vector are fused by the second multi-head attention layer to obtain the target vector, and the target vector is then classified by the sentiment classification layer to obtain the predicted sentiment information.

[0011] The model loss value is determined based on the labeled sentiment information and the predicted sentiment information;

[0012] When the model loss value is greater than the preset loss value, the parameters of the music emotion recognition model are updated, and the process returns to the step of selecting a training sample from the preset training sample set as the target training sample.

[0013] When the model loss value is less than or equal to the preset loss value, training of the music emotion recognition model is stopped, and the target music emotion recognition model is obtained.

[0014] Secondly, embodiments of this application also provide an emotion recognition method, including:

[0015] Obtain the target music track to be identified;

[0016] The target music track is processed by a target music emotion recognition model to obtain the emotion category of the target music track. The target music emotion recognition model is trained using the model training method described in the first aspect.

[0017] Thirdly, embodiments of this application also provide a model training apparatus for training a music emotion recognition model, the music emotion recognition model including a first Transformer encoding layer, a second Transformer encoding layer, a first multi-head attention layer, a second multi-head attention layer, and an emotion classification layer, the model training apparatus including:

[0018] The self-supervised learning module is used to train the music emotion recognition model in a self-supervised manner based on multiple different preset audio datasets in order to update the music emotion recognition model. The audio data in the same preset audio dataset has the same attributes, while the audio data in different preset audio datasets has different attributes.

[0019] The model training module is used for:

[0020] Select a training sample from a preset training sample set as the target training sample. The training sample includes music audio and labeled emotional information.

[0021] The music audio in the target training sample is encoded through the first Transformer encoding layer to obtain a first vector;

[0022] The first vector is encoded by the second Transformer encoding layer to obtain the second vector, and the first vector is processed by the first multi-head attention layer to obtain the third vector;

[0023] The second vector and the third vector are fused by the second multi-head attention layer to obtain the target vector, and the target vector is then classified by the sentiment classification layer to obtain the predicted sentiment information.

[0024] The model loss value is determined based on the labeled sentiment information and the predicted sentiment information;

[0025] When the model loss value is greater than the preset loss value, the parameters of the music emotion recognition model are updated, and the process returns to the step of selecting a training sample from the preset training sample set as the target training sample.

[0026] When the model loss value is less than or equal to the preset loss value, training of the music emotion recognition model is stopped, and the target music emotion recognition model is obtained.

[0027] Fourthly, embodiments of this application also provide a server, the server including a processor, a memory, and a computer program stored in the memory and executable by the processor, wherein when the computer program is executed by the processor, it implements the steps of the model training method as described in the first aspect or the emotion recognition method as described in the second aspect.

[0028] Fifthly, embodiments of this application also provide a storage medium for computer-readable storage, wherein a computer program is stored on the storage medium, and wherein the computer program, when executed by a processor, implements the steps of the model training method as described in the first aspect or the emotion recognition method as described in the second aspect.

[0029] This application provides a model training method, an emotion recognition method, an apparatus, a server, and a storage medium. The model training method uses unlabeled audio databases of different types to perform self-supervised learning training on a music emotion recognition model. This allows the music emotion recognition model to learn more latent feature representations without requiring the labeling of large amounts of training data, effectively reducing costs. Then, a labeled training sample set is used to further train the self-supervised music emotion recognition model, enabling the further trained target music emotion recognition model to more accurately identify the emotional information of music audio.

[0030] Meanwhile, during training, the first Transformer encoding layer and the first multi-head attention layer can mine vectors representing shallow emotional features of music audio. The first Transformer encoding layer combined with the second Transformer encoding layer can mine vectors representing deep emotional features of music audio. The second multi-head attention layer then fuses the vectors representing shallow and deep emotional features to obtain a target vector that comprehensively represents both shallow and deep emotional features. By performing emotional classification on the target vector through the emotional classification layer, more accurate emotional information can be obtained, making the determined model loss value more accurate and further improving the accuracy of the music emotion recognition model. Attached Figure Description

[0031] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0032] Figure 1 This is a schematic flowchart of a model training method provided in an embodiment of this application;

[0033] Figure 2 This is a schematic diagram of a network structure of the music emotion recognition model in an embodiment of this application;

[0034] Figure 3 This is another network structure diagram of the music emotion recognition model in the embodiments of this application;

[0035] Figure 4 This is another network structure diagram of the music emotion recognition model in the embodiments of this application;

[0036] Figure 5 This is another network structure diagram of the music emotion recognition model in the embodiments of this application;

[0037] Figure 6 This is a flowchart illustrating an emotion recognition method provided in an embodiment of this application;

[0038] Figure 7 This is a schematic block diagram of a model training device provided in an embodiment of this application;

[0039] Figure 8 This is a schematic block diagram of the structure of a server provided in an embodiment of this application.

[0040] The realization of the purpose, functional features and advantages of this application will be further explained in conjunction with the embodiments and the accompanying drawings. Detailed Implementation

[0041] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0042] The flowchart shown in the attached diagram is for illustrative purposes only and does not necessarily include all content and operations / steps, nor does it necessarily have to be performed in the order described. For example, some operations / steps can be broken down, combined, or partially merged, so the actual execution order may change depending on the actual situation.

[0043] Currently, to train a high-accuracy music emotion recognition model, a large amount of high-quality labeled music sample data is required. However, the emotion labeling of music sample data is usually done manually by labelers, which is costly. Furthermore, emotion is a very complex psychological activity, and different people have different emotional thresholds, which makes it more difficult to label the emotions of music tracks. This makes it impossible to guarantee the labeling quality of the training data, and thus the accuracy of the music emotion recognition model cannot be guaranteed.

[0044] To address the aforementioned issues, this application provides a model training method, an emotion recognition method, an apparatus, a server, and a storage medium. This model training method uses unlabeled audio databases of different types to perform self-supervised learning training on a music emotion recognition model. This allows the music emotion recognition model to learn more latent feature representations without requiring the labeling of large amounts of training data, effectively reducing costs. Then, a labeled training sample set is used to further train the self-supervised music emotion recognition model, enabling the further trained target music emotion recognition model to more accurately identify the emotional information in music audio.

[0045] Meanwhile, during training, the first Transformer encoding layer and the first multi-head attention layer can mine vectors representing shallow emotional features of music audio. The first Transformer encoding layer combined with the second Transformer encoding layer can mine vectors representing deep emotional features of music audio. The second multi-head attention layer then fuses the vectors representing shallow and deep emotional features to obtain a target vector that comprehensively represents both shallow and deep emotional features. By performing emotional classification on the target vector through the emotional classification layer, more accurate emotional information can be obtained, making the determined model loss value more accurate and further improving the accuracy of the music emotion recognition model.

[0046] This application embodiment can acquire and process relevant data based on artificial intelligence technology. Artificial intelligence (AI) is a theory, method, technology, and application system that uses digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results. This training method can be applied to a server. The server can be a standalone server or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms.

[0047] The following detailed description of some embodiments of this application is provided in conjunction with the accompanying drawings. Unless otherwise specified, the following embodiments and features can be combined with each other.

[0048] Please see Figure 1 , Figure 1 This is a flowchart illustrating a model training method provided in an embodiment of this application.

[0049] like Figure 1 As shown, the training method includes steps S101 to S108.

[0050] Step S101: Based on multiple different preset audio datasets, perform self-supervised learning training on the music emotion recognition model to update the music emotion recognition model. 。

[0051] In this embodiment, audio data within the same preset audio dataset has the same attributes, while audio data in different preset audio datasets has different attributes. The multiple different preset audio datasets include at least one music track dataset. For example, the multiple different preset audio datasets include multiple different music track datasets. The music tracks in the different music track datasets have different styles or languages. The styles of the music tracks can include pop music, jazz music, folk music, electronic music, Latin music, rap music, and classical music, etc. The languages ​​of the music tracks include Chinese (Mandarin), minority languages, local languages, and foreign languages, etc. Foreign languages ​​include English, German, Russian, French, Japanese, Korean, etc.

[0052] In some embodiments, the multiple different preset audio datasets include at least one music track dataset and at least one non-music track dataset. For example, the music track datasets in the multiple different preset audio datasets include the MSongsDB dataset, the unlabeled MusicNet dataset, the unlabeled FMA dataset, and the unlabeled MagnaTagATune dataset, while the non-music track datasets in the multiple different preset audio datasets include the Common Voice dataset and the TIMIT (The DARPA TIMIT Acoustic-Phonetic Continuous Speech Corpus) dataset, etc.

[0053] In some embodiments, such as Figure 2 As shown, the music emotion recognition model includes a first Transformer encoding layer 210, a second Transformer encoding layer 220, a first multi-head attention layer 230, a second multi-head attention layer 240, and an emotion classification layer 250. The first Transformer encoding layer 210 is connected to the second Transformer encoding layer 220 and the first multi-head attention layer 230, and the second multi-head attention layer 240 is connected to the first multi-head attention layer 230, the second Transformer encoding layer 220, and the emotion classification layer 250. For example, self-supervised learning training of the music emotion recognition model based on multiple different preset audio datasets may include: performing self-supervised learning training on the first Transformer encoding layer and the second Transformer encoding layer based on multiple different preset audio datasets.

[0054] In some embodiments, self-supervised learning training of the first Transformer encoding layer and the second Transformer encoding layer based on multiple different preset audio datasets may include: selecting a preset audio dataset as a target audio dataset from multiple preset audio datasets; selecting an audio data from the target audio dataset as target audio data; encoding the target audio data through the first Transformer encoding layer to obtain a first encoding vector, encoding the first encoding vector through the second Transformer encoding layer to obtain a first sample vector; masking the target audio data to obtain masked audio data, encoding the masked audio data through the first Transformer encoding layer to obtain a second encoding vector, and training the first sample vector through the second Transformer encoding layer... The two encoded vectors are encoded to obtain the second sample vector; the contrastive learning loss value is determined based on the first and second sample vectors; the parameters of the first and second Transformer encoding layers are updated based on the contrastive learning loss value; if there is unselected audio data in the target audio dataset, the process returns to the step of selecting one audio data from the target audio dataset as the target audio data; if all audio data in the target audio dataset has been selected once, the process returns to the step of selecting a preset audio dataset as the target audio dataset from multiple preset audio datasets; when all audio data in each preset audio dataset has been selected once or the contrastive learning loss value is minimized, the self-supervised learning training of the first and second Transformer encoding layers is stopped.

[0055] Step S102: Select a training sample from the preset training sample set as the target training sample. The training sample includes music audio and labeled emotional information.

[0056] In this embodiment, the music audio can be a complete music track or a part of a complete music track, and the labeled emotional information can include the labeled emotional category or the labeled valence and arousal.

[0057] Step S103: Encode the music audio in the target training sample through the first Transformer encoding layer to obtain the first vector.

[0058] In this embodiment, the first Transformer coding layer can encode a first vector representing shallow emotional features of the music audio. For example, Figure 3As shown, the first Transformer encoding layer 210 includes a feature encoder 211, a first position encoder 212 and a first Transformer encoder 213. The feature encoder 211 is connected to the first position encoder 212, the first position encoder 212 is connected to the first Transformer encoder 213, and the first Transformer encoder 213 is connected to the second Transformer encoding layer 220.

[0059] In some embodiments, encoding the music audio in the target training samples through a first Transformer encoding layer to obtain a first vector may include: encoding the emotional features of the music audio in the target training samples through a feature encoder to obtain an emotional feature vector; performing positional embedding encoding on the emotional feature vector through a first position encoder to obtain a new emotional feature vector; and encoding the emotional feature vector through a first Transformer encoder to obtain the first vector. The feature encoder includes a multi-layer convolutional network, for example, a 3-layer convolutional network. The first position encoder includes a single-layer convolutional network.

[0060] Step S104: Encode the first vector through the second Transformer encoding layer to obtain the second vector, and process the first vector through the first multi-head attention layer to obtain the third vector.

[0061] In this embodiment, the first Transformer coding layer has already encoded a first vector to represent the emotional features of the music audio. Then, by continuing to encode the first vector through the second Transformer coding layer, a second vector to represent deeper emotional features can be obtained.

[0062] In some embodiments, such as Figure 4 As shown, the second Transformer encoding layer 220 includes a second position encoder 221 and a second Transformer encoder 222. The second position encoder 221 is connected to the first Transformer encoder 213 and the second Transformer encoder 222, respectively. The second Transformer encoder 222 is connected to the second multi-head attention layer 240. Encoding the first vector through the second Transformer encoding layer to obtain the second vector may include: performing position embedding encoding on the first vector through the second position encoder to obtain a new first vector; and encoding the new first vector through the second Transformer encoder to obtain the second vector. This embodiment enhances the position embedding representation of the vector by adding a second position encoder.

[0063] In some embodiments, the weight parameters of the first multi-head attention layer include multiple weight matrix groups. Each weight matrix group includes a Query weight matrix, a Key weight matrix, and a Value weight matrix. Processing the first vector through the first multi-head attention layer to obtain a third vector may include: for each weight matrix group, calculating the product of the Query weight matrix and the first vector to obtain a Query vector; calculating the product of the Key weight matrix and the first vector to obtain a Key vector; calculating the product of the Value weight matrix and the first vector to obtain a Value vector; determining the attention weight matrix based on the Query vector and the Key vector, and calculating the product of the attention weight matrix and the Value vector to obtain an attention output vector; and fusing the attention output vectors corresponding to each weight matrix group to obtain the third vector.

[0064] Step S105: The second vector and the third vector are fused through the second multi-head attention layer to obtain the target vector. The target vector is then processed for sentiment classification through the sentiment classification layer to obtain the predicted sentiment information.

[0065] In this embodiment, the second vector is used to represent shallow emotional features, and the third vector is used to represent deep emotional features. By fusing the second and third vectors through the second multi-head attention layer, a target vector that comprehensively represents both shallow and deep emotional features can be obtained. Thus, by performing emotional classification processing on the target vector through the emotional classification layer, more accurate emotional information can be obtained.

[0066] In some embodiments, the weight parameters of the second multi-head attention layer include multiple weight matrix groups. The process of fusing the second vector and the third vector through the second multi-head attention layer to obtain the target vector may include: generating an attention output vector corresponding to each weight matrix group based on each weight matrix group, the second vector, and the third vector; and fusing the attention output vectors corresponding to each weight matrix group to obtain the target vector.

[0067] In some embodiments, a weight matrix group includes a Query weight matrix, a Key weight matrix, and a Value weight matrix. Determining the attention output vector corresponding to each weight matrix group based on the weight matrix group, the second vector, and the third vector may include: calculating the product of the Query weight matrix and the second vector to obtain the Query vector; calculating the product of the Key weight matrix and the third vector to obtain the Key vector; calculating the product of the Value weight matrix and the third vector to obtain the Value vector; determining the attention weight matrix based on the Query vector and the Key vector, and calculating the product of the attention weight matrix and the Value vector to obtain the attention output vector.

[0068] In some embodiments, determining the attention output vector corresponding to each of the weight matrix groups, the second vector, and the third vector, may include: calculating the product of the Query weight matrix with the second vector and the third vector respectively to obtain two different Query vectors; calculating the product of the Key weight matrix with the third vector to obtain the Key vector; calculating the product of the Value weight matrix with the third vector to obtain the Value vector; fusing the two different Query vectors to obtain the target Query vector; and determining the attention weight matrix based on the target Query vector and the Key vector, and calculating the product of the attention weight matrix with the Value vector to obtain the attention output vector.

[0069] In some embodiments, determining the attention output vector corresponding to each of the weight matrix groups, based on the weight matrix group, the second vector, and the third vector, may include: calculating the product of the Query weight matrix with the second vector and the third vector respectively to obtain two different Query vectors; calculating the product of the Key weight matrix with the second vector and the third vector respectively to obtain two different Key vectors; calculating the product of the Value weight matrix with the second vector and the third vector respectively to obtain two different Value vectors; fusing the two different Query vectors to obtain a target Query vector, fusing the two different Key vectors to obtain a target Key vector, fusing the two different Value vectors to obtain a target Value vector; and determining the attention weight matrix based on the target Query vector and the target Key vector, and calculating the product of the attention weight matrix and the target Value vector to obtain the attention output vector.

[0070] In some embodiments, such as Figure 5 As shown, the second multi-head attention layer 240 includes a first multi-head attention sub-layer 241 and a second multi-head attention sub-layer 242. The emotion classification layer 250 includes a valence classifier 251 and an arousal classifier 252. The second Transformer encoder 222 is connected to both the first and second multi-head attention sub-layers 241 and 242. The first multi-head attention layer 240 is connected to both the first and second multi-head attention sub-layers 241 and 242. The first multi-head attention sub-layer 241 is connected to the valence classifier 251, and the second multi-head attention sub-layer 242 is connected to the arousal classifier 252. The valence classifier 251 and the arousal classifier 252 have the same network structure and model parameters, but different classification labels. For example, both the valence classifier 251 and the arousal classifier 252 include a three-layer fully connected network and a softmax layer.

[0071] In some embodiments, fusing the second and third vectors through a second multi-head attention layer to obtain a target vector, and performing sentiment classification on the target vector through a sentiment classification layer to obtain predicted sentiment information may include: fusing the second and third vectors through a first multi-head attention sub-layer to obtain a first target vector, and fusing the second and third vectors through a second multi-head attention sub-layer to obtain a second target vector; performing valence classification on the first target vector through a valence classifier and arousal classification on the second target vector through an arousal classifier to obtain predicted sentiment information, wherein the predicted sentiment information includes predicted valence and arousal. It should be noted that the detailed implementation process of fusing the second and third vectors through the first and second multi-head attention sub-layers can be referred to the aforementioned specific implementation process of fusing the second and third vectors through the second multi-head attention sub-layer, and will not be repeated here.

[0072] Step S106: Determine the model loss value based on the labeled sentiment information and the predicted sentiment information.

[0073] In this embodiment, the cross-entropy loss function can be used to calculate the cross-entropy loss value based on the labeled sentiment information and the predicted sentiment information, thereby obtaining the model loss value.

[0074] Step S107: When the model loss value is greater than the preset loss value, update the parameters of the music emotion recognition model.

[0075] In this embodiment, when the model loss value is greater than the preset loss value, it can be determined that the music emotion recognition model has not converged and needs to continue training. At this time, the parameters of the music emotion recognition model can be updated based on the backpropagation algorithm and the model loss value, and the process returns to step S102 to select a training sample from the preset training sample set as the target training sample to continue training the updated music emotion recognition model.

[0076] Step S108: When the model loss value is less than or equal to the preset loss value, stop training the music emotion recognition model to obtain the target music emotion recognition model.

[0077] In this embodiment, when the model loss value is less than or equal to the preset loss value, it can be determined that the music emotion recognition model has converged. At this point, training of the music emotion recognition model is stopped, and the target music emotion recognition model is obtained. The model training method provided in this embodiment has low requirements for the annotation of training data and can also ensure the recognition accuracy of the target music emotion recognition model obtained through training.

[0078] Please see Figure 6 , Figure 6This is a flowchart illustrating an emotion recognition method provided in an embodiment of this application.

[0079] like Figure 6 As shown, the emotion recognition method includes steps S201 to S202.

[0080] Step S201: Obtain the target music track to be identified.

[0081] In this embodiment, the target music track to be identified can be any music track that requires emotional classification. For example, the target music track to be identified is a music track newly uploaded by the user.

[0082] Step S202: Perform music emotion recognition processing on the target music track using the target music emotion recognition model to obtain the emotion category of the target music track.

[0083] In this embodiment, the target music emotion recognition model is obtained by training the music emotion recognition model using the aforementioned model training method. The network structure of the target music emotion recognition model can be... Figures 2 to 5 This embodiment does not specifically limit the network structure used in this example. This embodiment can accurately determine the emotional category of a target music track through a high-precision target music emotion recognition model.

[0084] In some embodiments, performing music emotion recognition processing on a target music track using a target music emotion recognition model to obtain the emotion category of the target music track may include: encoding the target music track through a first Transformer encoding layer in the target music emotion recognition model to obtain a first encoding vector; encoding the first encoding vector through a second Transformer encoding layer in the target music emotion recognition model to obtain a second encoding vector; processing the first encoding vector through a first multi-head attention layer in the target music emotion recognition model to obtain a third encoding vector; fusing the second and third encoding vectors through a second multi-head attention layer in the target music emotion recognition model to obtain a target encoding vector; and performing emotion classification processing on the target encoding vector through an emotion classification layer in the target music emotion recognition model to obtain the emotion category of the target music track.

[0085] In some embodiments, the weight parameters of the second multi-head attention layer in the target music emotion recognition model include multiple target weight matrix groups. The second multi-head attention layer in the target music emotion recognition model can be used to fuse the second encoding vector and the third encoding vector to obtain the target encoding vector. This can include: generating an attention output vector corresponding to each target weight matrix group based on each target weight matrix group, the second encoding vector, and the third encoding vector; and fusing the attention output vectors corresponding to each target weight matrix group to obtain the target encoding vector.

[0086] In some embodiments, a target weight matrix group includes a target Query weight matrix, a target Key weight matrix, and a target Value weight matrix. Generating the attention output vector corresponding to each target weight matrix group based on each target weight matrix group, the second encoding vector, and the third encoding vector may include: calculating the product of the target Query weight matrix and the second encoding vector to obtain the Query vector; calculating the product of the target Key weight matrix and the third encoding vector to obtain the Key vector; calculating the product of the target Value weight matrix and the third encoding vector to obtain the Value vector; determining the attention weight matrix based on the Query vector and the Key vector, and calculating the product of the attention weight matrix and the Value vector to obtain the attention output vector.

[0087] Please see Figure 7 , Figure 7 This is a schematic block diagram of a model training device provided in an embodiment of this application.

[0088] like Figure 7 As shown, the model training device 300 includes:

[0089] The self-supervised learning module 310 is used to perform self-supervised learning training on the music emotion recognition model based on multiple different preset audio datasets to update the music emotion recognition model. The audio data in the same preset audio dataset has the same attributes, while the audio data in different preset audio datasets has different attributes.

[0090] Model training module 320 is used for:

[0091] Select a training sample from a preset training sample set as the target training sample. The training sample includes music audio and labeled emotional information.

[0092] The music audio in the target training sample is encoded through the first Transformer encoding layer to obtain a first vector;

[0093] The first vector is encoded by the second Transformer encoding layer to obtain the second vector, and the first vector is processed by the first multi-head attention layer to obtain the third vector;

[0094] The second vector and the third vector are processed by the second multi-head attention layer to obtain the target vector, and the target vector is processed by the sentiment classification layer to obtain the predicted sentiment information.

[0095] The model loss value is determined based on the labeled sentiment information and the predicted sentiment information;

[0096] When the model loss value is greater than the preset loss value, the parameters of the music emotion recognition model are updated, and the process returns to the step of selecting a training sample from the preset training sample set as the target training sample.

[0097] When the model loss value is less than or equal to the preset loss value, training of the music emotion recognition model is stopped, and the target music emotion recognition model is obtained.

[0098] In some embodiments, the weight parameters of the second multi-head attention layer include multiple weight matrix groups, and the model training module 320 is further configured to:

[0099] Based on each of the weight matrix groups, the second vector, and the third vector, generate an attention output vector corresponding to each of the weight matrix groups;

[0100] The attention output vectors corresponding to each set of weight matrices are fused to obtain the target vector.

[0101] In some embodiments, one of the weight matrix groups includes a Query weight matrix, a Key weight matrix, and a Value weight matrix, and the model training module 320 is further configured to:

[0102] Calculate the product of the Query weight matrix and the second vector to obtain the Query vector;

[0103] Calculate the product of the Key weight matrix and the third vector to obtain the Key vector; calculate the product of the Value weight matrix and the third vector to obtain the Value vector.

[0104] Based on the Query vector and the Key vector, an attention weight matrix is ​​determined, and the product of the attention weight matrix and the Value vector is calculated to obtain the attention output vector.

[0105] In some embodiments, the second multi-head attention layer includes a first multi-head attention sub-layer and a second multi-head attention sub-layer, the emotion classification layer includes a valence classifier and an arousal classifier, and the model training module 320 is further configured to:

[0106] The first target vector is obtained by fusing the second vector and the third vector through the first multi-head attention sublayer, and the second target vector is obtained by fusing the second vector and the third vector through the second multi-head attention sublayer.

[0107] The predicted sentiment information is obtained by classifying the first target vector by the valence classifier and the second target vector by the arousal classifier.

[0108] In some embodiments, the first Transformer encoding layer includes a feature encoder, a first position encoder, and a first Transformer encoder, and the model training module 320 is further configured to:

[0109] The emotional features of the music audio in the target training samples are encoded by the feature encoder, resulting in an emotional feature vector.

[0110] The first position encoder performs position embedding encoding on the emotion feature vector to obtain a new emotion feature vector;

[0111] The first vector is obtained by encoding the emotion feature vector using the first Transformer encoder.

[0112] In some embodiments, the second Transformer encoding layer includes a second position encoder and a second Transformer encoder, and the model training module 320 is further configured to:

[0113] The first vector is then subjected to position embedding encoding by the second position encoder to obtain a new first vector;

[0114] The new first vector is encoded by the second Transformer encoder to obtain the second vector.

[0115] It should be noted that those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the above-described device and its modules and units can be referred to the corresponding processes in the aforementioned model training method embodiments, and will not be repeated here.

[0116] The apparatus provided in the above embodiments can be implemented as a computer program, which can be used in, for example... Figure 8It is running on the server shown.

[0117] Please see Figure 8 , Figure 8 This is a schematic block diagram of the structure of a server provided in an embodiment of this application.

[0118] like Figure 8 As shown, the server includes a processor, memory, and network interface connected via a system bus, wherein the memory may include storage media and internal memory.

[0119] The storage medium may store the operating system and computer programs. The computer programs include program instructions that, when executed, cause the processor to perform any model training method or emotion recognition method.

[0120] The processor provides computing and control capabilities to support the operation of the entire server.

[0121] This network interface is used for network communication, such as sending assigned tasks. Those skilled in the art will understand that... Figure 8 The structure shown is merely a block diagram of a portion of the structure related to the solution of this application and does not constitute a limitation on the server to which the solution of this application is applied. A specific server may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0122] It should be understood that the processor can be a Central Processing Unit (CPU), but it can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among these, a general-purpose processor can be a microprocessor or any conventional processor.

[0123] In one embodiment, the processor is used to run a computer program stored in memory to perform the following steps:

[0124] The music emotion recognition model is trained using self-supervised learning based on multiple different preset audio datasets to update the music emotion recognition model. The audio data in the same preset audio dataset has the same attributes, while the audio data in different preset audio datasets has different attributes.

[0125] Select a training sample from a preset training sample set as the target training sample. The training sample includes music audio and labeled emotional information.

[0126] The music audio in the target training sample is encoded through the first Transformer encoding layer to obtain a first vector;

[0127] The first vector is encoded by the second Transformer encoding layer to obtain the second vector, and the first vector is processed by the first multi-head attention layer to obtain the third vector;

[0128] The second vector and the third vector are fused by the second multi-head attention layer to obtain the target vector, and the target vector is then classified by the sentiment classification layer to obtain the predicted sentiment information.

[0129] The model loss value is determined based on the labeled sentiment information and the predicted sentiment information;

[0130] When the model loss value is greater than the preset loss value, the parameters of the music emotion recognition model are updated, and the process returns to the step of selecting a training sample from the preset training sample set as the target training sample.

[0131] When the model loss value is less than or equal to the preset loss value, training of the music emotion recognition model is stopped, and the target music emotion recognition model is obtained.

[0132] In some embodiments, the weight parameters of the second multi-head attention layer include multiple weight matrix groups, and the processor, when performing fusion processing on the second vector and the third vector through the second multi-head attention layer to obtain the target vector, is used to:

[0133] Based on each of the weight matrix groups, the second vector, and the third vector, generate an attention output vector corresponding to each of the weight matrix groups;

[0134] The attention output vectors corresponding to each set of weight matrices are fused to obtain the target vector.

[0135] In some embodiments, a weight matrix group includes a query weight matrix, a key weight matrix, and a value weight matrix. When the processor generates an attention output vector corresponding to each weight matrix group based on each weight matrix group, the second vector, and the third vector, it is configured to:

[0136] Calculate the product of the Query weight matrix and the second vector to obtain the Query vector;

[0137] Calculate the product of the Key weight matrix and the third vector to obtain the Key vector; calculate the product of the Value weight matrix and the third vector to obtain the Value vector.

[0138] Based on the Query vector and the Key vector, an attention weight matrix is ​​determined, and the product of the attention weight matrix and the Value vector is calculated to obtain the attention output vector.

[0139] In some embodiments, the second multi-head attention layer includes a first multi-head attention sub-layer and a second multi-head attention sub-layer, and the sentiment classification layer includes a valence classifier and an arousal classifier. When the processor processes the second vector and the third vector through the second multi-head attention layer to obtain a target vector, and performs sentiment classification processing on the target vector through the sentiment classification layer to obtain predicted sentiment information, it is configured to:

[0140] The first target vector is obtained by fusing the second vector and the third vector through the first multi-head attention sublayer, and the second target vector is obtained by fusing the second vector and the third vector through the second multi-head attention sublayer.

[0141] The predicted sentiment information is obtained by classifying the first target vector by the valence classifier and the second target vector by the arousal classifier.

[0142] In some embodiments, the first Transformer encoding layer includes a feature encoder, a first position encoder, and a first Transformer encoder. When the processor encodes the music audio in the target training sample through the first Transformer encoding layer to obtain a first vector, it is used to:

[0143] The emotional features of the music audio in the target training samples are encoded by the feature encoder, resulting in an emotional feature vector.

[0144] The first position encoder performs position embedding encoding on the emotion feature vector to obtain a new emotion feature vector;

[0145] The first vector is obtained by encoding the emotion feature vector using the first Transformer encoder.

[0146] In some embodiments, the second Transformer encoding layer includes a second position encoder and a second Transformer encoder. When the processor encodes the first vector through the second Transformer encoding layer to obtain the second vector, it is configured to:

[0147] The first vector is then subjected to position embedding encoding by the second position encoder to obtain a new first vector;

[0148] The new first vector is encoded by the second Transformer encoder to obtain the second vector.

[0149] In some embodiments, the processor is further configured to implement:

[0150] Obtain the target music track to be identified;

[0151] The target music track is processed by a target music emotion recognition model to obtain the emotion category of the target music track.

[0152] It should be noted that those skilled in the art will understand that, for the sake of convenience and brevity, the specific working process of the server described above can be referred to the corresponding process in the aforementioned model training method or emotion recognition method embodiments, and will not be repeated here.

[0153] As can be seen from the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a server (which may be a personal computer, a server, or a network server, etc.) to execute the methods described in various embodiments or some parts of the embodiments of this application.

[0154] This application also provides a storage medium storing a computer program, which includes program instructions. The method implemented when the program instructions are executed can be found in various embodiments of the model training method or emotion recognition method of this application.

[0155] The storage medium can be volatile or non-volatile. It can be an internal storage unit of the server as described in the foregoing embodiments, such as the server's hard drive or memory. Alternatively, it can be an external storage server, such as a plug-in hard drive, Smart Media Card (SMC), Secure Digital (SD) card, or Flash Card.

[0156] Furthermore, the storage medium may mainly include a program storage area and a data storage area, wherein the program storage area may store the operating system, applications required for at least one function, etc.; and the data storage area may store data created based on the use of blockchain nodes, etc.

[0157] The blockchain referred to in this application is a novel application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanisms, and encryption algorithms. Essentially, a blockchain is a decentralized database, a chain of data blocks linked together using cryptographic methods. Each data block contains information about a batch of network transactions, used to verify the validity of the information (anti-counterfeiting) and generate the next block. A blockchain can include an underlying blockchain platform, a platform product service layer, and an application service layer.

[0158] It should be understood that the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the scope of the application. As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.

[0159] It should also be understood that the term "and / or" as used in this specification and the appended claims refers to any combination and all possible combinations of one or more of the associated listed items, and includes such combinations. It should be noted that, herein, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or system. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or system that includes that element.

[0160] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments. The above descriptions are merely specific implementations of this application, but the scope of protection of this application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and these modifications or substitutions should all be covered within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A model training method for training a music emotion recognition model, characterized in that, The music emotion recognition model includes a first Transformer encoding layer, a second Transformer encoding layer, a first multi-head attention layer, a second multi-head attention layer, and an emotion classification layer. The method includes: The music emotion recognition model is trained using self-supervised learning based on multiple different unlabeled preset audio datasets to update the music emotion recognition model. The audio data in the same preset audio dataset has the same attributes, while the audio data in different preset audio datasets has different attributes. Select a training sample from a pre-defined set of labeled training samples as the target training sample. The training sample includes music audio and labeled emotional information. The music audio in the target training sample is encoded through the first Transformer encoding layer to obtain a first vector; The first vector is encoded by the second Transformer encoding layer to obtain the second vector, and the first vector is processed by the first multi-head attention layer to obtain the third vector; The second vector and the third vector are fused by the second multi-head attention layer to obtain the target vector, and the target vector is then classified by the sentiment classification layer to obtain the predicted sentiment information. The model loss value is determined based on the labeled sentiment information and the predicted sentiment information; When the model loss value is greater than the preset loss value, the parameters of the music emotion recognition model are updated, and the process returns to the step of selecting a training sample from the preset training sample set as the target training sample. When the model loss value is less than or equal to the preset loss value, training of the music emotion recognition model is stopped, and the target music emotion recognition model is obtained.

2. The model training method according to claim 1, characterized in that, The weight parameters of the second multi-head attention layer include multiple weight matrix groups. The step of fusing the second vector and the third vector through the second multi-head attention layer to obtain the target vector includes: Based on each of the weight matrix groups, the second vector, and the third vector, generate an attention output vector corresponding to each of the weight matrix groups; The attention output vectors corresponding to each set of weight matrices are fused to obtain the target vector.

3. The model training method according to claim 2, characterized in that, A weight matrix group includes a query weight matrix, a key weight matrix, and a value weight matrix. Based on the weight matrix group, the second vector, and the third vector, attention output vectors corresponding to each weight matrix group are generated, including: Calculate the product of the Query weight matrix and the second vector to obtain the Query vector; Calculate the product of the Key weight matrix and the third vector to obtain the Key vector; calculate the product of the Value weight matrix and the third vector to obtain the Value vector. Based on the Query vector and the Key vector, an attention weight matrix is ​​determined, and the product of the attention weight matrix and the Value vector is calculated to obtain the attention output vector.

4. The model training method according to claim 1, characterized in that, The second multi-head attention layer includes a first multi-head attention sub-layer and a second multi-head attention sub-layer. The sentiment classification layer includes a valence classifier and an arousal classifier. The second multi-head attention layer processes the second vector and the third vector to obtain a target vector. The sentiment classification layer then performs sentiment classification processing on the target vector to obtain predicted sentiment information, including: The first target vector is obtained by fusing the second vector and the third vector through the first multi-head attention sublayer, and the second target vector is obtained by fusing the second vector and the third vector through the second multi-head attention sublayer. The predicted sentiment information is obtained by classifying the first target vector by the valence classifier and the second target vector by the arousal classifier.

5. The model training method according to any one of claims 1-4, characterized in that, The first Transformer encoding layer includes a feature encoder, a first position encoder, and a first Transformer encoder. Encoding the music audio in the target training samples through the first Transformer encoding layer to obtain a first vector includes: The emotional features of the music audio in the target training samples are encoded by the feature encoder, resulting in an emotional feature vector. The first position encoder performs position embedding encoding on the emotion feature vector to obtain a new emotion feature vector; The first vector is obtained by encoding the emotion feature vector using the first Transformer encoder.

6. The model training method according to claim 5, characterized in that, The second Transformer encoding layer includes a second position encoder and a second Transformer encoder. Encoding the first vector through the second Transformer encoding layer to obtain the second vector includes: The first vector is then subjected to position embedding encoding by the second position encoder to obtain a new first vector; The new first vector is encoded by the second Transformer encoder to obtain the second vector.

7. An emotion recognition method, characterized in that, include: Obtain the target music track to be identified; The target music track is processed by a target music emotion recognition model to obtain the emotion category of the target music track. The target music emotion recognition model is trained using the model training method described in any one of claims 1-6.

8. A model training device for training a music emotion recognition model, characterized in that, The music emotion recognition model includes a first Transformer encoding layer, a second Transformer encoding layer, a first multi-head attention layer, a second multi-head attention layer, and an emotion classification layer. The training device includes: The self-supervised learning module is used to train the music emotion recognition model in a self-supervised manner based on multiple different unlabeled preset audio datasets to update the music emotion recognition model. The audio data in the same preset audio dataset has the same attributes, while the audio data in different preset audio datasets has different attributes. The model training module is used for: Select a training sample from a pre-defined set of labeled training samples as the target training sample. The training sample includes music audio and labeled emotional information. The music audio in the target training sample is encoded through the first Transformer encoding layer to obtain a first vector; The first vector is encoded by the second Transformer encoding layer to obtain the second vector, and the first vector is processed by the first multi-head attention layer to obtain the third vector; The second vector and the third vector are processed by the second multi-head attention layer to obtain the target vector, and the target vector is processed by the sentiment classification layer to obtain the predicted sentiment information. The model loss value is determined based on the labeled sentiment information and the predicted sentiment information; When the model loss value is greater than the preset loss value, the parameters of the music emotion recognition model are updated, and the process returns to the step of selecting a training sample from the preset training sample set as the target training sample. When the model loss value is less than or equal to the preset loss value, training of the music emotion recognition model is stopped, and the target music emotion recognition model is obtained.

9. A server, characterized in that, The server includes a processor, a memory, and a computer program stored in the memory and executable by the processor, wherein when the computer program is executed by the processor, it implements the steps of the model training method as described in any one of claims 1 to 6 or the emotion recognition method as described in claim 7.

10. A storage medium for computer-readable storage, characterized in that, The storage medium stores a computer program, wherein when the computer program is executed by a processor, it implements the steps of the model training method as described in any one of claims 1 to 6 or the emotion recognition method as described in claim 7.

Images