A music recognition model training method, music recognition method and related equipment
Through multiple iterative training of feature extraction and mapping sub-models, the problem of insufficient training data for multi-modal music recognition models is solved, the recognition accuracy and training efficiency are improved, and efficient music recognition under small sample data is achieved.
Patent Information
- Application Number
- CN202411170712.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-26
- Publication Date
- 2024-11-15
- Estimated Expiration
- 2044-08-26
AI Technical Summary
In the prior art, the multimodal music recognition model requires a huge music data set for training because it contains a large number of model parameters, but the data volume of open source music data set is insufficient, resulting in low recognition accuracy.
Multiple iterative training is carried out using feature extraction sub-model and feature map sub-model. Spectral feature maps are used to describe text features, and the model parameters of the feature map module are adjusted to reduce the training data volume requirement and improve recognition accuracy.
It realizes the music recognition model with high recognition accuracy training under smaller sample data, avoiding the problem of low training accuracy, simplifying the training process, and improving training efficiency.
Smart Images

Figure CN118692425B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of computer technology, and in particular to a training method for a music recognition model, a music recognition method and related equipment. Background Art
[0002] With the continuous development of science and technology, more and more devices can identify music through multimodal music recognition models, so as to achieve the purpose of describing music in the form of text, so that the device can have richer music recognition functions. For example, when there is music playing in the environment, the device can identify the music through the music recognition model and describe the music to the user from multiple aspects such as the music name, performer, and instrument. Therefore, the device can provide the user with music question-and-answer services for the music, music retrieval services of similar music styles, or music friendship services, etc.
[0003] In related technologies, in order to describe music in text form, the model structure of the music recognition model usually adopts a multimodal large language model structure, and the method for training the music recognition model is usually to use open source music datasets in network resources to perform multiple rounds of iterative training on the music recognition model to be trained.
[0004] In each round of iterative training, the music recognition model to be trained is used to extract features of the music data to obtain music features, and the obtained music features are identified to obtain corresponding text descriptions; based on the error between the text description and the sample description associated with the music data, all model parameters of the music recognition model are adjusted.
[0005] However, since the multimodal music recognition model contains larger model parameters than the single-modal recognition model to recognize music features as text descriptions, the multimodal music recognition model requires a larger amount of music data for training.
[0006] However, since the usage scenarios of open source music datasets in online resources are relatively niche, the data volume of open source music datasets is insufficient to support the training of music recognition models with such a large number of model parameters, resulting in low recognition accuracy of music recognition models trained based on open source music datasets.
[0007] It can be seen that the music recognition model trained by the method of training the music recognition model under the related art has low recognition accuracy. Summary of the invention
[0008] The embodiments of the present application provide a music recognition model training method, a music recognition method and related equipment, which are used to solve the problem of low recognition accuracy of the trained music recognition model.
[0009] In a first aspect, a method for training a music recognition model is provided, comprising:
[0010] Acquire a sample data set; wherein each of the sample data includes: a corresponding sample music segment, and a sample music description including multiple sample music attributes of the sample music segment;
[0011] Based on the sample data set, the music recognition model to be trained is trained for multiple rounds of iterative training; wherein the music recognition model includes: a trained feature extraction sub-model and a text recognition sub-model, and a feature mapping sub-model to be trained; each round of iterative training includes:
[0012] Using the feature extraction sub-model, extracting the frequency spectrum features presented by the sample music clip;
[0013] The feature mapping sub-model is used to map the obtained spectral features into descriptive text features; wherein the descriptive text features are used to describe multiple training music attributes of the sample audio segment;
[0014] Using the text recognition sub-model, identifying the description text features, and obtaining training music descriptions;
[0015] Based on the difference between the obtained training music description and the corresponding sample music description, the model parameters of the feature mapping module are adjusted.
[0016] In a second aspect, a music recognition method is provided, comprising:
[0017] Get the music to be identified;
[0018] Using a trained music recognition model, extracting the spectral features to be recognized presented by the music to be recognized, and mapping the spectral features to be recognized into text features to be recognized; wherein the music recognition model is trained using the method described in the first aspect;
[0019] The music recognition model is used to identify the text features to be recognized and obtain a target music description; wherein the target music description is used to introduce multiple target music attributes of the music to be recognized in text form.
[0020] In a third aspect, a training device for a music recognition model is provided, comprising:
[0021] Acquisition module: used to acquire sample data sets; wherein each of the sample data includes: a corresponding sample music segment, and a sample music description including multiple sample music attributes of the sample music segment;
[0022] Processing module: used for performing multiple rounds of iterative training on the music recognition model to be trained based on the sample data set; wherein the music recognition model includes: a trained feature extraction sub-model and a text recognition sub-model, and a feature mapping sub-model to be trained; each round of iterative training includes:
[0023] The processing module is specifically used to: use the feature extraction sub-model to extract the frequency spectrum features presented by the sample music clip;
[0024] The processing module is specifically used to: use the feature mapping sub-model to map the obtained spectral features into descriptive text features; wherein the descriptive text features are used to describe multiple training music attributes of the sample audio segment;
[0025] The processing module is specifically used to: use the text recognition sub-model to identify the description text features and obtain the training music description;
[0026] The processing module is specifically used to adjust the model parameters of the feature mapping module based on the difference between the obtained training music description and the corresponding sample music description.
[0027] Optionally, the acquisition module is specifically used for:
[0028] Collecting multimedia files of a plurality of music, and collecting metadata of each multimedia file;
[0029] Based on the preset segment duration, the multimedia files are divided into segments to obtain a plurality of sub-files corresponding to the corresponding multimedia files, and the obtained sub-files are used as corresponding sample music segments;
[0030] Using the trained multimodal recognition model, respectively perform music attribute recognition on each obtained sample music clip to obtain multiple sample music attributes of the corresponding sample music clip;
[0031] Using the trained large language model, based on the obtained metadata and the sample music attributes, respectively generate sample music descriptions for the sample music clips;
[0032] Based on the sample music clips and their corresponding sample music descriptions, a sample data set is obtained.
[0033] Optionally, the processing module is further used for:
[0034] Before performing multiple rounds of iterative training on the music recognition model to be trained based on the sample data set, establishing a feature extraction sub-model and a feature mapping sub-model to be trained based on the structural building strategy of the music recognition model, and obtaining a trained text recognition sub-model;
[0035] Based on the collected multiple audio data, the feature extraction sub-model to be trained is trained for multiple rounds of iterations, and the trained feature extraction sub-model is output;
[0036] Based on the trained feature extraction sub-model and text recognition sub-model, and the feature mapping sub-model to be trained, a music recognition model to be trained is established.
[0037] Optionally, the processing module is specifically used for:
[0038] In each training iteration, the following operations are performed:
[0039] Performing sub-data masking on the audio data to obtain masked data that masks the random sub-data;
[0040] Performing feature extraction on the masked data to obtain data features;
[0041] Based on the data feature, predict the random sub-data masked in the masked data to obtain predicted sub-data;
[0042] Based on the difference between the obtained predicted sub-data and the random sub-data, the model parameters of the feature extraction sub-model are adjusted.
[0043] Optionally, the processing module is specifically used for:
[0044] Using the feature extraction sub-model, extracting the spectrum sub-features of each of the multiple sub-segments contained in the sample music segment, and extracting the spectrum correlation features between the spectrum sub-features;
[0045] The obtained spectral sub-features and spectral correlation features are integrated to obtain the spectral features presented by the sample music clip.
[0046] Optionally, the spectral sub-features represent: at least one of the pitch, timbre, sound receiving environment, beat, instrument and emotion of the corresponding sub-segment; and the spectral association features represent: at least one of the similarity, rhythmic relationship and pitch conversion between the sub-segments.
[0047] Optionally, the processing module is specifically used for:
[0048] Based on the model parameters of the feature mapping sub-model in this round of training, a first mapping relationship between each attribute spectrum feature learned by the feature mapping sub-model and each attribute text feature is obtained; wherein each attribute text feature represents one of a plurality of reference music attributes; the plurality of reference music attributes at least include: a union of the plurality of sample music attributes and the plurality of training music attributes;
[0049] Based on the first mapping relationship, the spectrum feature is mapped into a plurality of attribute text features; wherein the plurality of attribute text features represent the plurality of training music attributes;
[0050] Multiple attribute text features are obtained by fusing to generate description text features.
[0051] Optionally, the processing module is further used for:
[0052] Before the feature mapping submodel is used to map the obtained spectral features to describe text features, based on a preset second mapping relationship between each data format and each occupied data amount, a target data format whose corresponding occupied data amount is smaller than a reference data amount is selected from each data format; wherein the parameter data amount is: the occupied data amount corresponding to the data format used by the model parameter of the feature mapping submodel;
[0053] The frequency spectrum feature is converted into the target data format to obtain the converted frequency spectrum feature.
[0054] Optionally, the processing module is specifically used for:
[0055] Get the memory usage of the video memory used to run this round of training;
[0056] When it is determined that the video memory occupancy is greater than the preset occupancy, based on a preset second mapping relationship between each data format and each occupied data amount, a target data format whose corresponding occupied data amount is smaller than the reference data amount is selected from the data formats.
[0057] Optionally, the processing module is specifically used for:
[0058] Determining a mapping training loss based on differences between a plurality of training music attributes included in the obtained training music description and a plurality of sample music attributes included in the corresponding sample music description;
[0059] Based on the mapping training loss, model parameters of the feature mapping module are adjusted.
[0060] Optionally, the processing module is specifically used for:
[0061] determining a semantic training loss based on the obtained textual similarity between the training music descriptions and the corresponding sample music descriptions;
[0062] When it is determined based on the mapping training loss and the semantic training loss that a training target has not been achieved, a model parameter of the feature mapping module is adjusted.
[0063] In a fourth aspect, a music recognition device is provided, comprising:
[0064] Acquisition module: used to acquire music to be identified;
[0065] Processing module: used to use a trained music recognition model to extract the spectral features to be recognized presented by the music to be recognized, and map the spectral features to be recognized into text features to be recognized; wherein the music recognition model is trained using the method described in the first aspect;
[0066] The processing module is also used to: use the music recognition model to identify the text features to be recognized and obtain a target music description; wherein the target music description is used to: introduce multiple target music attributes of the music to be recognized in text form.
[0067] According to a fifth aspect, a computer program product is provided, comprising a computer program, which, when executed by a processor, implements the method according to the first aspect or the second aspect.
[0068] A sixth aspect provides a computer device, including:
[0069] A memory for storing program instructions;
[0070] The processor is used to call the program instructions stored in the memory, and execute the method as described in the first aspect or the second aspect according to the obtained program instructions.
[0071] In a seventh aspect, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer-executable instructions, wherein the computer-executable instructions are used to enable a computer to execute the method as described in the first aspect or the second aspect.
[0072] In the embodiment of the present application, a relatively easier to obtain trained text recognition sub-model is used to build the model structure of the music recognition model to be trained. There is no need to train the huge model parameters contained in the model structure of multimodal recognition, nor is there any need to train a large model such as a text recognition sub-model. This reduces the difficulty of training from multiple perspectives, shortens the training time, and thus improves the training efficiency.
[0073] Furthermore, since in any music recognition scenario, the training process of the music recognition model to be trained does not require training of the large language model of the text recognition sub-model, the problem of low training accuracy of the music recognition model due to insufficient sample data or overfitting of training is avoided, as well as the problem of low recognition accuracy of the trained music recognition model obtained through training.
[0074] Furthermore, when training the music recognition model, it is only necessary to train the feature mapping sub-model used to map spectral features into features describing text, so as to align the feature space of the music modality with the feature space of the text modality, thereby greatly reducing the amount of data of the model parameters that need to be trained in the music recognition model. Therefore, a trained music recognition model with high recognition accuracy can be trained without too much sample data, thereby reducing the difficulty of obtaining sample data and ensuring the recognition accuracy of the trained music recognition model. BRIEF DESCRIPTION OF THE DRAWINGS
[0075] Figure 1A Schematic diagram 1 of an application field of the training method of the music recognition model provided in the embodiment of the present application;
[0076] Figure 1B A schematic diagram of an application field of the training method of the music recognition model provided in the embodiment of the present application Figure 2 ;
[0077] Figure 1C An application scenario of the training method of the music recognition model provided in the embodiment of the present application;
[0078] Figure 2 A flowchart of a method for training a music recognition model provided in an embodiment of the present application is shown in FIG1 ;
[0079] Figure 3A A schematic diagram of a principle of a method for training a music recognition model provided in an embodiment of the present application;
[0080] Figure 3B A schematic diagram of a training method for a music recognition model provided in an embodiment of the present application Figure 2 ;
[0081] Figure 4A A third schematic diagram of a principle of a method for training a music recognition model provided in an embodiment of the present application;
[0082] Figure 4B A fourth schematic diagram of a principle of a method for training a music recognition model provided in an embodiment of the present application;
[0083] Figure 4C A fifth schematic diagram of a principle of a method for training a music recognition model provided in an embodiment of the present application;
[0084] Figure 4D A schematic diagram of a training method for a music recognition model provided in an embodiment of the present application Figure 6 ;
[0085] Figure 5AA schematic diagram of a principle of a method for training a music recognition model provided in an embodiment of the present application is shown in FIG7;
[0086] Figure 5B A schematic diagram of a principle of a method for training a music recognition model provided in an embodiment of the present application is shown in FIG8;
[0087] Figure 5C A schematic diagram of a training method for a music recognition model provided in an embodiment of the present application Figure 9 ;
[0088] Figure 5D A schematic diagram of a principle of a method for training a music recognition model provided in an embodiment of the present application is shown in Figure 10;
[0089] Figure 6 A schematic diagram of a principle of a method for training a music recognition model provided in an embodiment of the present application is shown in FIG11;
[0090] Fig. 7A A flow chart of a method for training a music recognition model provided in an embodiment of the present application Figure 2 ;
[0091] Figure 7B A schematic diagram 12 of a principle of a method for training a music recognition model provided in an embodiment of the present application;
[0092] Fig. 8A A schematic diagram of the structure of a training device for a music recognition model provided in an embodiment of the present application;
[0093] Figure 8B A schematic diagram of the structure of a music recognition device provided in an embodiment of the present application;
[0094] Fig. 9 Another structural schematic diagram of a music recognition model training device or a music recognition device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0095] In order to make the purpose, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application.
[0096] Some of the terms used in the embodiments of the present application are explained below to facilitate understanding by those skilled in the art.
[0097] (1) Multimodal:
[0098] In the field of machine learning, multimodality refers to learning tasks and techniques that involve two or more different types of data modes. These data modes can include but are not limited to text, images, audio, video, etc. Multimodal learning aims to improve the performance of the model by combining different types of input data.
[0099] (2) Pre-training:
[0100] The basic idea of pre-training is to first train a basic model on a large amount of unlabeled data, and then apply this basic model to a specific task, which usually requires further fine-tuning to adapt to specific downstream tasks.
[0101] (3) Large language model (LLM):
[0102] A large language model is a language model with a large number of model parameters and high complexity. Such models are usually built using deep learning technology and trained on large-scale data sets. Large language models can process and generate natural language text and have strong language understanding and generation capabilities.
[0103] (4) Music Large Language Model (MusicLLM or musicLLM):
[0104] Music language models refer to a type of multimodal language model specifically used for music generation and understanding. This type of model usually uses deep learning techniques, especially neural network architectures, to learn complex patterns of music data and can generate new music works or perform music-related tasks.
[0105] (5) waveform:
[0106] Waveform is used to describe the shape of a signal or wave over time. It can be used in physics, signal processing, audio engineering, electronics and other fields. Waveform can be a sound wave, an electrical signal or any wave that can be expressed as a function of time. Waveform can be observed and recorded by various devices (such as oscilloscope). Common waveforms include sine wave, square wave, triangle wave, sawtooth wave, etc.
[0107] (6) Transformer structure:
[0108] The transformer structure can include an encoder-decoder architecture, where the encoder is responsible for converting the input sequence into an intermediate representation, and the decoder is responsible for generating an output sequence using the representation produced by the encoder.
[0109] The transformer structure can also include a Self-Attention Mechanism layer, where Multi-Head Attention allows the model to learn information from different representation subspaces; Positional Encoding is used to retain the position information of the input sequence.
[0110] The transformer structure can also include a feed-forward neural network, which can be placed after each self-attention mechanism layer for further feature processing.
[0111] The transformer structure can also include residual connections, which are placed before and after each sub-layer to help alleviate the gradient vanishing problem and speed up the training process.
[0112] The transformer structure can also include layer normalization to act on the output of each sub-layer to help stabilize the training process.
[0113] It should be noted that the embodiments of the present application involve operations to obtain sample data sets and other data. When the above embodiments of the present application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards of relevant countries and regions.
[0114] In the embodiments of the present application, the term "module" or "unit" refers to a computer program or a part of a computer program that has a predetermined function and works together with other related parts to achieve a predetermined goal, and can be implemented in whole or in part by using software, hardware (such as processing circuits or memories), or a combination thereof. Similarly, a processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be part of an overall module or unit that includes the function of the module or unit.
[0115] The following is a brief introduction to the training method of the music recognition model and the application fields of the music recognition method provided in the embodiments of the present application.
[0116] With the continuous development of science and technology, more and more devices can recognize music through multimodal music recognition models (such as music large language models) to achieve the purpose of describing music in the form of text. As a result, the devices can have richer music recognition functions.
[0117] For example, see Figure 1A When there is music playing in the environment, the device can identify the music through the music recognition model and describe the music to the user from multiple aspects such as the music name, performer, and instrument.
[0118] For another example, please refer to Figure 1B , the device plays a piece of music, and the device can identify the music through the music recognition model. When the device collects the question voice or question text about the music, it can generate the corresponding reply voice or reply text. For example, if the question voice is "In which year was the music just played created?", the reply voice at least includes "The music was created in 1895."
[0119] Through the music recognition model, the device can provide not only music question-and-answer services for the music to the user, but also music retrieval services or music dating services with similar music styles, without any specific restrictions.
[0120] In related technologies, in order to describe music in text form, the model structure of the music recognition model usually adopts a multimodal large language model structure, and the method for training the music recognition model is usually to use open source music datasets in network resources to perform multiple rounds of iterative training on the music recognition model to be trained.
[0121] In each round of iterative training, the music recognition model to be trained is used to extract features of the music data to obtain music features, and the obtained music features are identified to obtain corresponding text descriptions; based on the error between the text description and the sample description associated with the music data, all model parameters of the music recognition model are adjusted.
[0122] However, since the multimodal music recognition model contains larger model parameters than the single-modal recognition model to recognize music features as text descriptions, the multimodal music recognition model requires a larger amount of music data for training.
[0123] However, since the usage scenarios of open source music datasets in online resources are relatively niche, the data volume of open source music datasets is insufficient to support the training of music recognition models with such a large number of model parameters, resulting in low recognition accuracy of music recognition models trained based on open source music datasets.
[0124] It can be seen that the music recognition model trained by the method of training the music recognition model under the related art has low recognition accuracy.
[0125] In order to solve the problem of low recognition accuracy of the trained music recognition model, the present application proposes a training method for a music recognition model. In the method, a sample data set is obtained, and each sample data includes: a corresponding sample music segment, and a sample music description containing multiple sample music attributes of the sample music segment. Based on the sample data set, the music recognition model to be trained is trained for multiple rounds of iterations. The music recognition model includes: a trained feature extraction sub-model and a text recognition sub-model, and a feature mapping sub-model to be trained.
[0126] Each round of iterative training includes:
[0127] The feature extraction sub-model is used to extract the spectral features presented by the sample music clip. The feature mapping sub-model is used to map the obtained spectral features into descriptive text features. The descriptive text features are used to describe various training music attributes of the sample audio clip. The text recognition sub-model is used to identify the descriptive text features and obtain the training music description. Based on the difference between the obtained training music description and the corresponding sample music description, the model parameters of the feature mapping module are adjusted.
[0128] In the embodiment of the present application, a relatively easier to obtain trained text recognition sub-model is used to build the model structure of the music recognition model to be trained. There is no need to train the huge model parameters contained in the model structure of multimodal recognition, nor is there any need to train a large model such as a text recognition sub-model. This reduces the difficulty of training from multiple perspectives, shortens the training time, and thus improves the training efficiency.
[0129] Furthermore, since in any music recognition scenario, the training process of the music recognition model to be trained does not require training of the large language model of the text recognition sub-model, the problem of low training accuracy of the music recognition model due to insufficient sample data or overfitting of training is avoided, as well as the problem of low recognition accuracy of the trained music recognition model obtained through training.
[0130] Furthermore, when training the music recognition model, it is only necessary to train the feature mapping sub-model used to map spectral features into features describing text, so as to align the feature space of the music modality with the feature space of the text modality, thereby greatly reducing the amount of data of the model parameters that need to be trained in the music recognition model. Therefore, a trained music recognition model with high recognition accuracy can be trained without too much sample data, thereby reducing the difficulty of obtaining sample data and ensuring the recognition accuracy of the trained music recognition model.
[0131] The following describes the application scenarios of the training method for the music recognition model provided in this application.
[0132] Please refer to Figure 1C , which is a schematic diagram of an application scenario of the training method of the music recognition model provided in the present application. The application scenario includes a client 101 and a server 102. The client 101 and the server 102 can communicate with each other, and the communication method can be to communicate using wired communication technology, for example, by connecting a network cable or a serial port cable; or to communicate using wireless communication technology, for example, by communicating through Bluetooth or wireless fidelity (WIFI) and other technologies, without specific limitation.
[0133] The client 101 generally refers to a device that can collect music, present corresponding music descriptions, etc., such as a terminal device, a third-party application that can be accessed by the terminal device, or a web page that can be accessed by the terminal device. Terminal devices include but are not limited to mobile phones, computers, smart medical devices, smart home appliances, vehicle-mounted terminals or aircraft, etc. The server 102 generally refers to a device that can train and use music recognition models, etc., such as a terminal device or a server, etc. The server includes but is not limited to a cloud server, a local server or an associated third-party server, etc. Both the client 101 and the server 102 can use cloud computing to reduce the occupation of local computing resources; cloud storage can also be used to reduce the occupation of local storage resources.
[0134] As an embodiment, the client 101 and the server 102 may be the same device or different devices, without any specific limitation.
[0135] The following is based on Figure 1C , the training method of the music recognition model provided in the embodiment of the present application is specifically introduced. Please refer to Figure 2 , which is a flow chart of the training method of the music recognition model provided in an embodiment of the present application.
[0136] S201, obtaining a sample data set.
[0137] Before training the music recognition model to be trained, a sample data set may be obtained first, for example, from network resources; for another example, a sample data set may be constructed based on music resources in network resources, etc., without any specific limitation.
[0138] Each sample data includes: a corresponding sample music clip and a sample music description including multiple sample music attributes of the sample music clip. The sample music clip is a piece of music of a certain duration, which can be represented by data points arranged in time sequence or by symbols.
[0139] For example, it can be expressed in the form of a waveform graph, please refer to Figure 3A (1) in ; for another example, it can be expressed in the form of a digital multimedia file, such as MP3 format, AAC (Advanced Audio Coding) format or FLAC (Free Lossless Audio Codec) format, please refer to Figure 3A (2) in ; for another example, the format defined by the Musical Instrument Digital Interface (MIDI) can be used for representation, see Figure 3A (3) in ; for another example, ASCII characters can be used to represent Figure 3A in (4).
[0140] The duration of the sample music clip can be pre-set, such as 30 seconds; or it can be a duration that ensures the integrity of the lyrics, melody or beat, etc., without specific restrictions. For example, for a complete sample music, the sample music can be divided into multiple sample music clips according to the punctuation or paragraph division in the lyrics; for another example, for a complete sample music, multiple music segments can be obtained according to the punctuation or paragraph division in the lyrics, and then each music segment can be divided into multiple sample music segments according to the beat, etc., without specific restrictions.
[0141] The sample music description includes multiple sample music attributes of the corresponding sample music clip. For example, the sample music description describes the multiple sample music attributes of the corresponding sample music clip in fluent sentences; for another example, the sample music description describes the multiple sample music attributes of the corresponding sample music clip in the style of classical Chinese, prose or poetry; for another example, the sample music description describes each sample music attribute in the form of pictures and texts to more intuitively and vividly describe the multiple sample music attributes of the corresponding sample music clip, etc., without specific limitation.
[0142] For example, the sample music description can be "This is a piece of pop music, which was collected by live recording and uses music elements of minority languages. This music uses a large number of string instruments, including cello, double bass, violin, and fiddle. The rhythm of the music is 71bpm, and the performance of string instruments creates a melodious atmosphere. The string section of string instruments is used in the climax of the music, creating a very infectious musical atmosphere that is intoxicating."
[0143] Among them, “Popular”, “Live”, “Minority Language Music Elements”, “Cello”, “Double bass”, “Violin”, “fiddle”, “71bpm”, “String Instruments” and “String section” are various sample music attributes.
[0144] As an embodiment, considering that the data sets in the network resources are mainly music descriptions in English, since the number of model parameters that need to be trained in the music recognition model to be trained provided in the embodiment of the present application is relatively small, music resources can be obtained from the network resources to construct a sample data set to obtain music descriptions in any language, thereby providing flexibility in training the music recognition model to improve the applicability of the trained music recognition model.
[0145] Multiple multimedia files of music can be collected from network resources or other devices, and metadata of each multimedia file can be collected. Each multimedia file can be any one of the representation forms of the sample music clips introduced above, or other representation forms, without specific limitation. The metadata of the multimedia file includes, for example, the music name, performer, singer, lyrics, comments of the corresponding music, and tags added to the music when uploading the music on platforms such as network resources, etc., without specific limitation.
[0146] Taking the sample music clips with the same length, such as 30 seconds, as an example, each multimedia file can be divided into segments based on the preset segment length, and multiple sub-files corresponding to the corresponding multimedia files can be obtained, and each obtained sub-file can be used as the corresponding sample music clip.
[0147] There are many methods for determining the sample music attributes of the sample music clips. For example, a trained multimodal recognition model is used to identify the music attributes of each sample music clip obtained, and obtain multiple sample music attributes of the corresponding sample music clips. The trained multimodal recognition model is used to identify the input music clip as multiple sample music attributes. The trained multimodal recognition model can be trained based on an open source database in network resources. The open source database is, for example, a 1-second music clip and its corresponding English music attributes; the trained multimodal recognition model can also be an open source multimodal large model in network resources, etc., without specific limitation.
[0148] For another example, a reference music clip corresponding to each of a plurality of reference music attributes is pre-stored, and by determining the music similarity between the sample music clip and each reference music clip, the reference music attribute corresponding to the reference music clip having a music similarity greater than a preset threshold is used as the sample music attribute of the sample music clip. The plurality of reference music attributes may include music attributes measured from various perspectives such as music style, music theme, rhythm, beat, instrument, sound type, tone and genre. For example, the sound type is male voice, female voice, rain sound, wind sound, environmental sound, animal sound, etc.
[0149] After obtaining each sample music clip and its metadata and sample music attributes, the trained large language model can be used to generate sample music descriptions of each sample music clip based on the obtained metadata and sample music attributes. Thus, a sample data set can be obtained based on each sample music clip and its corresponding sample music description.
[0150] The trained large language model is used to arrange the input words into paragraphs in the specified language. If the input words are in a language other than the specified language, the trained large language model can translate the words in the other language into the specified language and then arrange the translated words into paragraphs. Therefore, no matter what language the obtained sample music attributes are expressed in, they can be converted into the specified language when generating the sample music description, which reduces the language requirements for the process of obtaining the sample music attributes, thereby reducing the difficulty of obtaining the sample music attributes. The large language model is, for example, a Chinese large language model, which is used to generate high-quality Chinese texts, including dialogues, articles, poems, etc.
[0151] In the process of constructing a sample data set, there is no need to manually label the sample data, which greatly reduces the cost of data labeling. In addition, the use of open source models makes the process of constructing a sample data set more efficient and convenient, and a sample data set containing a large number of sample data can be constructed according to usage requirements.
[0152] Please refer to Figure 3B Taking a piece of music as an example, after collecting the multimedia file and metadata of the music, the multimedia file is divided into multiple sub-files, such as 6 sub-files, according to 30 seconds as a sample music clip. Each sub-file is a sample music clip, that is, sample music clip A, sample music clip B, sample music clip C, sample music clip D, sample music clip E and sample music clip F are obtained.
[0153] Each sample music clip is input into the trained multimodal recognition model to obtain a plurality of sample music attributes of the corresponding sample music clip output by the multimodal recognition model. The metadata and the plurality of sample music attributes of each sample music clip are input into the trained large language model to obtain a sample music description of the corresponding sample music clip output by the large language model, that is, sample music description A, sample music description B, sample music description C, sample music description D, sample music description E and sample music description F are obtained.
[0154] The large language model can perform semantic recognition based on the lyrics in the metadata, and integrate the atmosphere created by the semantics of the lyrics into the description of multiple sample music attributes. For example, if the attribute of a sample music is "orchestral music" and the lyrics have a light-hearted and cheerful semantics, then the sample music description can include sentences such as "through the performance of orchestral music, a light-hearted and cheerful atmosphere is created."
[0155] The large language model can also perform character recognition based on the singer in the metadata and incorporate the singer's timbre into the description of multiple sample music attributes. For example, if the attribute of a sample music is "popular" and the singer's timbre is relatively soft, the sample music description may include sentences such as "This piece of Cpop music is sung with a warm and soft voice, like the breeze in a summer evening, sweet and full of nostalgia."
[0156] The large language model can also describe the sample music attributes based on the characteristics of the sample music attributes themselves. For example, if the sample music attribute is "irregular beat", then the sample music description may include sentences such as "using irregular beats to create tension and mystery, conveying complex or unconventional emotions".
[0157] Therefore, each sample music clip and its corresponding sample music description can be added to the sample data set to achieve the purpose of establishing the sample data set. Through the vivid sample music description, when training the music recognition model based on the sample data set, the trained music recognition model can learn the vivid description method in the sample music description, so that the music recognition model can output a more readable and vivid description text to show the charm of Chinese characters and reach the level of writer's writing.
[0158] S202, based on the sample data set, performing multiple rounds of iterative training on the music recognition model to be trained.
[0159] After obtaining the sample data set, the music recognition model to be trained can be trained for multiple rounds of iterations based on the sample data set. The music recognition model to be trained can include: a trained feature extraction sub-model and a text recognition sub-model, and a feature mapping sub-model to be trained.
[0160] Then, using the sample dataset, only the feature mapping sub-model in the music recognition model needs to be trained. Therefore, compared with training the entire music recognition model or a large-scale model structure, the sample dataset contains a smaller number of sample data, constructing the sample dataset is easier and faster, and the training process is more efficient.
[0161] Please refer to Figure 4A After the sample data is input into the trained feature extraction sub-model, the trained feature extraction sub-model, the feature mapping sub-model to be trained and the trained text recognition sub-model are used in turn to process the sample data layer by layer. By adjusting the model parameters of the feature mapping sub-model to be trained, this round of iterative training for the music recognition model to be trained is completed.
[0162] As an embodiment, before performing multiple rounds of iterative training on the music recognition model to be trained based on the sample data set, the music recognition model to be trained can be built first. Since the music recognition model to be trained can include a trained feature extraction sub-model and a text recognition sub-model, as well as a feature mapping sub-model to be trained, for example, the trained feature extraction sub-model and the text recognition sub-model can be obtained first, and then the network layer structure of the feature mapping sub-model to be trained can be built.
[0163] For another example, you can first build a feature extraction sub-model to be trained, perform multiple rounds of iterative training on the feature extraction sub-model to be trained, obtain the trained feature extraction sub-model, and then obtain the trained text recognition sub-model, and build the network layer structure of the feature mapping sub-model to be trained. Then, for example, in the case where the sample data set is constructed by collecting multimedia files of music, using music collected with the sample data set or music similar to it to train the feature extraction sub-model, the feature extraction sub-model can be made to more accurately extract the features of each sample data in the sample data set, thereby improving the accuracy of the trained feature mapping sub-model, and making the mapping accuracy of the trained feature mapping sub-model higher.
[0164] Based on the structural construction strategy of the music recognition model, a feature extraction sub-model and a feature mapping sub-model to be trained are established, and a trained text recognition sub-model is obtained. The structural construction strategy can indicate the network layer structure of the feature extraction sub-model, the network layer structure of the feature mapping sub-model, and the acquisition location of the used text recognition sub-model, such as the download address in the network resource, etc., without specific limitation.
[0165] For example, see Figure 4BThe structural construction strategy indicates that the network layer structure of the feature extraction sub-model is the model structure of the AudioMAE model, or a 24-layer serial transformer structure, etc.; and indicates that the network layer structure of the feature mapping sub-model is a linear transformation layer; and indicates that the text recognition sub-model is the internlm2-chat-7b Chinese large model.
[0166] After collecting multiple audio data, establishing the feature extraction sub-model and feature mapping sub-model to be trained, and obtaining the trained text recognition sub-model, the feature extraction sub-model to be trained can be trained multiple rounds of iteratively based on the collected multiple audio data, and the trained feature extraction sub-model can be output. Thus, the music recognition model to be trained can be established based on the trained feature extraction sub-model and text recognition sub-model, and the feature mapping sub-model to be trained.
[0167] The multiple audio data may include sample music clips in the sample data set, or may come from open source audio data sets from network resources. Since the feature extraction sub-model can adopt an unsupervised training method and does not require labeling, the number of multiple audio data collected can be much larger than the number of sample data contained in the sample data set. For example, the number of sample data contained in the sample data set is 20 million, and the number of multiple audio data is 170 million, so as to ensure the feature extraction accuracy of the trained feature extraction sub-model.
[0168] As an embodiment, when training the feature extraction sub-model, one round of iterative training process is taken as an example for introduction. The other rounds of iterative training processes are similar and will not be described in detail here.
[0169] Sub-data masking is performed on the audio data to obtain masked data that masks the random sub-data. The audio data can also be converted into a specified format, such as a spectrogram format (Mel-frequency cepstral coefficient MFCC or log-Mel spectrum), and then the sub-data masking is performed on the converted audio data to ensure that the audio data can be processed in a unified format, avoiding data processing errors caused by complex and diverse data formats, and improving data processing accuracy.
[0170] Audio data can be divided into multiple sub-data. Taking audio data in spectrogram format as an example, audio data can be divided into multiple audio patches, each of which represents spectrum information within a continuous time window. Then, when sub-data masking is performed on audio data, data masking can be performed on any of the multiple sub-data divided from the audio data, i.e., random sub-data, so that the masked data obtained only contains other sub-data except the random sub-data, where the original position of the random sub-data is occupied by a specific mark.
[0171] Feature extraction is performed on the masked data to obtain data features. The masked data can also be encoded into a data vector form and then feature extracted to capture complex and diverse spectral features in the audio data.
[0172] Based on the obtained data features, data prediction can be performed on the random sub-data that is masked out in the masked data to obtain predicted sub-data. Based on the difference between the obtained predicted sub-data and the random sub-data, the model parameters of the feature extraction sub-model are adjusted.
[0173] The larger the difference between the predicted sub-data and the random sub-data, the less capable the feature extraction sub-model is in extracting the data features of the audio data accurately. Therefore, the model parameters of the feature extraction sub-model can be adjusted and the next round of iterative training can be entered. The smaller the difference between the predicted sub-data and the random sub-data, the more accurately the feature extraction sub-model has learned to extract the data features of the audio data. Therefore, when the difference between the predicted sub-data and the random sub-data is less than the preset value, the model parameters of the feature extraction sub-model of this round can be used as the final model parameters, and the trained feature extraction sub-model can be output.
[0174] By masking the sub-data, the feature extraction sub-model can be trained unsupervised without data labeling, and a trained feature extraction sub-model with accurate feature extraction capability can be obtained.
[0175] Please refer to Figure 4C Taking a 30-second music as an example, 30-second music can more completely present the melody, rhythm and other musical attributes of the music compared to shorter music such as 1 second. Therefore, in the embodiments of the present application, various music clips (such as sample music clips) and audio data are all longer music of 30 seconds. The training method of the music recognition model provided in the embodiments of the present application can recognize such longer music and give corresponding music descriptions.
[0176] During each round of training of the feature extraction sub-model, the feature extraction sub-model is used to divide the 30-second audio data into 128 sub-data arranged in chronological order, and the 128 sub-data are vector-encoded respectively to obtain 128 sub-data vectors of size (1, 1024) arranged in the above chronological order, which are recorded as sub-data vector_1, sub-data vector_2, ... and sub-data vector_128.
[0177] Continue to use the feature extraction sub-model to randomly replace a sub-data vector (such as sub-data vector_1) with a specific identifier (such as X) to obtain a sub-data vector sequence representing the masked data; use the feature extraction sub-model to extract features from the sub-data vector sequence, and predict the above-mentioned sub-data vector_1 based on the obtained data features to obtain a predicted vector.
[0178] Therefore, based on the difference between the prediction vector and the sub-data vector_1, the model parameters of the feature extraction sub-model can be adjusted, and the next round of iterative training can be entered until the obtained difference reaches the training target, and the trained feature extraction sub-model is output.
[0179] The following is a detailed introduction to the training process of the feature mapping sub-module. Please refer to S2021~S2024. Taking one round of iterative training process as an example, the other rounds of iterative training processes are similar and will not be repeated here.
[0180] S2021, uses a feature extraction sub-model to extract the spectral features presented by the sample music clip.
[0181] The trained feature extraction sub-model has the ability to extract the spectral features of music. Therefore, the feature extraction sub-model can be used to extract the spectral features presented by the sample music clip. If the sample music clip is in a unified format, the feature extraction sub-model can be directly used for feature extraction; if it is not in a unified format, the sample music clip can be format converted first, and the sample music clip can be converted into a specific format, such as a spectrogram format, and then feature extraction can be performed on the converted sample music clip. The specific format can be determined according to the format used by the feature extraction sub-model during training, etc., and there is no specific limitation.
[0182] Spectral features are used to characterize information about music attributes such as frequency components, timbre, and rhythm in music. Spectral features include, for example, spectrogram, Mel-frequency cepstral coefficients (MFCCs), fundamental frequency, harmonics, frequency bandwidth, energy, zero crossing rate, spectral roll-off, spectral peaks, spectral entropy, spectral flatness, spectral centroid, spectral slope, spectral band ratio, frequency resolution, and temporal features, etc., without specific limitation.
[0183] As an embodiment, when using a feature extraction sub-model to extract the spectral features of a sample music clip, the sample music clip can be first divided into multiple sub-segments, and the spectral sub-features of each sub-segment are extracted, and the spectral correlation features between each spectral sub-features are extracted, such as the spectral correlation features between every two spectral sub-features. Then, the obtained spectral sub-features and spectral correlation features are fused to obtain the spectral features presented by the sample music clip. Therefore, not only can the spectral sub-features of each sub-segment be extracted from a microscopic perspective, but also the spectral correlation features between each spectral sub-features can be extracted from a macroscopic perspective, thereby ensuring the accuracy of feature extraction.
[0184] Please refer to Figure 4D , the sample music clip is divided into 128 sub-segments, so that 128 spectral sub-features can be extracted, which are recorded as sub-feature vector_1, sub-feature vector_2, ... and sub-feature vector_128. For example, the extracted spectral correlation features include: the correlation feature vector_12 between sub-feature vector_1 and sub-feature vector_2; the correlation feature vector_24 between sub-feature vector_2 and sub-feature vector_4; the correlation feature vector_57 between sub-feature vector_5 and sub-feature vector_7, etc. Then, when merging, the sub-feature vector_1 and the correlation feature vector_12 can be weighted summed to obtain feature vector_1; the sub-feature vector_2, the correlation feature vector_24, and the correlation feature vector_12 can be weighted summed to obtain feature vector_2; and so on, finally the obtained feature vector_1, feature vector_2, ..., feature vector_128 are arranged in sequence to obtain the spectral features.
[0185] As an embodiment, the spectrum sub-feature characterization: at least one of the pitch, timbre, sound receiving environment, beat, instrument and emotion of the corresponding sub-segment, and at least one of the spectrum features described above can also be characterized, without specific limitation. The spectrum association feature characterization: at least one of the similarity, rhythm relationship and pitch conversion between the sub-segments, and at least one of the spectrum features described above can also be characterized, without specific limitation.
[0186] S2022, using a feature mapping sub-model, mapping the obtained spectral features into descriptive text features.
[0187] After obtaining the spectral features of the sample music clip, the feature mapping sub-model can be used to map the obtained spectral features into descriptive text features. The descriptive text features are used to describe various training music attributes of the sample audio clip. By converting the spectral features into descriptive text features, the sample music clip can be understood in the form of text, so that the music recognition model can learn the ability to introduce a piece of music through text description.
[0188] The spectral features are mapped to the text feature space through the feature mapping sub-model, and the dense spectral features extracted by the feature extraction sub-model are aggregated into several learnable text tokens. Compared with the spectral features, these text tokens are more abstract and semantic, that is, higher-level semantic representations can be obtained. At the same time, since the amount of data is reduced, the amount of computation can be reduced.
[0189] In the field of unimodal data processing, such as the legal field, medical field, coding field, and mathematics field, the process of training a unimodal recognition model is equivalent to combining the text data of the unimodal recognition model in the general field and the specific field through the joint probability distribution P (X, Y), and training on the basis of the general field data, so as to better understand and generate the text in the specific field. Among them, X represents the text corpus in the general field, such as a wide range of text data from the Internet, books, news articles, etc.; Y represents the text corpus in the specific field, such as in the legal field, which can include legal documents, case studies, regulations, etc.
[0190] In the field of multimodal data processing, such as audio and video, the process of training a multimodal recognition model is equivalent to training a multimodal recognition model to convert information of one modality into information of another modality through the conditional probability distribution P (X | Y). For example, a multimodal recognition model can extract information from non-text modal data and convert it into text. Among them, X represents general text, that is, usual natural language text; Y represents data of new modalities, such as non-text data such as images and audio. P (X | Y) represents the conditional probability distribution of generating X given Y.
[0191] It can be seen that whether in the field of unimodality or multimodality, the training of this recognition model focuses on learning basic abilities and new knowledge, and the amount of corpus used is usually large. Therefore, in the embodiment of the present application, the unimodal or multimodal recognition process of the model is not trained, and the recognition process directly uses the text recognition submodel that has been trained based on general text for data processing. In the embodiment of the present application, the training process is set on the feature mapping submodel, so that the feature mapping submodel learns the ability to map features from the music dimension to the text dimension, thereby simplifying the training process of the multimodal recognition model to the training process of the feature mapping submodel. If the feature mapping submodel is a linear transformation layer, it can be seen that the embodiment of the present application greatly reduces the number of model parameters that need to be trained and improves the training efficiency.
[0192] Please refer to Figure 5ATaking the feature mapping sub-model as a linear transformation layer as an example, the linear transformation layer can be represented as a weight matrix, such as a weight matrix of size (1024×4096). The values of each weight element in the weight matrix are model parameters of the feature mapping sub-model. Through iterative training, the values of each weight element in the weight matrix can be adjusted so that the linear transformation layer can transform the spectral features into descriptive text features that can be accurately recognized by the trained text recognition model.
[0193] For example, if the size of the spectrum feature is (128×1024), then the spectrum feature and the weight matrix are matrix multiplied to obtain the description text feature, and the size of the description text feature is (128×4096). Through iterative training, the obtained description text feature is recognized by the trained text recognition model as a training music description containing multiple training music attributes, which is very similar to the corresponding sample music description.
[0194] As an embodiment, the model parameters of the feature mapping sub-model are continuously adjusted through each round of iterative training, so that the feature mapping sub-model has the ability to map features from the music dimension to the text dimension. Then, in each round of iterative training, the model parameters of the feature mapping sub-model can characterize a first mapping relationship between each attribute spectral feature currently learned by the feature mapping sub-model and each attribute text feature. Therefore, the model parameters of the feature mapping sub-model are continuously adjusted through multiple rounds of iterative training, so as to achieve the purpose of continuously adjusting the first mapping relationship, making the first mapping relationship more accurate, thereby making the recognition of the trained music recognition model more accurate.
[0195] Therefore, when mapping the obtained spectral features to descriptive text features, it is actually possible to obtain the first mapping relationship between each attribute spectral feature and each attribute text feature learned by the feature mapping sub-model based on the model parameters of the feature mapping sub-model in this round of training. Then, based on the current first mapping relationship, the spectral features are mapped to multiple attribute text features, and the multiple attribute text features represent multiple training music attributes. Thus, the obtained multiple attribute text features can be fused to generate descriptive text features.
[0196] Wherein, each attribute text feature represents one of a plurality of reference music attributes; the plurality of reference music attributes at least include: the union of a plurality of sample music attributes and a plurality of training music attributes. The plurality of reference music attributes may be all the music attributes contained in the music field. The model parameters of the feature mapping sub-model in this round of training learn a first mapping relationship between each attribute spectrum feature and the attribute text features of all the music attributes contained in the music field. Then, after obtaining the spectrum feature, the attribute text features corresponding to each of the plurality of attribute spectrum features contained in the spectrum feature may be determined in the current first mapping relationship, and the reference music attributes corresponding to each of the plurality of attribute text features determined may be recorded as a plurality of training music attributes.
[0197] Please refer to Figure 5B , in this round of iterative training, the model parameters of the feature mapping sub-model can be represented as the mapping relationship between attribute spectrum feature_1 and attribute text feature_1, the mapping relationship between attribute spectrum feature_2 and attribute text feature_2, ..., the mapping relationship between attribute spectrum feature_N and attribute text feature_N. After the feature mapping sub-model obtains a spectrum feature, the feature mapping sub-model determines that the spectrum feature includes attribute spectrum feature_1, attribute spectrum feature_4 and attribute spectrum feature_12. Then, according to the first mapping relationship, it can be determined that the attribute text feature_1 corresponds to the attribute spectrum feature_1, the attribute text feature_4 corresponds to the attribute spectrum feature_4, and the attribute text feature_12 corresponds to the attribute spectrum feature_12.
[0198] Then, attribute text feature_1, attribute text feature_4 and attribute text feature_12 are fused to obtain description text feature. The fusion may be to arrange attribute text feature_1, attribute text feature_4 and attribute text feature_12 in sequence, or to perform weighted summation of attribute text feature_1, attribute text feature_4 and attribute text feature_12, or to concatenate attribute text feature_1, attribute text feature_4 and attribute text feature_12 from beginning to end, etc., and there is no specific limitation.
[0199] As an embodiment, each round of iterative training may process only one sample music clip, or may process multiple sample music clips as a batch. When processing multiple sample music clips as a batch, each sample music clip may be processed in parallel on the same device, or may be processed in a distributed manner on different devices, etc., without specific limitation. For example, a device is configured with 8 graphics processing units (GPUs), each GPU can process 4 sample music clips in parallel, then a device can process 32 sample music clips in parallel. If a batch of sample music clips contains 128 sample music clips, then 4 devices can be used for distributed processing to improve the efficiency of training the music recognition model.
[0200] As an embodiment, in the process of training a music recognition model, various data can be represented in a variety of data formats, for example, data formats include single-precision floating point numbers (FP32), half-precision floating point numbers (FP16), and lower precision (such as BF16). Different data formats occupy different amounts of data, and data formats with higher precision occupy larger amounts of data. In order to further improve the efficiency of model training, it is possible to consider converting the data format to a data format that occupies a smaller amount of data for data processing at some stages of the training process, so as to speed up the processing speed and achieve the purpose of improving the efficiency of model training.
[0201] Before using the feature mapping sub-model to map the obtained spectral features to descriptive text features, based on the second mapping relationship between each preset data format and each occupied data amount, a target data format whose corresponding occupied data amount is less than the reference data amount can be selected from each data format. The parameter data amount is: the occupied data amount corresponding to the data format used by the model parameters of the feature mapping sub-model. The spectral features are converted into the target data format to obtain the converted spectral features. The feature mapping sub-model is then used to map the converted spectral features to descriptive text features, thereby reducing the amount of data that needs to be processed during the feature mapping process and improving the efficiency of model training.
[0202] After obtaining the descriptive text features, a scaling factor may be used to convert the data format of the descriptive text features from the target data format back to the data format corresponding to the reference data volume to ensure data accuracy during the model training process.
[0203] As an embodiment, before converting the data format, it can be determined whether the data format needs to be converted based on the video memory occupancy of the video memory running the current round of training. If the video memory occupancy is low, it means that the video memory currently has more idle processing resources available for use, so there is no need to convert the data format to keep the model training efficient; if the video memory occupancy is high, it means that the idle processing resources of the video memory are relatively tight, so data format conversion can be performed to improve the model training efficiency. Therefore, through the video memory occupancy, it is possible to choose whether to convert the data format in a targeted manner, thereby improving the flexibility of model training.
[0204] After obtaining the video memory occupancy of the video memory running the current round of training, when it is determined that the video memory occupancy is greater than the preset occupancy, based on the second mapping relationship between each preset data format and each occupied data amount, a target data format whose corresponding occupied data amount is less than the reference data amount is selected from each data format. When it is determined that the video memory occupancy is not greater than the preset occupancy, the feature mapping sub-model can be directly used to map the obtained spectrum features into descriptive text features.
[0205] S2023, using the text recognition sub-model to identify the descriptive text features and obtain the training music description.
[0206] After obtaining the descriptive text features, the trained text recognition sub-model may be used to recognize the descriptive text features to obtain a training music description, where the training music description contains a variety of training music attributes of the corresponding sample music clip.
[0207] S2024, adjusting the model parameters of the feature mapping module based on the difference between the obtained training music description and the corresponding sample music description.
[0208] After obtaining the training music description, it can be determined whether the training goal has been achieved based on the difference between the obtained training music description and the corresponding sample music description. If the training goal has not been achieved, the model parameters of the feature mapping module are adjusted; if the training goal has been achieved, the trained music recognition model can be output based on the current model parameters of the feature mapping sub-model. The training goal is, for example, that the difference between the obtained training music description and the corresponding sample music description is less than a preset difference value; another example is that the difference between the obtained training music description and the corresponding sample music description is within a preset difference range, and the number of iterative training times reaches a preset number, etc., without specific limitation.
[0209] As an embodiment, the obtained training music description contains multiple training music attributes, then the mapping training loss can be determined based on the difference between the multiple training music attributes contained in the obtained training music description and the multiple sample music attributes contained in the corresponding sample music description. Based on the mapping training loss, the model parameters of the feature mapping module are adjusted. For example, when the mapping training loss does not reach the training target, the model parameters of the feature mapping module are adjusted; when the mapping training loss has reached the training target, the trained music recognition model can be output based on the current model parameters of the feature mapping sub-model.
[0210] Therefore, as long as the training music description introduces the same training music attributes as the sample music description, or the training music description introduces each training music attribute included in the multiple sample music attributes introduced in the sample music description, and the ratio of the number of each training music attribute to the number of multiple sample music attributes is not less than a preset ratio, etc., regardless of whether the other text in the sentence introducing the training music attribute is the same, it can be determined that the mapping training loss has reached the training target. Then, when determining the difference, it is not necessary to understand the semantics of the music description, which reduces the difficulty of determining the difference and improves the efficiency of determining the difference.
[0211] The mapping training loss can be calculated using the cross entropy loss function, see formula (1), which measures the difference between the probability distribution recognized by the music recognition model and the probability distribution of the sample data.
[0212]
[0213] in, represents the probability that the training music description output by the music recognition model contains any sample music attribute, The training music description output by the music recognition model includes the sum of the probabilities of multiple sample music attributes in the sample music description.
[0214] Please refer to Figure 5C The sample music is described as "This piece of music incorporates local characteristics, cleverly combining traditional and modern elements to show a unique artistic charm. Its BPM (beats per minute) is 2, with a moderate rhythm, which is very suitable for dancing. The sense of rhythm is 1, which makes the whole song full of vitality. The modern pop music elements make the song more fashionable. At the same time, the integration of opera elements makes the whole song full of strong local charm. In addition, singing in Cantonese can bring a unique language beauty to the audience, making the song more regional in character and showing the profoundness of Chinese culture."
[0215] The training music is described as "This piece of music belongs to the category of BPM (beats per minute) 2 and rhythm 1. It has a strong sense of rhythm, which can make the audience dance along with the song involuntarily. The song belongs to the popular Cpop style, which combines modern pop elements with traditional music elements, showing a unique musical style. At the same time, it is sung in Cantonese, which gives it a unique musical style. It not only has pop music elements, but also integrates the characteristics of traditional music, making the song unique."
[0216] Among them, the sample music attributes include "traditional elements", "modern elements", "local characteristics", "BPM (beats per minute) 2 levels", "rhythm 1 level", "popular elements", "opera elements" and "Cantonese".
[0217] The training music attributes include "BPM (beats per minute) 2 levels", "Rhythm 1 level", "Pop style", "Cpop style", "Traditional elements", "Modern elements" and "Cantonese".
[0218] Then, the mapping training loss may be determined based on the differences between the various training music attributes included in the obtained training music description and the various sample music attributes included in the corresponding sample music description.
[0219] As an embodiment, in order to further improve the training accuracy, the semantic training loss can be determined based on the text similarity between the obtained training music description and the corresponding sample music description, based on the determination of the mapping training loss. When it is determined that the training target has not been achieved based on the mapping training loss and the semantic training loss, the model parameters of the feature mapping module are adjusted; when it is determined that the training target has been achieved, the trained music recognition model is output based on the current model parameters of the feature mapping sub-model.
[0220] The semantic training loss can also be calculated using the above formula (1). In this case, represents the probability that the training music description output by the music recognition model contains any phrase in the sample music description, The training music description output by the music recognition model contains the sum of the probabilities of each phrase in the sample music description.
[0221] The semantic training loss can also be determined by calculating the text similarity by the distance between the obtained text features of the training music description and the text features of the corresponding sample music description. The smaller the distance, the greater the text similarity, and the smaller the semantic training loss; the larger the distance, the smaller the text similarity, and the greater the semantic training loss.
[0222] As an embodiment, if multiple sample data are processed in the same batch, then when determining whether to adjust the model parameters of the feature mapping module, the number of sample data that have not reached the training target in the multiple sample data can be determined. When the ratio of the number of sample data that has not reached the training target to the total number of multiple sample data reaches a preset number ratio, it means that more sample data have not reached the training target, and then the model parameters of the feature mapping module are adjusted; when the preset number ratio is not reached, it means that more sample data have reached the training target, and then the trained music recognition model can be output. Evaluate the recognition ability learned by the music recognition model from a macro perspective to avoid overfitting and other situations.
[0223] Please refer to Figure 5D , when processing two sample data in the same batch, for example, the first sample music is described as "this music integrates local characteristics, cleverly combines traditional and modern elements, and shows a unique artistic charm. Its BPM (beats per minute) is 2, with a moderate rhythm, which is very suitable for dancing. The sense of rhythm is 1, which makes the whole song full of vitality. The modern pop music elements make the song more fashionable. At the same time, the integration of opera elements makes the whole song full of strong local charm. In addition, singing in Cantonese can bring a unique language beauty to the audience, making the song more regional in nature and showing the profoundness of Chinese culture."
[0224] The first training music is described as "This piece of music belongs to the category of BPM (beats per minute) 2 and rhythm 1. It has a strong sense of rhythm, which can make the audience dance along with the song involuntarily. The song belongs to the popular Cpop style, which combines modern pop elements with traditional music elements, showing a unique musical style. At the same time, the use of Cantonese singing gives it a unique musical style. It not only has pop music elements, but also integrates the characteristics of traditional music, making the song unique."
[0225] The second sample music is described as "This piece of music belongs to the category of electronic music, with a particular emphasis on its rhythm and dynamics. BPM4 means it has a fast tempo and is suitable for stimulating vitality and enthusiasm. As a type of electronic music, it is usually classified as House Music or EDM (Electronic Dance Music). This music is known for its strong rhythm and sense of movement, which can make people feel energetic and excited. At the same time, this piece of music is commented as "chicken blood dynamic", indicating that it is not only strong in rhythm, but also full of energy and vitality. It is a kind of music that can make people's heart beat faster and blood boil. It is very suitable for promoting activities or sports, or creating a warm atmosphere in parties and social occasions. In addition, the label of "pure music" means that this piece of music has no lyrics and focuses on the expressiveness and emotional communication of the music itself. Without the interference of lyrics, the music can touch the senses and emotions of the audience more directly. Finally, the label of "5-speed rhythm" emphasizes the strong rhythm and sense of movement of this music. Its rhythm and beat are very suitable for dancing and sports, and can make people dance to the rhythm involuntarily. In general, this piece of music is a dynamic, strong-beat, and energetic electronic dance music, which is suitable for inspiring vitality and enthusiasm and creating a warm atmosphere. At the same time, it can also be appreciated as pure music and feel the charm of the music itself."
[0226] The second training music is described as "This music belongs to BPM4. It is a Remix version with a strong sense of rhythm and rhythm. It belongs to the category of European and American music, which combines elements of European and American music and electronic music to create a unique musical style. This song belongs to the category of electronic music and DJ dance music, with a strong sense of rhythm and rhythm, which is very suitable for playing during dancing. DJ J dance music with clang, ringing and sound is a very passionate and energetic music style. It is full of various characteristics of sound, including sound, sound and sound, making the whole music more lively and interesting. This music style is very suitable for parties, dances and other social activities."
[0227] Whether the training goal is achieved can be determined based on the difference between the first sample music description and the first training music description. For example, the difference between "this piece of music incorporates local characteristics", "moderate rhythm", "makes the whole song full of vitality", "makes the song more fashionable", "incorporates opera elements, making the whole song full of strong local charm" and "makes the song more regional and shows the profoundness of Chinese culture" in the first sample music description, and "has a strong sense of rhythm" and "the song belongs to the popular Cpop style" in the first training music description.
[0228] And based on the difference between the second sample music description and the second training music description, determine whether the training goal is achieved. For example, in the second sample music description, "it is usually classified as house dance music or EDM", "this music is commented as "chicken blood dynamic", "it is a kind of music that can make people's heart beat faster and blood boil", "the "pure music" label means that this music has no lyrics and focuses on the expressiveness and emotional communication of the music itself. Without the interference of lyrics, the music can touch the listeners' senses and emotions more directly", and "at the same time, it can also be appreciated as pure music to feel the charm of the music itself". And the difference between "Remix version", "DJ dance music", "DJ J dance music with clang, loud and sound" and "it is full of various characteristics of sound, including sound, sound and sound" in the second training music description.
[0229] When one of the two sample data does not reach the training target, the model parameters of the feature mapping module are adjusted until both sample data reach the training target, and the trained music recognition model is output.
[0230] The following is an example introduction to the training method of the music recognition model provided in the embodiment of the present application.
[0231] Please refer to Figure 6 The music recognition model includes a trained feature extraction sub-model, denoted as Audio encoder; a feature mapping sub-model to be trained, denoted as Adapter; and a trained text recognition sub-model, denoted as LLM.
[0232] The model structure of the feature extraction sub-model is a 24-layer transformer structure, which is used to encode the input 30-second sample music clip, thereby extracting the spectral features of the sample music clip and obtaining the spectral features of the feature extraction sub-model output with a size of (128, 1024). The model parameters of the feature extraction sub-model are obtained by unsupervised training using the collected 170 million 30-second audio data.
[0233] The sample music clip can be data obtained by performing log-mel transformation on the sub-file of the music clip, and the original audio signal can be converted into a log-Mel spectrogram. The Mel spectrogram is a two-dimensional array, in which each column represents a time point and each row represents a Mel frequency. The Mel spectrogram after logarithmic transformation is a log-Mel spectrogram, and the Mel frequency at each time point can be recorded as an audio token. Then, the sample music clip can be represented by multiple audio tokens.
[0234] The model structure of the feature mapping sub-model is a linear transformation layer, which is used to map the spectral features output by the feature extraction sub-model to the text feature space that the text recognition sub-model can recognize. For example, the linear transformation layer is represented by a weight matrix of size (1024, 4096). Then, by multiplying the spectral features with the weight matrix, a descriptive text feature of size (128, 4096) can be obtained. The dense spectral features are mapped to 128 text tokens that are more abstract and semantically rich.
[0235] The text recognition sub-model uses the open source 7B Chinese large model internlm2 model to recognize the input description text features as training music descriptions.
[0236] Thus, the model parameters of the feature extraction sub-model and the model parameters of the text recognition sub-model can be fixed, and based on the difference between the training music description and the corresponding sample music description, the model parameters of the feature mapping sub-model can be adjusted until the training target is reached, thereby obtaining a trained feature mapping sub-model and a trained music recognition model.
[0237] In the embodiment of the present application, by mining music tags, genres, instruments, pitches, metadata, etc., a large number of sample music clips-sample music description pairs are constructed as sample data sets, and multiple music attributes of music are integrated, and music descriptions are generated for multiple music attributes using an open source Chinese large model. Tens of millions of sample data can be accumulated. And no manual annotation is required, which greatly reduces the cost of data annotation.
[0238] Furthermore, by pre-training the alignment of large-scale spectral features to large-model text features, robust and highly accurate descriptive text features were obtained. The large-scale pre-trained model achieved better results than a single modality in multiple downstream tasks, which is a huge improvement.
[0239] Furthermore, there is no need to train a large model, and the large model is used as a trained text recognition model, so that the music recognition model has better accuracy and generalization ability without increasing the difficulty of training, and can be applied to a wider range of fields.
[0240] After the trained music recognition model is obtained by the above training method, the trained music recognition model can be used for music recognition. Figure 1C , the music recognition method provided in the embodiment of the present application is specifically introduced.
[0241] Please refer to Fig. 7A , which is a flow chart of a music recognition method provided in an embodiment of the present application.
[0242] S701, obtaining music to be identified.
[0243] There are many ways to obtain the music to be identified. For example, the music to be identified can be obtained through a sound collection device; for example, the music to be identified can be received from other devices; for example, the music to be identified can be generated based on the received note playing instructions, etc., without specific limitation.
[0244] S702, using the trained music recognition model, extracting the spectral features to be recognized presented by the music to be recognized, and mapping the spectral features to be recognized into text features to be recognized.
[0245] The music recognition model is trained using the training method of the music recognition model introduced above. Then, the trained music recognition model includes a trained feature extraction sub-model, a trained feature mapping sub-model and a trained text recognition sub-model. When the music to be recognized is input into the music recognition model, the feature extraction sub-model can extract features of the music to be recognized and obtain the spectral features to be recognized presented by the music to be recognized. The feature mapping sub-model can map the spectral features to be recognized into the text features to be recognized through linear transformation, for example.
[0246] S703, using a music recognition model to identify features of the text to be recognized, and obtaining a target music description.
[0247] The text recognition sub-model in the music recognition model can identify the text features to be recognized as target music descriptions, and the target music descriptions are used to introduce various target music attributes of the music to be recognized in text form.
[0248] Please refer to Figure 7B In a music application running on a terminal device, when a music recognition button displayed on the display interface of the music application is clicked, the music application calls the music recognition method provided in the embodiment of the present application, and continuously collects music to be recognized through the sound receiving device on the terminal device. When the collected music is less than 30 seconds, the music is filled with 30 seconds of music to be recognized and input into the trained music recognition model; when the collected music is more than 30 seconds, 30 seconds of it is selected as the music to be recognized and input into the trained music recognition model. Thus, the target music description output by the trained music recognition model can be obtained.
[0249] As an embodiment, when inputting into a trained music recognition model, for the same piece of music, a piece of music to be recognized can be generated every 10 seconds of music collected, and each time a piece of music to be recognized is generated, the collected music to be recognized is input into the trained music recognition model once, so that the trained music recognition model can combine the obtained multiple text features to be recognized to generate a more accurate description of the target music.
[0250] Based on the same inventive concept, the present application embodiment provides a music recognition model training device, which can realize the functions corresponding to the aforementioned music recognition model training method. Fig. 8A , the device comprises an acquisition module 81 and a processing module 82, wherein:
[0251] Acquisition module 81: used to acquire a sample data set; wherein each sample data includes: a corresponding sample music segment, and a sample music description including multiple sample music attributes of the sample music segment;
[0252] Processing module 82: for performing multiple rounds of iterative training on the music recognition model to be trained based on the sample data set; wherein the music recognition model includes: a trained feature extraction sub-model and a text recognition sub-model, and a feature mapping sub-model to be trained; each round of iterative training includes:
[0253] The processing module 82 is specifically used to: use a feature extraction sub-model to extract the frequency spectrum features presented by the sample music clip;
[0254] The processing module 82 is specifically used to: use a feature mapping sub-model to map the obtained spectral features into descriptive text features; wherein the descriptive text features are used to describe a variety of training music attributes of the sample audio segment;
[0255] The processing module 82 is specifically used to: use the text recognition sub-model to identify the description text features and obtain the training music description;
[0256] The processing module 82 is specifically used to adjust the model parameters of the feature mapping module based on the difference between the obtained training music description and the corresponding sample music description.
[0257] In a possible embodiment, the acquisition module 81 is specifically used for:
[0258] Collecting multimedia files of a plurality of music, and collecting metadata of each multimedia file;
[0259] Based on the preset segment duration, each multimedia file is segmented to obtain a plurality of sub-files corresponding to the corresponding multimedia file, and each obtained sub-file is used as a corresponding sample music segment;
[0260] Using the trained multimodal recognition model, respectively perform music attribute recognition on each obtained sample music clip to obtain multiple sample music attributes of the corresponding sample music clip;
[0261] Using the trained large language model, based on the obtained metadata and the sample music attributes, respectively generate sample music descriptions for each sample music clip;
[0262] Based on each sample music segment and its corresponding sample music description, a sample data set is obtained.
[0263] In a possible embodiment, the processing module 82 is further configured to:
[0264] Before performing multiple rounds of iterative training on the music recognition model to be trained based on the sample data set, a feature extraction sub-model and a feature mapping sub-model to be trained are established based on the structural construction strategy of the music recognition model, and a trained text recognition sub-model is obtained;
[0265] Based on the collected multiple audio data, the feature extraction sub-model to be trained is trained for multiple rounds of iterations, and the trained feature extraction sub-model is output;
[0266] Based on the trained feature extraction sub-model and text recognition sub-model, and the feature mapping sub-model to be trained, a music recognition model to be trained is established.
[0267] In a possible embodiment, the processing module 82 is specifically used for:
[0268] In each training iteration, the following operations are performed:
[0269] Perform sub-data masking on the audio data to obtain masked data that masks the random sub-data;
[0270] Perform feature extraction on the masked data to obtain data features;
[0271] Based on the data features, predict the masked random sub-data in the masked data to obtain the predicted sub-data;
[0272] Based on the obtained differences between the predicted sub-data and the random sub-data, the model parameters of the feature extraction sub-model are adjusted.
[0273] In a possible embodiment, the processing module 82 is specifically used for:
[0274] Using a feature extraction sub-model, extracting the spectrum sub-features of each of the multiple sub-segments contained in the sample music segment, and extracting the spectrum correlation features between the spectrum sub-features;
[0275] The obtained spectral sub-features and spectral correlation features are integrated to obtain the spectral features presented by the sample music clip.
[0276] In a possible embodiment, the spectral sub-features represent: at least one of the pitch, timbre, sound receiving environment, beat, instrument and emotion of the corresponding sub-segment; the spectral association features represent: at least one of the similarity, rhythmic relationship and pitch conversion between the sub-segments.
[0277] In a possible embodiment, the processing module 82 is specifically used for:
[0278] Based on the model parameters of the feature mapping sub-model in this round of training, a first mapping relationship between each attribute spectrum feature learned by the feature mapping sub-model and each attribute text feature is obtained; wherein each attribute text feature represents one of a plurality of reference music attributes; the plurality of reference music attributes at least include: a union of a plurality of sample music attributes and a plurality of training music attributes;
[0279] Based on the first mapping relationship, the spectrum feature is mapped into a plurality of attribute text features; wherein the plurality of attribute text features represent a plurality of training music attributes;
[0280] Multiple attribute text features are obtained by fusing to generate description text features.
[0281] In a possible embodiment, the processing module 82 is further configured to:
[0282] Before the obtained spectral features are mapped to descriptive text features by using the feature mapping sub-model, based on a preset second mapping relationship between each data format and each occupied data amount, a target data format whose corresponding occupied data amount is smaller than the reference data amount is selected from each data format; wherein the parameter data amount is: the occupied data amount corresponding to the data format used by the model parameter of the feature mapping sub-model;
[0283] The spectrum features are converted into a target data format to obtain the converted spectrum features.
[0284] In a possible embodiment, the processing module 82 is specifically used for:
[0285] Get the memory usage of the video memory used to run this round of training;
[0286] When it is determined that the video memory occupancy is greater than the preset occupancy, based on the preset second mapping relationship between each data format and each occupied data amount, a target data format whose corresponding occupied data amount is smaller than the reference data amount is selected from each data format.
[0287] In a possible embodiment, the processing module 82 is specifically used for:
[0288] Determining a mapping training loss based on differences between a plurality of training music attributes included in the obtained training music description and a plurality of sample music attributes included in the corresponding sample music description;
[0289] Based on the mapping training loss, the model parameters of the feature mapping module are adjusted.
[0290] In a possible embodiment, the processing module 82 is specifically used for:
[0291] determining a semantic training loss based on the obtained textual similarity between the training music descriptions and the corresponding sample music descriptions;
[0292] Based on the mapping training loss and the semantic training loss, when it is determined that the training target is not achieved, the model parameters of the feature mapping module are adjusted.
[0293] Based on the same inventive concept, the present application embodiment provides a music recognition device that can realize the functions corresponding to the aforementioned music recognition method. Figure 8B , the device includes an acquisition module 801 and a processing module 802, wherein:
[0294] Acquisition module 801: used to acquire music to be identified;
[0295] Processing module 802: used to use a trained music recognition model to extract the spectral features to be recognized presented by the music to be recognized, and map the spectral features to be recognized to text features to be recognized; wherein the music recognition model is trained by the method of the first aspect;
[0296] The processing module 802 is also used to: use a music recognition model to identify the text features to be recognized and obtain a target music description; wherein the target music description is used to: introduce multiple target music attributes of the music to be recognized in text form.
[0297] Please refer to Fig. 9 , is a computer device 900 provided in an embodiment of the present application, and the computer device 900 can be, for example, Figure 1C The client 101 or the server 102 in the data storage program. The current version and the historical version of the data storage program and the application software corresponding to the data storage program can be installed on the computer device 900, and the computer device 900 includes a processor 980 and a memory 920. In some embodiments, the computer device 900 may include a display unit 940, and the display unit 940 includes a display panel 941 for displaying a user interactive operation interface, etc.
[0298] In a possible embodiment, the display panel 941 may be configured in the form of a liquid crystal display (LCD) or an organic light-emitting diode (OLED).
[0299] The processor 980 is used to read the computer program and then execute the method defined by the computer program. For example, the processor 980 reads the data storage program or file, so as to run the data storage program on the computer device 900 and display the corresponding interface on the display unit 940. The processor 980 may include one or more general-purpose processors and may also include one or more DSPs (Digital Signal Processors) to perform related operations to implement the technical solutions provided in the embodiments of the present application.
[0300] The memory 920 generally includes internal memory and external memory. The internal memory may be a random access memory (RAM), a read-only memory (ROM), and a cache (CACHE), etc. The external memory may be a hard disk, an optical disk, a USB disk, a floppy disk, or a tape drive, etc. The memory 920 is used to store computer programs and other data. The computer program includes applications corresponding to each client, etc. Other data may include data generated after the operating system or application is run, and the data includes system data (such as configuration parameters of the operating system) and user data. In the embodiment of the present application, the computer program is stored in the memory 920, and the processor 980 executes the computer program in the memory 920 to implement any of the methods discussed in the previous figure.
[0301] The display unit 940 is used to receive input digital information, character information or contact touch operation / contactless gesture, and generate signal input related to user settings and function control of the computer device 900. Specifically, in the embodiment of the present application, the display unit 940 may include a display panel 941. The display panel 941 is, for example, a touch screen, which can collect user touch operations on or near it (such as operations performed by the user using fingers, stylus, or any other suitable object or accessory on or on the display panel 941), and drive corresponding connection devices according to a pre-set program.
[0302] In a possible embodiment, the display panel 941 may include two parts: a touch detection device and a touch controller. The touch detection device detects the touch position of the player, detects the signal brought by the touch operation, and transmits the signal to the touch controller; the touch controller receives the touch information from the touch detection device, converts it into the touch point coordinates, and then sends it to the processor 980, and can receive and execute the command sent by the processor 980.
[0303] The display panel 941 may be implemented in various types such as resistive, capacitive, infrared, and surface acoustic wave. In addition to the display unit 940, in some embodiments, the computer device 900 may further include an input unit 930, which may include an image input device 931 and other input devices 932, wherein the other input devices may include, but are not limited to, one or more of a physical keyboard, a function key (such as a volume control key, a switch key, etc.), a trackball, a mouse, a joystick, and the like.
[0304] In addition to the above, the computer device 900 may also include a power supply 990 for supplying power to other modules, an audio circuit 960, a near field communication module 970, and an RF circuit 910. The computer device 900 may also include one or more sensors 950, such as an acceleration sensor, a light sensor, a pressure sensor, etc. The audio circuit 960 specifically includes a speaker 961 and a microphone 962, etc. For example, the computer device 900 can collect the user's voice through the microphone 962 to perform corresponding operations, etc.
[0305] As an embodiment, the number of the processors 980 may be one or more, and the processor 980 and the memory 920 may be coupled or relatively independently configured.
[0306] As an example, Fig. 9 The processor 980 in the embodiment can be used to implement the following Fig. 8A The functions of the acquisition module 81 and the processing module 82 in the embodiment can also be used to implement the following Figure 8B The functions of the acquisition module 801 and the processing module 802 in .
[0307] As an example, Fig. 9 The processor 980 in can be used to implement the corresponding functions of the server or terminal device discussed above.
[0308] A person of ordinary skill in the art can understand that: all or part of the steps of implementing the above method embodiments can be completed by a computer program, and the aforementioned computer program can be stored in a computer-readable storage medium. When the computer program is executed, it executes the steps of the above method embodiments; and the aforementioned storage medium includes: a mobile storage device, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and other media that can store program codes.
[0309] Alternatively, if the above-mentioned integrated unit of the present invention is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present invention can be essentially or partly reflected in the form of a software product, for example, through a computer program product, which is stored in a storage medium, including a computer program for a computer device to execute all or part of the methods described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, ROM, RAM, magnetic disks or optical disks.
[0310] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.
Claims
1. A method for training a music recognition model, characterized in that: include: Acquire a sample data set; wherein each of the sample data includes: a corresponding sample music segment, and a sample music description including multiple sample music attributes of the sample music segment; Based on the sample data set, the music recognition model to be trained is trained for multiple rounds of iterative training; wherein the music recognition model includes: a trained feature extraction sub-model and a text recognition sub-model, and a feature mapping sub-model to be trained; each round of iterative training includes: Using the feature extraction sub-model, extracting the frequency spectrum features presented by the sample music clip; Based on the model parameters of the feature mapping sub-model in this round of training, a first mapping relationship between each attribute spectrum feature and each attribute text feature learned by the feature mapping sub-model is obtained; wherein each attribute text feature represents one of a plurality of reference music attributes; Based on the first mapping relationship, the obtained spectrum feature is mapped into a plurality of attribute text features; wherein the plurality of attribute text features represent a plurality of training music attributes among the plurality of reference music attributes; Fusion of the obtained multiple attribute text features to generate description text features; wherein the description text features are used to describe the multiple training music attributes of the sample music segment; Using the text recognition sub-model, identifying the features of the description text, and obtaining a training music description; Based on the difference between the obtained training music description and the corresponding sample music description, the model parameters of the feature mapping sub-model are adjusted.
2. The method according to claim 1, characterized in that The obtaining of the sample data set comprises: Collecting multimedia files of a plurality of music and collecting metadata of each multimedia file; Based on the preset segment duration, the multimedia files are divided into segments to obtain a plurality of sub-files corresponding to the corresponding multimedia files, and the obtained sub-files are used as corresponding sample music segments; Using the trained multimodal recognition model, respectively perform music attribute recognition on each obtained sample music clip to obtain multiple sample music attributes of the corresponding sample music clip; Using the trained large language model, based on the obtained metadata and the sample music attributes, respectively generate sample music descriptions for the sample music clips; Based on the sample music clips and their corresponding sample music descriptions, a sample data set is obtained.
3. The method according to claim 1, characterized in that: Before performing multiple rounds of iterative training on the music recognition model to be trained based on the sample data set, the method further includes: Based on the structural building strategy of the music recognition model, a feature extraction sub-model and a feature mapping sub-model to be trained are established, and a trained text recognition sub-model is obtained; Based on the collected multiple audio data, the feature extraction sub-model to be trained is trained for multiple rounds of iterations, and the trained feature extraction sub-model is output; Based on the trained feature extraction sub-model and text recognition sub-model, and the feature mapping sub-model to be trained, a music recognition model to be trained is established.
4. The method according to claim 3, characterized in that Based on the collected multiple audio data, the feature extraction sub-model to be trained is trained for multiple rounds of iterative training, including: In each training iteration, the following operations are performed: Performing sub-data masking on the audio data to obtain masked data that masks the random sub-data; Performing feature extraction on the masked data to obtain data features; Based on the data feature, predict the random sub-data masked in the masked data to obtain predicted sub-data; Based on the difference between the obtained predicted sub-data and the random sub-data, the model parameters of the feature extraction sub-model are adjusted.
5. The method according to any one of claims 1 to 4, characterized in that: The extracting the frequency spectrum features presented by the sample music clip by using the feature extraction sub-model includes: Using the feature extraction sub-model, extracting the spectrum sub-features of each of the multiple sub-segments contained in the sample music segment, and extracting the spectrum correlation features between the spectrum sub-features; The obtained spectral sub-features and spectral correlation features are integrated to obtain the spectral features presented by the sample music clip.
6. The method according to claim 5, characterized in that The spectrum sub-feature representation includes: at least one of the pitch, timbre, sound receiving environment, beat, instrument and emotion of the corresponding sub-segment; the spectrum association feature representation includes: at least one of the similarity, rhythm relationship and pitch conversion between the sub-segments.
7. The method according to any one of claims 1 to 4, characterized in that: Before mapping the obtained spectrum feature into a plurality of attribute text features based on the first mapping relationship, the method further includes: Based on a preset second mapping relationship between each data format and each occupied data amount, a target data format whose corresponding occupied data amount is smaller than a reference data amount is selected from the data formats; wherein the reference data amount is: the occupied data amount corresponding to the data format used by the model parameters of the feature mapping sub-model; The frequency spectrum feature is converted into the target data format to obtain the converted frequency spectrum feature.
8. The method according to claim 7, characterized in that The selecting, from the data formats based on the preset second mapping relationship between each data format and each occupied data amount, a target data format whose corresponding occupied data amount is smaller than the reference data amount comprises: Get the memory usage of the video memory used to run this round of training; When it is determined that the video memory occupancy is greater than the preset occupancy, based on a preset second mapping relationship between each data format and each occupied data amount, a target data format whose corresponding occupied data amount is smaller than the reference data amount is selected from the data formats.
9. The method according to any one of claims 1 to 4, characterized in that: The adjusting the model parameters of the feature mapping sub-model based on the difference between the obtained training music description and the corresponding sample music description comprises: Determining a mapping training loss based on differences between a plurality of training music attributes included in the obtained training music description and a plurality of sample music attributes included in the corresponding sample music description; Based on the mapping training loss, model parameters of the feature mapping sub-model are adjusted.
10. The method according to claim 9, characterized in that The adjusting the model parameters of the feature mapping sub-model based on the mapping training loss includes: determining a semantic training loss based on the obtained textual similarity between the training music descriptions and the corresponding sample music descriptions; When it is determined based on the mapping training loss and the semantic training loss that the training target has not been achieved, the model parameters of the feature mapping sub-model are adjusted.
11. A music recognition method, characterized in that: include: Get the music to be identified; Using a trained music recognition model, extracting the spectral features to be recognized presented by the music to be recognized, and mapping the spectral features to be recognized into text features to be recognized; wherein the music recognition model is trained using the method according to any one of claims 1 to 10; The music recognition model is used to identify the text features to be recognized and obtain a target music description; wherein the target music description is used to introduce multiple target music attributes of the music to be recognized in text form.
12. A training device for a music recognition model, characterized in that: include: Acquisition module: used to acquire sample data sets; wherein each of the sample data includes: a corresponding sample music segment, and a sample music description including multiple sample music attributes of the sample music segment; Processing module: used for performing multiple rounds of iterative training on the music recognition model to be trained based on the sample data set; wherein the music recognition model includes: a trained feature extraction sub-model and a text recognition sub-model, and a feature mapping sub-model to be trained; each round of iterative training includes: The processing module is specifically used to: use the feature extraction sub-model to extract the frequency spectrum features presented by the sample music clip; The processing module is specifically used for: Based on the model parameters of the feature mapping sub-model in this round of training, a first mapping relationship between each attribute spectrum feature and each attribute text feature learned by the feature mapping sub-model is obtained; wherein each attribute text feature represents one of a plurality of reference music attributes; Based on the first mapping relationship, the obtained spectrum feature is mapped into a plurality of attribute text features; wherein the plurality of attribute text features represent a plurality of training music attributes among the plurality of reference music attributes; Fusion of the obtained multiple attribute text features to generate description text features; wherein the description text features are used to describe the multiple training music attributes of the sample music segment; The processing module is specifically used to: use the text recognition sub-model to identify the description text features and obtain the training music description; The processing module is specifically used to adjust the model parameters of the feature mapping sub-model based on the difference between the obtained training music description and the corresponding sample music description.
13. A music recognition device, characterized in that: include: Acquisition module: used to acquire music to be identified; Processing module: used to use a trained music recognition model to extract the spectral features to be recognized presented by the music to be recognized, and map the spectral features to be recognized into text features to be recognized; wherein the music recognition model is trained by the method according to any one of claims 1 to 10; The processing module is also used to: use the music recognition model to identify the text features to be recognized and obtain a target music description; wherein the target music description is used to: introduce multiple target music attributes of the music to be recognized in text form.
14. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the method according to any one of claims 1 to 11 is implemented.
15. A computer device, characterized in that: include: A memory for storing program instructions; A processor, configured to call the program instructions stored in the memory, and execute the method according to any one of claims 1 to 11 according to the obtained program instructions.
16. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are used to enable a computer to execute the method according to any one of claims 1 to 11.
