Model training method and device, audio processing method and device, medium and program product

Abstract

The invention discloses a model training and audio processing method and device, a medium and a program product in the technical field of computers, and the method comprises the steps: carrying out the combined training based on an auxiliary training model and an initial music model, and obtaining a target music model suitable for processing an audio task; wherein the feature extraction capability of the auxiliary training model can ensure the accurate coding of the music features, and the compression processing can further perform feature compression extraction on the second music features extracted by the auxiliary training model, so that the expression of the music features is further simplified. According to the target music model obtained through final training, the structure simplification of the music model is realized, the feature coding capability of the music model is improved, the music model is more suitable for being applied to a specific category of audio tasks, and the generalization capability of the model is enhanced.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and particularly relates to a model training, audio processing method, device, medium and program product. Background Art

[0002] Currently, based on a pre-trained model, specific modules in the pre-trained model can be directly trained to implement audio tasks such as song quality evaluation and song classification. However, since the parameters of other modules in the pre-trained model remain unchanged except for the specific modules, problems such as feature distortion or catastrophic forgetting may occur; moreover, the specific modules in the pre-trained model may not be applicable to song-related audio tasks, which limits the application of the model in audio tasks such as song quality evaluation and song classification. Summary of the Invention

[0003] In view of this, the purpose of the present application is to provide a model training, audio processing method, device, medium and program product to train a model suitable for specific category audio tasks. The specific solutions are as follows:

[0004] To achieve the above purpose, on the one hand, the present application provides a model training method, including:

[0005] Performing feature extraction on audio data corresponding to an audio task through an initial music model to obtain a first music feature, and generating an initial prediction result based on the first music feature;

[0006] Performing feature extraction on the audio data through an auxiliary training model to obtain a second music feature, and performing compression processing on the second music feature to obtain a third music feature;

[0007] Training the initial music model based on an initial loss value between the initial prediction result and a training label of the audio data, and a feature loss value between the first music feature and the third music feature until a preset training stop condition is met, to obtain a trained target music model; the target music model is used to process the audio task.

[0008] Optionally, the initial music model includes an initial feature extraction module and an initial classifier;

[0009] Correspondingly, performing feature extraction on audio data corresponding to an audio task through an initial music model to obtain a first music feature, and generating an initial prediction result based on the first music feature, includes:

[0010] Performing feature extraction on the audio data through the initial feature extraction module to obtain the first music feature;

[0011] Classify the first music feature through the initial classifier to obtain the initial prediction result;

[0012] Correspondingly, train the initial music model based on the initial loss value between the initial prediction result and the training label of the audio data, and the feature loss value between the first music feature and the third music feature, including:

[0013] Calculate a comprehensive loss value based on the initial loss value and the feature loss value;

[0014] Update the parameters of the initial feature extraction module and the parameters of the initial classifier respectively based on the comprehensive loss value.

[0015] Optionally, the auxiliary training model includes: a feature auxiliary module and a compression module;

[0016] Correspondingly, extract features from the audio data through the auxiliary training model to obtain a second music feature, and perform compression processing on the second music feature to obtain a third music feature, including:

[0017] Extract features from the audio data through the feature auxiliary module to obtain the second music feature;

[0018] Perform compression processing on the second music feature through the compression module to obtain the third music feature.

[0019] Optionally, the auxiliary training model further includes: an auxiliary classifier;

[0020] Correspondingly, the method further includes:

[0021] Classify the third music feature through the auxiliary classifier to obtain an auxiliary prediction result;

[0022] Train the auxiliary classifier and the compression module based on the auxiliary loss value between the auxiliary prediction result and the training label.

[0023] Optionally, the generation process of the feature auxiliary module includes:

[0024] Obtain an audio set;

[0025] Adjust the model parameters of the pre-trained model using the audio set until the pre-trained model converges;

[0026] Use the converged pre-trained model as the feature auxiliary module.

[0027] Optionally, before using the converged pre-trained model as the feature auxiliary module, it further includes:

[0028] In a variety of audio tasks, evaluate the music feature encoding ability of the pre-trained model after convergence;

[0029] If the evaluation results of the pre-trained model after convergence in a variety of audio tasks meet the expected requirements, perform the step of using the pre-trained model after convergence as the feature assistance module;

[0030] If the evaluation results of the pre-trained model after convergence in a variety of audio tasks do not meet the expected requirements, enhance the sample quality of the audio set; replace the audio set with the audio set with enhanced sample quality, replace the pre-trained model with the pre-trained model after convergence, and then perform the step of adjusting the model parameters of the pre-trained model using the audio set until the pre-trained model converges and subsequent steps.

[0031] Optionally, compressing the second music feature through the compression module to obtain the third music feature includes:

[0032] Determine the key audio features corresponding to the audio task;

[0033] Use the compression module to extract the key audio features from the second music feature as the third music feature.

[0034] Optionally, the satisfaction of the preset training stop condition includes:

[0035] Reaching the preset maximum number of iterations; or the changes in the initial loss value and the feature loss value meet the expectations.

[0036] On the other hand, the present application also provides an audio processing method, including:

[0037] Obtain audio data;

[0038] Input the audio data into the target music model to use the target music model to output the classification result of the audio data; the target music model is obtained based on any one of the foregoing methods.

[0039] Optionally, it further includes:

[0040] Determine the category to which the audio data belongs according to the classification result;

[0041] Search for songs belonging to the category, and / or recommend songs belonging to the category.

[0042] On the other hand, the present application also provides an electronic device, the electronic device includes a processor and a memory; wherein, the memory is used to store a computer program, and the computer program is loaded and executed by the processor to implement any one of the foregoing methods.

[0043] In another aspect, the present application also provides a storage medium storing computer-executable instructions, which, when loaded and executed by a processor, implement the method described in any one of the foregoing items.

[0044] In another aspect, the present application also provides a computer program product, including a computer program / instructions, characterized in that when the computer program / instructions are executed by a processor, the method described in any one of the foregoing items is implemented.

[0045] Based on an auxiliary training model and an initial music model, the present application jointly trains to obtain a target music model suitable for processing audio tasks; among them, the feature extraction ability of the auxiliary training model can ensure the accurate encoding of music features, and the compression process can further compress and refine the features of the second music features extracted by the auxiliary training model, thereby further streamlining the expression of music features. Among them, based on the initial loss value between the initial prediction result of the initial music model and the training label of the audio data, and the feature loss value between the first music features of the initial music model and the third music features of the auxiliary training model, the parameters of the initial music model are trained, so that the initial music model can learn the music feature encoding ability of the auxiliary training model, streamline the expression of music features, not only realizes the structural streamlining of the music model, but also improves the feature encoding ability of the music model, and also makes the music model more suitable for application in specific category audio tasks, such as audio tasks like song quality evaluation and song classification, enhancing the generalization ability of the model.

[0046] Correspondingly, an audio processing method, device, storage medium and program product provided by the present application also have the above technical effects. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the drawings in the following description are only the embodiments of the present application. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained according to the provided drawings.

[0048] Figure 1 It is a schematic diagram of a physical architecture applicable to the present application provided by the present application;

[0049] Figure 2 It is a flowchart of a model training method provided by the present application;

[0050] Figure 3 It is a schematic diagram of a model training solution provided by the present application;

[0051] Figure 4 Schematic diagram of the application process of an initial music model provided by this application;

[0052] Figure 5 Flowchart of an audio processing method provided by this application;

[0053] Figure 6 Server structure diagram provided by this application;

[0054] Figure 7 Terminal structure diagram provided by this application. Detailed implementation manners

[0055] Next, the technical solutions in the embodiments of this application will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of this application. Obviously, the described embodiments are only a part of the embodiments of this application, rather than all the embodiments. Based on the embodiments in this application, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of this application. Additionally, in the embodiments of this application, "first", "second", etc. are used to distinguish similar objects, and do not necessarily need to describe a specific order or sequence.

[0056] Currently, based on a pre-trained model, specific modules in the pre-trained model can be directly trained to implement audio tasks such as song quality evaluation and song classification. However, since the parameters of other modules except the specific modules in the pre-trained model remain unchanged, problems such as feature distortion or catastrophic forgetting may occur; moreover, the specific modules in the pre-trained model may not be applicable to song-like audio tasks, which will limit the application of the model in audio tasks such as song quality evaluation and song classification.

[0057] In view of the above problems currently existing, this application proposes a model training scheme. This scheme is based on an auxiliary training model and an initial music model, and jointly trains to obtain a music model suitable for processing audio tasks; moreover, the auxiliary training model has the ability of music feature encoding in advance, which can ensure the accurate encoding of music features, and the compression processing therein can further compress and extract the initial music features of the audio data encoded by the auxiliary training model, thereby further streamlining the expression of music features. Using the finally converged music model as the music model for processing audio tasks not only realizes the structural streamlining of the music model but also improves the feature encoding ability of the music model.

[0058] For ease of understanding, the physical framework applicable to this application will be introduced first.

[0059] It should be understood that the model training method provided in this application can be applied to a system or program with model training capabilities. Specifically, a system or program with model training capabilities can run on devices such as servers and personal computers. Correspondingly, the audio processing method provided in this application can also be applied to a system or program with audio processing capabilities. Specifically, a system or program with audio processing capabilities can run on devices such as servers and personal computers.

[0060] As Figure 1 shown, Figure 1 is a schematic diagram of the physical architecture applicable to this application. In Figure 1 , a system or program with model training and audio processing capabilities can run on a server, and the server is used to implement the model training and audio processing methods.

[0061] It can be seen from Figure 1 that the server can establish communication connections with multiple devices, and the server obtains the required data such as the model structure and audio data corresponding to the audio task from these devices. The server then executes the model training and audio processing methods accordingly.

[0062] Figure 1 shows a variety of terminal devices. In actual scenarios, there may be more or fewer types of terminal devices participating in the model training and audio processing processes. The specific quantity and types depend on the actual scenario and are not limited here. Additionally, Figure 1 shows one server, but in actual scenarios, multiple servers may also participate. The specific number of servers depends on the actual scenario.

[0063] It should be noted that the model training and audio processing methods provided in this embodiment can be performed offline, that is, the server locally stores the required data such as the model structure and audio data corresponding to the audio task, and it can directly use the solution provided in this application to implement the model training and audio processing methods.

[0064] It can be understood that the above-mentioned system and program with model training and audio processing capabilities can be a type of cloud service program, and the specific operation mode depends on the actual scenario and is not limited here.

[0065] Specifically, after the model training is completed, the obtained music model can be used for processing various audio tasks such as singer tone classification, singing skill classification, musical instrument classification tasks, and song genre classification.

[0066] Combining the above commonalities, please refer to Figure 2 , Figure 2 which is a flowchart of a model training method provided in an embodiment of this application. As Figure 2 shown, the model training method may include the following steps:

[0067] S201. Extract features from the audio data corresponding to the audio task through the initial music model to obtain the first music feature, and generate an initial prediction result based on the first music feature.

[0068] In this embodiment, the audio task can be various music-related tasks such as a song quality evaluation task, a singer timbre classification task, a singing skill classification task, an instrument classification task, a song genre classification task, etc. The audio data corresponding to the audio task can be a song, pure music, etc. The initial music model mainly performs two operations, one is feature extraction, and the other is feature classification. Therefore, the initial music model includes an initial feature extraction module and an initial classifier. The initial classifier mainly consists of a fully connected layer. Stacking the sigmoid function can achieve a binary classification task, and stacking the softmax function can achieve a multi-classification task. In one implementation, the initial music model includes an initial feature extraction module and an initial classifier; correspondingly, extracting features from the audio data corresponding to the audio task through the initial music model to obtain the first music feature, and generating an initial prediction result based on the first music feature includes: extracting features from the audio data through the initial feature extraction module to obtain the first music feature; classifying the first music feature through the initial classifier to obtain the initial prediction result (i.e., the classification result of the first music feature). Music features can include: phoneme features, pitch features, and / or melody features, etc.

[0069] S202. Extract features from the audio data through the auxiliary training model to obtain the second music feature, and perform compression processing on the second music feature to obtain the third music feature.

[0070] It should be noted that the auxiliary training model mainly performs three operations, one is feature extraction, the other is feature compression, and the third is feature classification. Therefore, the auxiliary training model includes a feature auxiliary module for implementing feature extraction, a compression module, and a classifier. Among them, the feature auxiliary module has the ability to encode music features. The music feature encoding ability of the feature auxiliary module is pre-trained based on a pre-trained model. The pre-trained model can be selected from MERT, MULE, etc. In one implementation, the auxiliary training model includes: a feature auxiliary module and a compression module; correspondingly, extracting features from the audio data through the auxiliary training model to obtain the second music feature, and performing compression processing on the second music feature to obtain the third music feature includes: extracting features from the audio data through the feature auxiliary module to obtain the second music feature; performing compression processing on the second music feature through the compression module to obtain the third music feature.

[0071] In this embodiment, the same audio data is input to the auxiliary training model and the initial music model. The second music feature of the audio data is obtained by utilizing the strong encoding ability of the feature auxiliary module in the auxiliary training model, and the first music feature of the audio data is obtained at a relatively fast speed by using the initial feature extraction module in the structurally lightweight initial music model. Theoretically, the second music feature contains more information and a larger data volume than the first music feature. Therefore, in this embodiment, the compression module is used to remove the redundant information in the second initial feature and extract and retain the effective information (the information more suitable for the current audio task). It can be seen that the compression module is used to achieve feature alignment, removal of redundant information, and extraction and retention of effective information. Correspondingly, the second music feature is compressed by the compression module to obtain the third music feature, including: determining the key audio feature corresponding to the audio task; using the compression module to extract the key audio feature from the second music feature as the third music feature. The key audio feature is the information more suitable for the current audio task.

[0072] In one implementation, the generation process of the feature auxiliary module includes: obtaining an audio set; adjusting the model parameters of the pre-trained model using the audio set until the pre-trained model converges; using the converged pre-trained model as the feature auxiliary module. Among them, the audio set may include various music audio data such as numerous songs and pure music audio, enabling the pre-trained model to learn the encoding ability of various music audio data.

[0073] To make the feature assistance module applicable to various types of audio tasks, in this embodiment, after the pre-trained model converges, the converged pre-trained model can be used to process various audio tasks to evaluate the music feature encoding ability of the converged pre-trained model; if the converged pre-trained model shows good music feature encoding ability in various audio tasks, then the converged pre-trained model can be used as the feature assistance module; if the converged pre-trained model only shows good music feature encoding ability in individual audio tasks and has poor music feature encoding ability in most audio tasks, then the audio set can be enriched to continue to adjust and update the model parameters of the converged pre-trained model. In one implementation, before using the converged pre-trained model as the feature assistance module, it further includes: evaluating the music feature encoding ability of the converged pre-trained model in various audio tasks; if the evaluation results of the converged pre-trained model in various audio tasks meet the expected requirements, then perform the step of using the converged pre-trained model as the feature assistance module; if the evaluation results of the converged pre-trained model in various audio tasks do not meet the expected requirements, then enhance the sample quality of the audio set; after replacing the audio set with the audio set with enhanced sample quality and replacing the pre-trained model with the converged pre-trained model, perform the steps of adjusting the model parameters of the pre-trained model using the audio set until the pre-trained model converges and subsequent steps. Among them, enhancing the sample quality of the audio set includes: adding new samples to the audio set, removing repetitive samples in the audio set, and / or performing processing such as mixing old samples in the audio set to obtain new samples.

[0074] S203. Based on the initial loss value between the initial prediction result and the training label of the audio data, and the feature loss value between the first music feature and the third music feature, train the initial music model until the preset training stop condition is met to obtain the trained target music model.

[0075] Among them, the target music model is used to process various music tasks such as song quality evaluation tasks, singer timbre classification tasks, singing skill classification tasks, instrument classification tasks, and song genre classification tasks.

[0076] When the initial music model includes an initial feature extraction module and an initial classifier, then training the initial music model means: training the initial feature extraction module and the initial classifier. Then, based on the initial loss value between the initial prediction result and the training label of the audio data, and the feature loss value between the first music feature and the third music feature, training the initial music model includes: calculating a comprehensive loss value based on the initial loss value and the feature loss value; updating the parameters of the initial feature extraction module and the parameters of the initial classifier respectively based on the comprehensive loss value.

[0077] When the auxiliary training model includes: feature auxiliary module, compression module and auxiliary classifier, the third music feature output by the compression module is classified and processed by the auxiliary classifier to obtain the auxiliary prediction result; based on the auxiliary loss value between the auxiliary prediction result and the training label of the audio data, the auxiliary classifier and the compression module are trained to enable the compression module to learn the feature alignment ability, the redundant information removal ability and the effective information extraction and retention ability, so that the auxiliary classifier learns the classification ability of the third music feature. If the preset maximum number of iterations is reached; or the change of the auxiliary loss value is in line with expectations, it is considered that the auxiliary classifier and the compression module have completed convergence.

[0078] In one example, the specific structures of the initial feature extraction module and the feature auxiliary module can use conventional related structures. For example, the initial feature extraction module uses Short-chunk CNN with residual connection or Harmonic CNN. The initial feature extraction module has a simpler structure than the feature auxiliary module and can achieve music feature encoding more quickly.

[0079] It should be noted that the pre-trained model can be trained with a large amount of audio first, so that the pre-trained model has a strong ability to extract and encode music features such as phoneme features, pitch features, and melody features. After determining the specific task to be trained (such as the task of evaluating the sound quality of songs), the input audio is encoded using the feature auxiliary module trained by the aforementioned pre-trained model, and the encoding result is compressed using the compression module, and the compressed result is classified into good or bad sound quality using the auxiliary classifier, and the auxiliary loss value is calculated. In this process, the auxiliary loss value is used to update the parameters of the auxiliary classifier and the compression module; at the same time, the input audio is encoded using the initial feature extraction module in the initial music model to obtain the first music feature, and the first music feature is classified using the initial classifier in the initial music model, and the initial loss value between the initial prediction result and the training label of the audio data is calculated, and the feature loss value between the first music feature and the third music feature output by the compression module is calculated. The initial feature extraction module and the initial classifier are trained with these two loss values ​​until the preset training stop condition is met to obtain a trained target music model. That is, the classification loss of the auxiliary training model is used to update the parameters of the compression module and the auxiliary classifier; the classification loss of the initial music model and the knowledge distillation loss (that is, the feature loss value) are used to update the parameters of the initial feature extraction module and the initial classifier in the initial music model.

[0080] It can be seen that the compression module, the auxiliary classifier, the initial feature extraction module, and the initial classifier can be trained synchronously, that is: a batch of audio data is synchronously input to the auxiliary training model and the initial music model, so as to perform a synchronous update on the compression module, the auxiliary classifier, the initial feature extraction module, and the initial classifier. Repeat this process until the maximum number of iterations is reached or the changes in each loss value are not significant. It is considered that the compression module, the auxiliary classifier, the initial feature extraction module, and the initial classifier have completed convergence. Then, the current initial music model is used as the music model for processing audio tasks.

[0081] In this embodiment, the training label of the audio data is the standard classification result corresponding to the current audio task. For example: when the current audio task is a song quality evaluation task, the training label of the audio data is the true score of the audio data; when the current audio task is a timbre classification task, the training label of the audio data is the true timbre category to which the audio data belongs. This embodiment adjusts the parameters of the initial music model by using the initial loss value between the initial prediction result and the training label of the audio data, and the feature loss value between the first music feature and the third music feature, so that the initial music model can learn the music feature encoding ability of the auxiliary training model, and the amount of feature data output by the initial music model is small and the features are accurate, which is more conducive to the subsequent classification processing of the classifier. After the initial music model converges; the finally converged music model is used as the music model for processing audio tasks. Subsequently, the corresponding audio data can be directly classified using the music model, that is: the audio data is input into the music model so that the music model outputs the classification result of the audio data.

[0082] In this embodiment, the maximum number of iterations can be preset. If the training reaches the maximum number of iterations or the change in the corresponding loss value meets the expectation, then the current music model is used as the trained target music model. If the training does not reach the maximum number of iterations or the change in the corresponding loss value does not meet the expectation, then more model training is continued until the initial music model completes convergence. In one implementation manner, meeting the preset training stop condition includes: reaching the preset maximum number of iterations; or the changes in the initial loss value and the feature loss value meet the expectation.

[0083] It can be seen that based on the auxiliary training model and the initial music model, the target music model suitable for processing audio tasks is jointly trained; among them, the feature extraction ability of the auxiliary training model can ensure the accurate encoding of music features, and the compression process can further compress and refine the second music features extracted by the auxiliary training model, thereby further streamlining the expression of music features. Among them, based on the initial loss value between the initial prediction result of the initial music model and the training label of the audio data, and the feature loss value between the first music features of the initial music model and the third music features of the auxiliary training model, the parameters of the initial music model are trained, so that the initial music model can learn the music feature encoding ability of the auxiliary training model, streamline the expression of music features, not only realizes the structural simplification of the music model, but also improves the feature encoding ability of the music model, and also makes the music model more suitable for application in specific category audio tasks, such as audio tasks like song quality evaluation and song classification, enhancing the generalization ability of the model.

[0084] In a possible implementation manner, a converged compression module and an auxiliary classifier can be trained first, and then the initial music model can be trained using the converged compression module and the converged auxiliary classifier. In one implementation manner, the initial prediction result is obtained by classifying the first music features using the initial classifier; the auxiliary prediction result is obtained by classifying the third music features using the auxiliary classifier; based on the auxiliary loss value between the auxiliary prediction result and the training label of the audio data, the parameters of the compression module and the auxiliary classifier are adjusted until both the compression module and the auxiliary classifier converge; after confirming that both the compression module and the auxiliary classifier converge, based on the initial loss value between the initial prediction result and the training label of the audio data, and the feature loss value between the first music features and the third music features, the parameters of the initial music model are adjusted until the initial music model converges; the finally converged music model is used as the music model for processing audio tasks. In this method, based on the initial loss value and the feature loss value, the parameters of the initial music model are adjusted by these two loss values, and the calculation methods of these two loss values can adopt the cross-entropy loss function, etc. Exemplarily, the loss comprehensive value S1 of the initial loss value and the feature loss value can be calculated according to the formula S1 = A×a + B×b, and the parameters of the initial music model are adjusted using the loss comprehensive value S1; A represents the initial loss value, a represents the weight value corresponding to the initial loss value, B represents the feature loss value, b represents the weight value corresponding to the feature loss value, and a + b = 1.

[0085] In a possible implementation, the compression module, the auxiliary classifier, and the initial music model are synchronously trained. The method provided in this embodiment further includes: if it is confirmed that the compression module has not converged, using the auxiliary classifier to classify the third music feature to obtain an auxiliary prediction result; calculating an auxiliary loss value between the auxiliary prediction result and the training label of the audio data; using the auxiliary loss value to adjust the parameters of the compression module and the auxiliary classifier, and at the same time adjusting the parameters of the initial music model based on the initial loss value and the feature loss value, and repeating this process until the compression module, the auxiliary classifier, and the initial music model all converge; using the finally converged music model as the target music model. In another implementation, the parameters of the compression module, the auxiliary classifier, and the initial music model can also be adjusted simultaneously based on the comprehensive value of the three loss values, namely the initial loss value, the feature loss value, and the auxiliary loss value. The calculation methods of these three loss values can use a cross-entropy loss function, etc. Exemplarily, the loss comprehensive value S2 of the initial loss value, the feature loss value, and the auxiliary loss value can be calculated according to the formula S2 = A×a + B×b + C×c, and the loss comprehensive value S2 is used to adjust the parameters of the compression module, the auxiliary classifier, and the initial music model; A represents the initial loss value, a represents the weight value corresponding to the initial loss value, B represents the feature loss value, b represents the weight value corresponding to the feature loss value, C represents the auxiliary loss value, c represents the weight value corresponding to the auxiliary loss value, and a + b + c = 1.

[0086] Please refer to Figure 3, in a model training scheme, the teacher model (i.e., the auxiliary training model) includes: a pre-trained model, a compression module, and a classification module; the student model (i.e., the initial music model) includes a proprietary model and a classification module; synchronously input a batch of audio data to the teacher model and the student model, and use the teacher model to obtain an auxiliary prediction result; calculate the auxiliary loss value between the auxiliary prediction result and the training label of the audio data; use the auxiliary loss value to adjust the parameters of the compression module and the classifier in the teacher model, and at the same time, based on the loss of the classification result output by the student model and the knowledge distillation loss between the compression module and the proprietary model, perform a synchronous update on the student model. Repeat this process until the maximum number of iterations is reached or the changes in each loss value are not significant. It is considered that the compression module, the auxiliary classifier, and the student model are converged. Then, the current music model is used as the target music model for processing audio tasks. Among them, use a pre-trained model to encode the input audio, use the compression module to compress the encoding result, use the auxiliary classifier to classify the quality of the compressed result, and calculate the auxiliary loss value. In this process, the auxiliary loss value is used to update the parameters of the auxiliary classifier and the compression module; at the same time, use the proprietary model (i.e., the initial feature extraction module) to encode the input audio, so that the initial feature extraction module outputs intermediate features (i.e., the first music features), and then calculate the feature loss value between the intermediate features and the third music features output by the aforementioned compression module. After the initial classifier classifies the intermediate features, calculate the initial classification loss; the initial classification loss and the feature loss value are added up to update the parameters of the initial classifier and the proprietary model.

[0087] It should be noted that the compression module can also be called the feature alignment module, which is used in the training process of the teacher model to make the pre-trained model more suitable for the current task, so that the student model can better learn the feature information output by the compression module. The pre-trained model generally outputs general audio representations, and with the help of the compression module, the general audio representations can be made more suitable for the current task. Moreover, since the student model usually has a relatively simple structure and its output vector dimension is also small, the student model does not need an additional compression module here. If a compression module is also set for the student model, the simultaneous training of the two compression modules may cause gradient collapse and affect the training performance.

[0088] Pre-trained models are often pre-trained using unsupervised methods, which can be: contrastive learning models, language models based on Masked-AutoEncoder or BERT (Bidirectional Encoder Representations from Transformer). The training data can be: music data, speech data of sound scenes, etc. The pre-trained model can also be tested in various dimensions. For example, using open-source datasets such as GTZAN or VocalSet, evaluations are conducted on tasks such as singer timbre classification, technique classification, instrument classification, genre classification, etc., so that the pre-trained model can achieve good results in each task.

[0089] In this embodiment, the data dimensions input to the pre-trained model and the proprietary model are the same. If the sampling rate of the data to be input is different from the input dimension, the training data is resampled to adapt to the sampling rate during the pre-training of the pre-trained model. Generally, the input of the pre-trained model can be a one-dimensional vector, that is, the sampling points of the audio, and the output can be a feature matrix such as [T, 24, 1024]. The compression module can extract the Embedding representation, that is: transform [T, 24, 1024] into

[256] . The compression module can be a trainable linear transformation layer, such as a fully connected or CRNN structure, which can convert the output vector of the pre-trained model into a vector with a lower dimension. In the compression module, 1-2 layers of fully connected layers can be used to ensure that the dimension of the compressed vector is the same as the dimension of the output of the proprietary model in the student model, such as

[256] ->

[40] , thus avoiding an additional linear projection layer.

[0090] For the proprietary model, a specific network model suitable for the current task can be selected. For example, for the timbre classification task, Short-chunk CNN with residual connection or Harmonic CNN can be used as the proprietary model. The proprietary model can include: a feature extraction layer, and the feature extraction layer of the proprietary model includes a convolutional layer and a pooling layer for extracting audio features.

[0091] In another example, the compression module, the proprietary model, and the two classification modules can be trained simultaneously. The losses to be calculated for each training include: the initial loss value, the feature loss value, and the auxiliary loss value.

[0092] Among them, the calculation formula for the auxiliary loss value is: ; the calculation formula for the feature loss value (i.e., the knowledge distillation loss) is: ; the calculation formula for the initial loss value is: ; represents the auxiliary loss value, represents the auxiliary prediction result, Represents the true classification label of the input audio, Represents the feature loss value, Represents the output of the compression module, Represents the output of the proprietary model, Represents the initial loss value, Represents the initial prediction result.

[0093] After training is completed, the student model can learn the encoding ability of the teacher model. The encoding ability of the student model not only removes irrelevant knowledge but also expands on proprietary tasks, enabling the student model to achieve knowledge balance and improve its generalization ability.

[0094] Please refer to Figure 4 , when new audio data is input into the trained proprietary model and the corresponding classification module, intermediate representations can be obtained through the proprietary model, and classification results can be output through the classification module. The achievable audio tasks include: song preference recommendation tasks, such as recommending songs of a specific genre, specific singer, or specific timbre that the user likes; song search tasks, such as searching for different versions of a song, the original version by the record company, and various cover versions under the same search term; song sound quality evaluation tasks, such as determining whether a song belongs to the good, medium, or bad level; or a comprehensive task of song search and sound quality evaluation can also be achieved, such as after judging the sound quality of 10 songs retrieved under the same search term, they can then be sorted according to the sound quality.

[0095] It can be seen that through the joint training architecture in this embodiment, not only can the output representation of the teacher model better adapt to various tasks, reducing the need for model fine-tuning and linear probing and improving the efficiency of knowledge transfer. Moreover, the compression module is used to effectively remove irrelevant information in the output features of the teacher model, enabling the student model to focus on task-related features, thereby improving its performance on tasks, enhancing the generalization ability of the model, and also performing well on unseen data.

[0096] Next, an audio processing method provided by an embodiment of the present application will be introduced. The relevant implementation steps of the audio processing method described below can be referred to each other with the above embodiments.

[0097] Please refer to Figure 5 , this embodiment provides an audio processing method, including:

[0098] S501. Obtain audio data.

[0099] In this embodiment, the audio data can be: music audio data such as songs, pure audio, etc.

[0100] S502. Input the audio data into the target music model to use the target music model to output the classification result of the audio data.

[0101] It should be noted that the music model in this embodiment is applicable to processing song preference recommendation tasks, such as recommending songs of a specific genre, a specific singer, or a specific timbre that the user likes; song search tasks, such as searching for different versions of a song, the original version of the record company, and various cover versions under the same search term; song sound quality evaluation tasks, such as determining which level of good, medium, or bad a song belongs to; or a comprehensive task of song search and sound quality evaluation can also be achieved. For example, after judging the sound quality of 10 songs queried under the same search term, they can then be sorted according to the sound quality. Therefore, in one implementation manner, this embodiment can also determine the category to which the audio data belongs according to the classification result; search for songs belonging to the category, and / or recommend songs belonging to the category.

[0102] The music model in this embodiment is obtained based on the model training method provided in any of the foregoing embodiments. For the specific model training method, reference can be made to the relevant introduction in other embodiments, and this embodiment will not elaborate here.

[0103] It can be seen that this embodiment realizes the structural simplification of the music model, improves the feature encoding ability of the music model, makes the music model more suitable for application in audio tasks such as song quality evaluation and song classification, and enhances the generalization ability of the model.

[0104] Next, an electronic device provided in an embodiment of the present application will be introduced. The relevant implementation steps of the electronic device described below can be referred to each other with those in the above embodiments.

[0105] An embodiment of the present application discloses an electronic device, including:

[0106] A memory for storing a computer program;

[0107] A processor for executing the computer program to implement the method disclosed in any of the foregoing embodiments.

[0108] In this embodiment, when the processor executes the computer program stored in the memory, the following steps can be specifically implemented: extracting features of audio data corresponding to an audio task through an initial music model to obtain first music features, and generating an initial prediction result based on the first music features; extracting features of the audio data through an auxiliary training model to obtain second music features, and performing compression processing on the second music features to obtain third music features; training the initial music model based on the initial loss value between the initial prediction result and the training label of the audio data, and the feature loss value between the first music features and the third music features until a preset training stop condition is met to obtain a trained target music model; the target music model is used to process audio tasks.

[0109] In this embodiment, when the processor executes the computer program stored in the memory, the following steps can be specifically implemented: extracting features from the audio data through the initial feature extraction module to obtain the first music feature; classifying the first music feature through the initial classifier to obtain the initial prediction result.

[0110] In this embodiment, when the processor executes the computer program stored in the memory, the following steps can be specifically implemented: calculating a comprehensive loss value based on the initial loss value and the feature loss value; updating the parameters of the initial feature extraction module and the parameters of the initial classifier respectively based on the comprehensive loss value.

[0111] In this embodiment, when the processor executes the computer program stored in the memory, the following steps can be specifically implemented: extracting features from the audio data through the feature assistance module to obtain the second music feature; compressing the second music feature through the compression module to obtain the third music feature.

[0112] In this embodiment, when the processor executes the computer program stored in the memory, the following steps can be specifically implemented: classifying the third music feature through the auxiliary classifier to obtain the auxiliary prediction result; training the auxiliary classifier and the compression module based on the auxiliary loss value between the auxiliary prediction result and the training label.

[0113] In this embodiment, when the processor executes the computer program stored in the memory, the following steps can be specifically implemented: obtaining an audio set; adjusting the model parameters of the pre-trained model by using the audio set until the pre-trained model converges; using the converged pre-trained model as the feature assistance module.

[0114] In this embodiment, when the processor executes the computer program stored in the memory, the following steps can be specifically implemented: evaluating the music feature encoding ability of the converged pre-trained model in multiple audio tasks; if the evaluation results of the converged pre-trained model in multiple audio tasks meet the expected requirements, then execute the step of using the converged pre-trained model as the feature assistance module; if the evaluation results of the converged pre-trained model in multiple audio tasks do not meet the expected requirements, then enhance the sample quality of the audio set; after replacing the audio set with the audio set with enhanced sample quality and replacing the pre-trained model with the converged pre-trained model, execute the step of adjusting the model parameters of the pre-trained model by using the audio set until the pre-trained model converges and subsequent steps.

[0115] In this embodiment, when the processor executes the computer program stored in the memory, the following steps can be specifically implemented: determining the key audio features corresponding to the audio task; extracting the key audio features from the second music feature by using the compression module as the third music feature.

[0116] In this embodiment, when the processor executes the computer program stored in the memory, the following steps may be specifically implemented: obtaining audio data; inputting the audio data into a target music model to output a classification result of the audio data by using the target music model.

[0117] In this embodiment, when the processor executes the computer program stored in the memory, the following steps may be specifically implemented: determining the category to which the audio data belongs according to the classification result; searching for songs belonging to the category and / or recommending songs belonging to the category.

[0118] Furthermore, an embodiment of the present application also provides an electronic device. Among them, the above-mentioned electronic device may be either a server 50 as shown in Figure 6 or a terminal 60 as shown in Figure 7 . Figure 6 And Figure 7 are both structural diagrams of electronic devices shown according to an exemplary embodiment, and the content in the figure cannot be considered as any limitation on the scope of use of the present application.

[0119] Figure 6 This is a schematic structural diagram of a server provided by an embodiment of the present application. The server 50 may specifically include: at least one processor 51, at least one memory 52, a power supply 53, a communication interface 54, an input / output interface 55, and a communication bus 56. Among them, the memory 52 is used to store a computer program, and the computer program is loaded and executed by the processor 51 to implement the relevant steps in the model training disclosed in any of the foregoing embodiments.

[0120] In this embodiment, the power supply 53 is used to provide operating voltage for each hardware device on the server 50; the communication interface 54 can create a data transmission channel between the server 50 and external devices, and the communication protocol it follows is any communication protocol applicable to the technical solution of the present application, and no specific limitation is imposed on it here; the input / output interface 55 is used to obtain external input data or output data to the outside, and its specific interface type can be selected according to specific application needs, and no specific limitation is made here.

[0121] In addition, as a carrier for resource storage, the memory 52 may be a read-only memory, a random access memory, a magnetic disk, or an optical disc, etc. The resources stored thereon include an operating system 521, a computer program 522, and data 523, etc., and the storage method may be temporary storage or permanent storage.

[0122] Among them, the operating system 521 is used to manage and control each hardware device and computer program 522 on the server 50, so as to implement the operation and processing of the data 523 in the memory 52 by the processor 51. It can be Windows Server, Netware, Unix, Linux, etc. In addition to the computer program that can be used to complete the model training method disclosed in any of the foregoing embodiments, the computer program 522 may further include a computer program that can be used to complete other specific tasks. In addition to data such as update information of the application program, the data 523 may further include data such as developer information of the application program.

[0123] Figure 7 FIG. is a schematic structural diagram of a terminal provided by an embodiment of the present application. The terminal 60 may specifically include, but is not limited to, a smart phone, a tablet computer, a notebook computer, or a desktop computer, etc.

[0124] Generally, the terminal 60 in this embodiment includes: a processor 61 and a memory 62.

[0125] Among them, the processor 61 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 61 may be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), or PLA (Programmable Logic Array). The processor 61 may also include a main processor and a coprocessor. The main processor is a processor used to process data in the wake state, also known as the CPU (Central Processing Unit); the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, the processor 61 may be integrated with a GPU (Graphics Processing Unit), and the GPU is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 61 may further include an AI (Artificial Intelligence) processor, and the AI processor is used to process computational operations related to machine learning.

[0126] The memory 62 may include one or more computer-readable storage media, which may be non-transitory. The memory 62 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices and flash storage devices. In this embodiment, the memory 62 is at least used to store the following computer program 621. After the computer program is loaded and executed by the processor 61, it can implement the relevant steps in the model training method executed by the terminal side disclosed in any of the foregoing embodiments. In addition, the resources stored in the memory 62 may also include an operating system 622, data 623, etc., and the storage method may be transient storage or permanent storage. Among them, the operating system 622 may include Windows, Unix, Linux, etc. The data 623 may include, but is not limited to, update information of the application program.

[0127] In some embodiments, the terminal 60 may further include a display screen 63, an input / output interface 64, a communication interface 65, sensors 66, a power supply 67, and a communication bus 68.

[0128] Those skilled in the art can understand that Figure 7 the structure shown in does not constitute a limitation on the terminal 60, and it may include more or fewer components than shown in the figure.

[0129] Next, a storage medium provided by an embodiment of the present application will be introduced. The relevant implementation steps of the storage medium described below can be referred to each other with the above embodiments.

[0130] Furthermore, an embodiment of the present application also discloses a storage medium, in which computer-executable instructions are stored. When the computer-executable instructions are loaded and executed by a processor, the model training method disclosed in any of the foregoing embodiments is implemented. For the specific steps of this method, reference may be made to the corresponding content disclosed in the foregoing embodiments, and details will not be repeated here.

[0131] Furthermore, an embodiment of the present application also discloses a computer program product, including a computer program / instructions, characterized in that when the computer program / instructions are executed by a processor, the method described in any of the foregoing embodiments is implemented. For the specific steps of this method, reference may be made to the corresponding content disclosed in the foregoing embodiments, and details will not be repeated here.

[0132] It should be noted that the above are only the preferred embodiments of the present application, and are not intended to limit the present application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

[0133] In the present specification, the various embodiments are described in a progressive manner. Each embodiment focuses on the differences from other embodiments. For the same or similar parts among the various embodiments, reference may be made to each other.

[0134] In this article, specific examples are used to illustrate the principles and implementation manners of the present application. The descriptions of the above embodiments are only used to help understand the method and its core idea of the present application. At the same time, for those of ordinary skill in the art, according to the idea of the present application, there will be changes in the specific implementation manners and application scopes. In summary, the content of this specification should not be construed as a limitation to the present application.

Claims

1. A model training method, characterized in that: include: Extracting features from audio data corresponding to the audio task using an initial music model to obtain a first music feature, and generating an initial prediction result based on the first music feature; Extracting features from the audio data using an auxiliary training model to obtain a second music feature, and compressing the second music feature to obtain a third music feature; Based on the initial loss value between the initial prediction result and the training label of the audio data, and the feature loss value between the first music feature and the third music feature, the initial music model is trained until a preset training stop condition is met to obtain a trained target music model; the target music model is used to process the audio task.

2. The method according to claim 1, characterized in that The initial music model includes an initial feature extraction module and an initial classifier; Accordingly, feature extraction is performed on the audio data corresponding to the audio task through the initial music model to obtain a first music feature, and an initial prediction result is generated based on the first music feature, including: Extracting features from the audio data using the initial feature extraction module to obtain the first music feature; Classify the first music feature by the initial classifier to obtain the initial prediction result; Accordingly, based on the initial loss value between the initial prediction result and the training label of the audio data, and the feature loss value between the first music feature and the third music feature, the initial music model is trained, including: Based on the initial loss value and the characteristic loss value, a comprehensive loss value is calculated; Based on the comprehensive loss value, the parameters of the initial feature extraction module and the parameters of the initial classifier are updated respectively.

3. The method according to claim 1, characterized in that The auxiliary training model includes: a feature auxiliary module and a compression module; Accordingly, extracting features from the audio data through the auxiliary training model to obtain a second music feature, and compressing the second music feature to obtain a third music feature, including: Extracting features from the audio data using a feature auxiliary module to obtain the second music feature; The second music feature is compressed by the compression module to obtain the third music feature.

4. The method according to claim 3, characterized in that The auxiliary training model also includes: an auxiliary classifier; Accordingly, the method further comprises: Classify the third music feature by the auxiliary classifier to obtain an auxiliary prediction result; The auxiliary classifier and the compression module are trained based on the auxiliary loss value between the auxiliary prediction result and the training label.

5. The method according to claim 3, characterized in that: The generation process of the feature auxiliary module includes: Get audio collection; Adjusting model parameters of a pre-trained model using the audio set until the pre-trained model converges; The converged pre-trained model is used as the feature auxiliary module.

6. The method according to claim 5, characterized in that Before using the converged pre-trained model as the feature auxiliary module, the method further includes: Evaluate the music feature encoding capability of the converged pre-trained model in a variety of audio tasks; If the evaluation results of the converged pre-trained model in the multiple audio tasks meet the expected requirements, the step of using the converged pre-trained model as the feature auxiliary module is executed; If the evaluation results of the converged pre-trained model in multiple audio tasks do not meet the expected requirements, the sample quality of the audio set is enhanced; the audio set is replaced with the audio set with enhanced sample quality, and after replacing the pre-trained model with the converged pre-trained model, the model parameters of the pre-trained model are adjusted using the audio set until the pre-trained model converges and subsequent steps.

7. The method according to claim 3, characterized in that The second music feature is compressed by the compression module to obtain the third music feature, including: Determining key audio features corresponding to the audio task; The compression module is used to extract the key audio feature from the second music feature as the third music feature.

8. The method according to any one of claims 1 to 7, characterized in that: The preset training stop condition is met, including: The preset maximum number of iterations is reached; or the changes in the initial loss value and the feature loss value are in line with expectations.

9. An audio processing method, characterized in that: include: Get audio data; Inputting the audio data into a target music model to output a classification result of the audio data using the target music model; The target music model is obtained based on the method described in any one of claims 1 to 8.

10. The method according to claim 9, characterized in that Also includes: Determine the category to which the audio data belongs according to the classification result; Search for songs belonging to the category, and / or recommend songs belonging to the category.

11. An electronic device, characterized in that: The electronic device comprises a processor and a memory; wherein the memory is used to store a computer program, and the computer program is loaded and executed by the processor to implement the method according to any one of claims 1 to 10.

12. A storage medium, characterized in that: The storage medium stores computer executable instructions, and when the computer executable instructions are loaded and executed by the processor, the method according to any one of claims 1 to 10 is implemented.

13. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the method according to any one of claims 1 to 10 is implemented.

Images