Network training, song recommendation and fusion representation generation method, equipment, medium and product

Abstract

The invention discloses a network training, song recommendation and fusion representation generation method and device, a medium and a product, and relates to the technical field of audio processing, and the method comprises the steps: obtaining a plurality of first audio representations obtained after processing a song sample through at least two large models; performing preset processing on each first audio representation to obtain a second audio representation; the preset processing comprises a mask operation and a splicing operation on the audio representation; and inputting the second audio representation into the to-be-trained self-encoding network, training the to-be-trained self-encoding network by using the target loss function, and obtaining a trained self-encoding network used for generating song fusion representation corresponding to the to-be-processed song. In this way, the dimension of the audio representation is reduced through the mask operation and the splicing operation, and the training efficiency can be improved; and training is carried out by using a loss function constructed based on the second audio representation and the decoded audio representation, so that the self-encoding model can be ensured to have the capability of generating the representation, and the generalization of the model is improved.

Description

Technical Field

[0001] The present invention relates to the field of audio processing technology, and in particular to a network training, song recommendation and fusion representation generation method, device, medium and product. Background Art

[0002] At present, there are some music understanding base models active in the industry, which can bring better results for music downstream tasks, but these base models have their own advantages. Some perform better in genre classification, and some perform better in timbre tasks. No music understanding model can dominate. For example, MERT (Music Understanding Model with Large-Scale Self-supervised Training), LAION-CLAP (Contrastive Language-Audio Pretraining, LAION-open-source contrastive language-audio pretraining model), MULE (Musicset Unsupervised Large Embedding, music representation model), etc. These models have their own strengths in different aspects. The traditional feature fusion method is to splice different features. Direct splicing will bring dimensionality problems. Many large model representation vectors are thousands of dimensions. If they are directly spliced, it may lead to problems such as too large dimensions and too slow training of downstream tasks.

[0003] It can be seen that how to combine the advantages of different music models to fuse the audio representations output by each model is a problem to be solved in this field. Summary of the invention

[0004] In view of this, the purpose of the present invention is to provide a network training, song recommendation and fusion representation generation method, device, medium and product, which can improve the training efficiency of the autoencoder network by reducing the dimension of the audio representation through masking and splicing operations; and, the target loss function constructed based on the second audio representation and the decoded audio representation is used to train the autoencoder network to ensure that the model has the ability to generate representations and improve the generalization of the model. The specific scheme is as follows:

[0005] In a first aspect, the present application provides an autoencoder network training method, comprising:

[0006] Obtaining a plurality of first audio representations obtained after at least two large models process the song samples;

[0007] Performing a preset process on each of the first audio representations to obtain a second audio representation; the preset process includes a masking operation and a splicing operation on the audio representation;

[0008] Inputting the second audio representation into the autoencoder network to be trained, so as to train the autoencoder network to be trained using the target loss function, and obtaining a trained autoencoder network for generating a song fusion representation corresponding to the song to be processed;

[0009] The target loss function includes a loss function constructed based on the second audio representation and a decoded audio representation, and the decoded audio representation is an audio representation obtained after the decoder in the autoencoder network to be trained decodes the second audio representation.

[0010] Optionally, the obtaining of a plurality of first audio representations obtained after processing the song sample by at least two large models includes:

[0011] Slicing the song sample based on the preset song segment duration to obtain a number of song segments;

[0012] Determine the size relationship between the input sample duration supported by each of the large models and the preset song segment duration, and process the song segment through each of the large models and based on the corresponding size relationship to obtain corresponding first audio representations.

[0013] Optionally, the song segments are processed by each of the large models based on the corresponding size relationship to obtain corresponding first audio representations, including:

[0014] If the duration of the first input sample supported by the first large model of the at least two large models is consistent with the duration of the preset song segment, the song segment to be processed is directly input into the corresponding first large model for processing to obtain a corresponding first audio representation;

[0015] If the second input sample duration supported by the second largest model of the at least two large models is less than the preset song segment duration, the song segment to be processed is cropped at least twice to obtain at least two cropped segments, and the at least two cropped segments are input into the second largest model, so that the second largest model extracts and averages the audio representations of each of the cropped segments to obtain the corresponding first audio representation;

[0016] The duration of each of the cropped segments is consistent with the duration of the second input sample, and the contents of the at least two cropped segments include all the contents of the corresponding song segments.

[0017] Optionally, performing preset processing on each of the first audio representations to obtain a second audio representation includes:

[0018] Performing random masking operations on the first audio representations according to a preset masking ratio to obtain corresponding masked audio representations;

[0019] A splicing operation is performed on the plurality of masked audio representations to obtain a second audio representation corresponding to the song sample.

[0020] Optionally, performing preset processing on each of the first audio representations to obtain a second audio representation includes:

[0021] Performing a splicing operation on the plurality of the first audio representations to obtain a spliced ​​audio representation;

[0022] A random masking operation is performed on the concatenated audio representation to obtain a second audio representation corresponding to the song sample.

[0023] Optionally, the target loss function is a loss function constructed based on a first loss between the second audio representation and the decoded audio representation;

[0024] Accordingly, the training of the autoencoder network to be trained by using the target loss function includes:

[0025] Calculating the loss between the second audio representation and the decoded audio representation using the target loss function to obtain a corresponding loss calculation result;

[0026] The autoencoder network to be trained is adjusted based on the loss calculation result.

[0027] Optionally, the target loss function is a loss function constructed based on a first loss between the second audio representation and the decoded audio representation and a second loss between the third audio representation and the decoded audio representation; wherein the third audio representation is an audio representation obtained by directly concatenating the plurality of first audio representations;

[0028] Accordingly, the training of the autoencoder network to be trained by using the target loss function includes:

[0029] Calculating a first loss between the second audio representation and the decoded audio representation and a second loss between the third audio representation and the decoded audio representation using the target loss function to obtain corresponding loss calculation results;

[0030] The autoencoder network to be trained is adjusted based on the loss calculation result.

[0031] Optionally, the method further includes:

[0032] Based on the requirements of model generalization performance and model convergence speed, the weight values ​​corresponding to the first loss and the second loss in the objective loss function are adjusted.

[0033] In a second aspect, the present application provides a method for generating song fusion representation, comprising:

[0034] Acquire several initial audio representations obtained after processing the song to be processed by at least two large models, and perform preset processing on each of the initial audio representations to obtain a processed audio representation; the preset processing includes a masking operation and a splicing operation on the audio representation;

[0035] The processed audio representation is input into the trained autoencoder network to obtain a song fusion representation generated by the trained autoencoder network and corresponding to the song to be processed; the trained autoencoder network is a trained autoencoder network obtained using the autoencoder network training method as described above.

[0036] Optionally, the process of generating a song fusion representation corresponding to the song to be processed by the trained autoencoder network includes:

[0037] The encoder in the trained autoencoder network processes the received processed audio representation to obtain compressed initial fusion representations corresponding to the song to be processed output by the encoder; wherein different initial fusion representations correspond to different processed audio representations;

[0038] The average of each of the initial fusion representations is calculated to obtain a song fusion representation corresponding to the song to be processed.

[0039] In a third aspect, the present application provides a song recommendation method, comprising:

[0040] Obtaining a song fusion representation corresponding to the song to be processed; the song fusion representation is a song fusion representation obtained by using the above-mentioned song fusion representation generation method;

[0041] Performing spatial distance calculation on the song fusion representation and a plurality of preset reference fusion representations to obtain corresponding spatial distances; the plurality of reference fusion representations are fusion representations obtained by processing historical songs set as liked and disliked by the target user using a trained autoencoder network;

[0042] Based on the size relationship between a preset spatial distance threshold and the spatial distance, it is determined whether to recommend the song to be processed corresponding to the song fusion representation to the target user.

[0043] In a fourth aspect, the present application provides an electronic device, including:

[0044] Memory, used to store computer programs;

[0045] A processor is used to execute the computer program to implement the above method.

[0046] In a fifth aspect, the present application provides a computer-readable storage medium for storing a computer program, wherein the computer program implements the above-mentioned method when executed by a processor.

[0047] In a sixth aspect, the present application provides a computer program product, comprising a computer program / instruction, which implements the above method when executed by a processor.

[0048] It can be seen that the present application first obtains several first audio representations obtained after processing the song samples by at least two large models; then performs preset processing on each of the first audio representations to obtain a second audio representation; the preset processing includes masking and splicing operations on the audio representation; then the second audio representation is input into the autoencoder network to be trained, so as to train the autoencoder network to be trained using the target loss function, and obtain a trained autoencoder network for generating a song fusion representation corresponding to the song to be processed; wherein the target loss function includes a loss function constructed based on the second audio representation and the decoded audio representation, and the decoded audio representation is an audio representation obtained after the decoder in the autoencoder network to be trained decodes the second audio representation. In this way, the present application reduces the dimension of the audio representation through masking and splicing operations, which can improve the representation learning ability of the autoencoder network and improve the training efficiency; and, using the target loss function constructed based on the second audio representation and the decoded audio representation to train the autoencoder network to be trained can ensure that the model has the ability to generate representations and improve the generalization of the model; the autoencoder network trained in this way can output a higher quality and more comprehensive fusion representation, which can better adapt to downstream music tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0050] Figure 1 This is an architecture diagram of an autoencoder network training method disclosed in this application;

[0051] Figure 2 A flowchart of a specific autoencoder network training method disclosed in this application;

[0052] Figure 3 A flowchart of a specific song fusion representation generation method disclosed in this application;

[0053] Figure 4 A specific song recommendation method flow chart disclosed in this application;

[0054] Figure 5 Another specific flow chart of the autoencoder network training method disclosed in this application;

[0055] Figure 6 This is a structural diagram of an electronic device disclosed in this application. DETAILED DESCRIPTION

[0056] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0057] The existing large music understanding models have their own strengths in different aspects. The traditional feature fusion method is to splice different features. Direct splicing will bring dimensionality problems. Many large model representation vectors are thousands of dimensions. If they are directly spliced, it may lead to problems such as too large dimensions and slow training of downstream tasks. To this end, the present application provides an autoencoder network training method, device, storage medium and program product. The dimension of the audio representation is reduced by masking and splicing operations, which can improve the training efficiency of the autoencoder network; and, the target loss function constructed based on the second audio representation and the decoded audio representation is used to train the autoencoder network to ensure that the model has the ability to generate representations and improve the generalization of the model; the autoencoder model trained in this way can output higher-quality and comprehensive fusion representations, which can better adapt to downstream music tasks.

[0058] In the autoencoder network training method of this application, the system framework used can be specifically referred to Figure 1 As shown, it may specifically include: a backend server 01 and a user terminal 02 that establishes a connection with the backend server 01 .

[0059] In this application, the user terminal 02 is used to obtain song samples for training the autoencoder network and songs to be processed for audio representation fusion. The user terminal 02 can be a client, such as a mobile phone, tablet, computer, etc., or a web page, which is not limited here.

[0060] Furthermore, the backend server 01 is used to perform the steps of the autoencoder network training method and the song fusion representation generation method, including obtaining the song sample and the song to be processed sent by the user end 02; wherein, for the song sample, the song sample is processed by at least two large models to obtain the corresponding first audio representation; then the first audio representation is processed by masking operation and splicing operation to obtain the second audio representation; then the second audio representation is input into the autoencoder network to be trained, so as to train the autoencoder network to be trained by using the target loss function, and obtain the trained autoencoder network for generating the song fusion representation corresponding to the song to be processed. Correspondingly, for the music to be processed, several initial audio representations are obtained after processing the song to be processed by at least two large models, and each initial audio representation is masked and spliced ​​to obtain a processed audio representation; then the processed audio representation is input into the trained autoencoder network to obtain the song fusion representation generated by the trained autoencoder network corresponding to the song to be processed.

[0061] See also Figure 2 As shown, an embodiment of the present invention discloses a method for training an autoencoder network, comprising:

[0062] Step S11, obtaining a plurality of first audio representations obtained after at least two large models process the song samples.

[0063] In the embodiment of the present application, first, it is necessary to perform audio representation extraction and processing on the song sample through at least two existing large models; for a single song sample, a corresponding number of audio representations can be extracted through at least two large models. For example, three audio representations can be obtained by processing a single song sample through three large models.

[0064] In a specific embodiment, the obtaining of several first audio representations obtained after processing the song sample by at least two large models may include: slicing the song sample based on a preset song segment duration to obtain several song segments; determining the size relationship between the input sample duration supported by each of the large models and the preset song segment duration, and processing the song segment through each of the large models based on the corresponding size relationship to obtain the corresponding several first audio representations. Specifically, since the input sample durations corresponding to each of the large models are different, a song segment duration applicable to most large models can be set in advance so that the relevant song samples can be sliced ​​based on the song segment duration, and a single song sample can be sliced ​​to obtain several song segments. Furthermore, in the process of extracting the audio representation of the song segment corresponding to the song sample through the large model, it is necessary to determine the size relationship between the input sample duration supported by the currently used large model and the preset song segment duration, and then extract the audio representation of the song segment corresponding to the song sample according to the size relationship.

[0065] In another specific embodiment, the processing of the song clip through each of the large models and based on the corresponding size relationship to obtain several corresponding first audio representations may include: if the first input sample duration supported by the first large model of the at least two large models is consistent with the preset song clip duration, the song clip to be processed is directly input into the corresponding first large model for processing to obtain the corresponding first audio representation; if the second input sample duration supported by the second large model of the at least two large models is less than the preset song clip duration, the song clip to be processed is cropped at least twice to obtain at least two cropped clips, and the at least two cropped clips are input into the second large model, so that the second large model extracts and averages the audio representations of each of the cropped clips to obtain the corresponding first audio representation; wherein the clip duration of each of the cropped clips is consistent with the second input sample duration, and the content of the at least two cropped clips contains all the content of the corresponding song clip. Specifically, if the duration of the first input sample supported by the current large model is consistent with the preset song segment duration, it means that the song segment obtained by slicing the song sample can be directly input into the current large model, and the corresponding first audio representation is obtained after being processed by the large model. Correspondingly, if the duration of the second input sample supported by the current large model is less than the preset song segment duration, it means that the song segment obtained by slicing the song sample cannot be directly input into the current large model, and further cropping is required. It can be understood that in the process of cropping the song segment, it is necessary to ensure that the content of all the cropped segments finally obtained by cropping contains all the content of the corresponding song segment, and the segment duration of each cropped segment needs to be consistent with the duration of the second input sample, so that the cropped segment can be input into the large model corresponding to the duration of the second input sample, and the corresponding first audio representation is obtained after being processed by the large model. It can be understood that the first audio representation obtained by the above steps is an audio representation corresponding to the song sample, which includes audio representations of several song segments corresponding to a single song sample; for a single song sample, at least two first audio representations can be obtained by processing at least two large models.

[0066] Step S12: performing a preset process on each of the first audio representations to obtain a second audio representation; the preset process includes a masking operation and a splicing operation on the audio representation.

[0067] In the embodiment of the present application, the above steps can be processed by several large models to obtain several first audio representations corresponding to the song sample; then each first audio representation can be processed by masking and splicing operations, the masking process can reduce the representation dimension, and the splicing can merge the audio representations processed by each large model together; by performing masking and splicing operations on each audio representation, the second audio representation corresponding to the song sample can be finally obtained. It should be pointed out that the second audio representation obtained here is an audio representation obtained after masking and splicing of several first audio representations corresponding to a single song sample; it includes audio representations of several song clips corresponding to the song sample.

[0068] In a specific embodiment, the preset processing of each of the first audio representations to obtain the second audio representation may include: performing random masking operations on several of the first audio representations according to a preset mask ratio to obtain several corresponding masked audio representations; and performing a splicing operation on the several masked audio representations to obtain the second audio representation corresponding to the song sample. Specifically, in the process of processing each first audio representation to obtain the second audio representation, several first audio representations may first be randomly masked according to a preset mask ratio, that is, the first audio representations output by each major model may be randomly masked to obtain several corresponding masked audio representations; and then the masked audio representations may be spliced; in this way, by masking and splicing each first audio representation corresponding to the song sample, the second audio representation corresponding to the song sample may be obtained.

[0069] In another specific embodiment, the preset processing of each of the first audio representations to obtain the second audio representation may include: performing a splicing operation on a number of the first audio representations to obtain a spliced ​​audio representation; performing a random masking operation on the spliced ​​audio representation to obtain the second audio representation corresponding to the song sample. Specifically, in the process of processing each of the first audio representations to obtain the second audio representation, each of the first audio representations may be firstly spliced ​​to obtain a corresponding spliced ​​audio representation, and then the spliced ​​audio representations may be uniformly randomly masked to finally obtain the second audio representation corresponding to the song sample.

[0070] Step S13, inputting the second audio representation into the autoencoder network to be trained, so as to train the autoencoder network to be trained using the target loss function, and obtaining a trained autoencoder network for generating a song fusion representation corresponding to the song to be processed; wherein the target loss function includes a loss function constructed based on the second audio representation and the decoded audio representation, and the decoded audio representation is an audio representation obtained after the decoder in the autoencoder network to be trained decodes the second audio representation.

[0071] In the embodiment of the present application, during the process of training the autoencoder network, the second audio representation is input into the autoencoder network to be trained so as to train the autoencoder network to be trained using the target loss function; it can be understood that the autoencoder network to be trained includes an encoder and a decoder, and the second audio representation obtained in the above step is compressed by the encoder, and then the compressed audio representation is decoded by the decoder to obtain a decoded audio representation corresponding to the second audio representation. It should be pointed out that during the process of training the autoencoder network, the target loss function used can be a loss function constructed based on the second audio representation and the decoded audio representation; the encoder in the autoencoder network is specifically trained and learned through the loss function, and finally a trained autoencoder network can be obtained for processing the song to be processed to generate a corresponding song fusion representation.

[0072] In a specific embodiment, the target loss function is a loss function constructed based on the first loss between the second audio representation and the decoded audio representation; accordingly, the use of the target loss function to train the autoencoder network to be trained may include: using the target loss function to calculate the loss between the second audio representation and the decoded audio representation to obtain a corresponding loss calculation result; adjusting the autoencoder network to be trained based on the loss calculation result. Specifically, the target loss function used to train the autoencoder network may be a loss function constructed based on the first loss between the second audio representation and the decoded audio representation, specifically calculating the loss between the second audio representation and the decoded audio representation, and the loss calculation result between the second audio representation and the decoded representation may be obtained through the target loss function; and then the autoencoder network to be trained is trained and learned based on the loss calculation result.

[0073] In another specific embodiment, the target loss function is a loss function constructed based on the first loss between the second audio representation and the decoded audio representation and the second loss between the third audio representation and the decoded audio representation; wherein the third audio representation is an audio representation obtained by directly splicing the several first audio representations; accordingly, the use of the target loss function to train the autoencoder network to be trained may include: using the target loss function to calculate the first loss between the second audio representation and the decoded audio representation and the second loss between the third audio representation and the decoded audio representation to obtain a corresponding loss calculation result; adjusting the autoencoder network to be trained based on the loss calculation result. Specifically, the target loss function used in the process of training the autoencoder network to be trained may also be a loss function constructed based on the first loss between the second audio representation and the decoded audio representation and the second loss between the third audio representation and the decoded audio representation; the third audio representation used here is an audio representation obtained by directly splicing the first audio representations corresponding to the song sample. Furthermore, in the process of training the network, the first loss between the second audio representation and the decoded audio representation, and the second loss between the third audio representation and the decoded audio representation are calculated by the target loss function to obtain corresponding loss calculation results. Then, the autoencoder network to be trained is adjusted based on the loss calculation results, and finally a trained autoencoder network for generating a song fusion representation of the song to be processed can be obtained.

[0074] It should be pointed out that, in a specific embodiment, it may also include: adjusting the weight values ​​corresponding to the first loss and the second loss in the target loss function based on the model generalization performance requirements and the model convergence speed requirements. In other words, in the process of training the autoencoder network to be trained based on the first loss and the second loss, the weight values ​​of the first loss and the second loss in the autoencoder network to be trained can be adjusted considering the generalization of the network and the loss convergence speed. The first loss is to ensure that the model has the ability to generate representations, which can improve the generalization of the model. The second representation is to ensure that the model has the ability to restore the original representation. Finally, a trained autoencoder network can be trained to generate a fusion song representation of the song to be processed.

[0075] It can be seen that the present application reduces the dimensionality of audio representation through masking operations and splicing operations, which can improve the representation learning ability of the autoencoder network and improve the training efficiency; and in the training process, the loss between the audio representation before and after the masking operation and the audio representation after the final decoding can be used to learn the autoencoder network, which can improve the generalization of representation fusion and the ability to restore the original representation. In this way, the trained autoencoder model can output higher-quality and comprehensive fusion representation features, which can better adapt to downstream music tasks.

[0076] The above embodiment introduces the process of training the autoencoder network, and performs masking and splicing operations on the first audio representations of song samples obtained by processing multiple large models; the dimension of the audio representation is reduced by masking and splicing operations, which can improve the training efficiency of the autoencoder network; and the obtained second audio representation is used to train and learn the trained autoencoder network, and finally a trained autoencoder network can be obtained for processing the song to be processed and generating a corresponding song fusion representation; the loss between the audio representation before and after the masking operation and the final decoded audio representation is used to learn the autoencoder network, which can improve the generalization of the representation fusion and the ability to restore the original representation, so that the trained autoencoder model can output a higher quality and comprehensive fusion representation that can better adapt to downstream music tasks. Furthermore, the following embodiment will specifically introduce the process of generating a song fusion representation using a trained autoencoder network, such as Figure 3 As shown, the embodiment of the present application discloses a method for generating song fusion representation, including:

[0077] Step S21, obtaining several initial audio representations obtained after processing the song to be processed by at least two large models, and performing preset processing on each of the initial audio representations to obtain a processed audio representation; the preset processing includes masking operations and splicing operations on the audio representation.

[0078] In the embodiment of the present application, after the trained autoencoder network is obtained by the autoencoder network training method, the audio representation processing can be performed on the song to be processed; specifically, first obtain at least two large models to perform audio representation extraction processing on the song to be processed to obtain a number of initial audio representations; then perform masking and splicing operations on these initial audio representations to obtain processed audio representations. It can be understood that the process of performing masking and splicing operations on the initial audio representations here can be to first mask each initial audio representation, and then splice the corresponding masked audio representations to obtain the processed audio representation; or it can be to first splice each initial audio representation, and then mask the corresponding spliced ​​audio representation to obtain the processed audio representation.

[0079] Step S22, inputting the processed audio representation into the trained autoencoder network to obtain a song fusion representation generated by the trained autoencoder network and corresponding to the song to be processed; the trained autoencoder network is a trained autoencoder network obtained using the autoencoder network training method.

[0080] In the embodiment of the present application, the processed audio representation corresponding to the song to be processed can be obtained through the above steps. The process of obtaining the processed audio representation is similar to the process of processing the song samples during the training of the autoencoder network, and the corresponding audio representation is obtained through masking operations and splicing operations. Furthermore, the process of generating a song fusion representation through the trained autoencoder network is to input the processed audio representation obtained in the above steps into the trained autoencoder network, and compress the processed audio representation through the encoder in the trained autoencoder network, and finally obtain the song fusion representation corresponding to the song to be processed.

[0081] In a specific embodiment, the process of generating a song fusion representation corresponding to the song to be processed by the trained autoencoder network may include: the encoder in the trained autoencoder network processes the received processed audio representation to obtain the compressed initial fusion representations corresponding to the song to be processed output by the encoder; wherein different initial fusion representations correspond to different processed audio representations; averaging the initial fusion representations to obtain the song fusion representation corresponding to the song to be processed. Specifically, since the processed audio representation corresponds to the initial audio representation of the song to be processed, a single processed audio representation is an audio representation obtained by masking and splicing several initial audio representations output by several large models, so a single song to be processed will correspond to multiple processed audio representations; then the trained autoencoder network compresses each processed audio representation to obtain each compressed initial fusion representation corresponding to the single song to be processed; further, for a single song to be processed, the initial fusion representations related to it can be averaged to obtain a song fusion representation corresponding to the single song to be processed containing all audio representations. The song fusion representation obtained by the above steps can be used for music-related downstream tasks.

[0082] It can be seen that the present application can utilize the encoder of the trained autoencoder network to compress the audio representation corresponding to the processed song, and can obtain a compressed representation with a smaller dimension, so that the final output song fusion representation is more flexible and usable; and the trained autoencoder network is trained and learned using the loss between the audio representation before and after the mask operation and the final decoded audio representation, and has good representation fusion generalization and the ability to restore the original representation, so that a higher quality and more comprehensive fusion representation can be output, which can better adapt to downstream music tasks.

[0083] The above embodiment introduces the process of using the trained autoencoder network to process the processed song to obtain the corresponding song fusion representation, and the final song fusion representation can be used for music-related downstream tasks; the following embodiment will specifically introduce the process of using song fusion representation for downstream tasks in combination with song recommendation, such as Figure 4 As shown, the present application provides a song recommendation method, comprising:

[0084] Step S31, obtaining a song fusion representation corresponding to the song to be processed; the song fusion representation is a song fusion representation obtained by using a song fusion representation generation method.

[0085] In the embodiment of the present application, the song fusion representation obtained by the trained autoencoder network after processing the song to be processed can be directly obtained, and the song fusion representation can be applied to the downstream song recommendation task; it can be understood that the original audio representation of the input trained autoencoder network corresponding to the song fusion representation is also the (initial) audio representation obtained by the song fusion representation generation method of the above embodiment; and the trained autoencoder network here is an autoencoder network trained by the above-mentioned autoencoder network training method.

[0086] Step S32, calculating the spatial distance between the song fusion representation and several preset reference fusion representations to obtain the corresponding spatial distance; the several reference fusion representations are fusion representations obtained by processing the historical songs set as liked and disliked by the target user using the trained autoencoder network.

[0087] Furthermore, in the process of song recommendation, it is necessary to prepare in advance several historical songs that are set as liked and disliked by the target user, and process these historical songs through the trained autoencoder network to obtain the audio representation of the historical songs that the target user likes, and the audio representation of the historical songs that the target user dislikes. Accordingly, the spatial distance calculation can be performed between the song fusion representation corresponding to the song to be processed and the audio representations of the two historical songs corresponding to the target user, and the spatial distance between the song fusion representation corresponding to the song to be processed and the audio representation corresponding to the historical songs that the target user likes, and the spatial distance between the song fusion representation corresponding to the song to be processed and the audio representation corresponding to the historical songs that the target user dislikes can be obtained.

[0088] Step S33: determine whether to recommend the song to be processed corresponding to the song fusion representation to the target user based on the relationship between the preset spatial distance threshold and the spatial distance.

[0089] In this embodiment, the song to be processed can be judged according to the size relationship between the spatial distances between the song fusion representation corresponding to the song to be processed and the audio representations corresponding to the two historical songs; specifically, first determine the first distance between the song fusion representation corresponding to the song to be processed and the audio representations corresponding to the historical songs liked by the target user, and the second distance between the song fusion representation and the audio representations corresponding to the historical songs disliked by the target user; if the first distance is less than the first preset spatial distance threshold, and the second distance is greater than the second preset spatial distance threshold, the song to be processed corresponding to the song fusion representation can be judged as a song that the target user may like, so as to recommend the song to the target user.

[0090] It can be seen that in the embodiments of the present application, a trained autoencoder network can be obtained through the above embodiments, and the corresponding song fusion representation can be obtained by processing the song to be processed through the trained autoencoder network; then the song fusion representation can be used in the downstream song recommendation task to determine whether the relevant songs need to be recommended to the target user. In the process, the trained autoencoder network needs to be used to process the historical songs that the target user has previously set to like and dislike, so as to calculate the spatial distance between the audio representation corresponding to the historical songs and the song fusion representation corresponding to the song to be processed, and then determine whether to recommend the song corresponding to the song fusion representation to the target user based on the spatial distance. The trained autoencoder network here has good representation fusion generalization and the ability to restore the original representation, and combines the advantages of different large models to process audio representations, and can output more comprehensive audio representations, which can improve the accuracy of the extracted audio representations and improve the song recommendation effect of downstream tasks.

[0091] like Figure 5 As shown, the embodiment of the present application discloses a method for training an autoencoder network, which specifically includes:

[0092] In the embodiment of the present application, firstly, any song is selected from the music library as a song sample, and then the single song sample is sliced ​​in turn to obtain several 15-second segments corresponding to the single song sample (since the input of most music representation large models is a segment of less than 15 seconds, 15 seconds can be used as the preset song segment length in the above embodiment). Then each song segment is input into multiple different music representation large models, such as the MULE model, the LAION-CLAP model, and the MERT model, to obtain the first audio representations (representation 1, representation 2, and representation 3) obtained by the large model processing the song segment. It can be understood that the currently commonly used audio representation large models have their own characteristics and advantages due to their different structures and training. For example, the MULE model, the LAION-CLAP model, and the MERT model have their own strengths in different aspects; accordingly, the first audio representations corresponding to the song samples obtained by processing the song samples using these audio representation models also have characteristics in different aspects; for example, the first audio representation obtained by processing a certain audio representation model performs better in timbre tasks, or performs better in music genre classification.

[0093] It should be pointed out that different large models accept different input lengths and output dimensions. First, for different input lengths, if the large model supports 15 seconds of input, such as the MERT model, then directly input a 15-second audio clip and extract the representation embedding. If the model supports inputs of less than 15 seconds, such as LAION-CLAP and MULE support 10 seconds of input, you can extract the embeddings of the 0-10 second and 5-15 second clips and calculate the average as the representation embedding of the 15-second clip. This multi-segment average method can be used for large models that support inputs of less than 15 seconds.

[0094] Furthermore, since different large representation models have different representation dimensions, for example, the output dimension of MULE is 1536, LAION-CLAP is 512, and MERT is 1024. In this embodiment, the audio representations output by different large models can be masked by using a multi-segment proportional random masking method; specifically, assuming that the mask ratio is 40%, 1536*0.4=614 feature points are randomly selected and set to 0 in the audio representation (representation 1) output by MULE, 512*0.4=204 feature points are selected and set to 0 in the audio representation (representation 2) output by LAION-CLAP, and 1024*0.4=409 feature points are selected and set to 0 in the audio representation (representation 3) output by MERT. Then, the masked audio representations are spliced ​​to obtain the masked spliced ​​representation and input into the convolutional autoencoder network (or fully connected autoencoder network), and the encoder compresses the representation. The size of the compression can be controlled by controlling the number of convolution kernels of the last network layer of the encoder. The decoder restores the compressed representation to obtain the decoded representation, and the output of the last network layer of the decoder is consistent with the scale of the input of the encoding layer.

[0095] Furthermore, when training the autoencoder network, mean square error loss can be used for training, and the network loss can be composed of the concatenated representation (directly concatenated by representation 1, representation 2 and representation 3) and the decoded representation, the masked concatenated representation and the decoded representation;

[0096] ;

[0097] , represents the representation of the decoder output corresponding to the i-th song segment of the current batch, represents the splicing representation, represents the masked concatenated representation, and M is the number of fragments in the current batch. a is a weight parameter, which is used to control the proportion of concatenated representation and masked concatenated representation in the loss function. The larger a is, the smaller the weight of the masked representation, the smaller the randomness of the model, and the worse the generalization performance; on the contrary, the larger the weight of the masked representation, the greater the randomness, the more difficult it is to converge the training, but the generalization performance will be stronger, so a is an adjustable parameter. Through this loss function, the autoencoder network can be trained to converge the MSE (Mean Square Error) loss, and then the final autoencoder network can be saved to obtain the trained autoencoder network.

[0098] In the process of using the trained autoencoder network to generate a song fusion representation for the song to be processed, the song is also sliced, and the corresponding concatenated representation is obtained and then input into the encoder of the trained autoencoder network, and the initial fusion representation is obtained from the output of the encoder. For multiple clips of a song, it is necessary to average the initial fusion representation to obtain a compressed song fusion representation corresponding to the song to be processed; the song fusion representation can be used for related downstream tasks later.

[0099] It can be seen that the present application reduces the dimensionality of audio representation through masking operations and splicing operations, which can enhance the representation learning ability of the autoencoder network and improve the training efficiency; subsequently, the encoder of the trained autoencoder network can be used to compress the audio representation corresponding to the song to be processed, and a compressed representation with a smaller dimension can be obtained, making the final output song fusion representation more flexible and usable; and, the trained autoencoder network is trained and learned using the loss between the audio representation before and after the masking operation and the final decoded audio representation during the training process, and has good representation fusion generalization and the ability to restore the original representation; in this way, the trained autoencoder model can output a higher quality and more comprehensive fusion representation, which can better adapt to downstream music tasks.

[0100] Furthermore, the present application also discloses an electronic device. Figure 6 This is a structural diagram of an electronic device 20 according to an exemplary embodiment. The content in the diagram cannot be regarded as any limitation on the scope of use of the present application.

[0101] Figure 6 A schematic diagram of the structure of an electronic device 20 provided in an embodiment of the present application. The electronic device 20 may specifically include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. The memory 22 is used to store a computer program, which is loaded and executed by the processor 21 to implement the relevant steps in the autoencoder network training method disclosed in any of the aforementioned embodiments. In addition, the electronic device 20 in this embodiment may specifically be an electronic computer.

[0102] In this embodiment, the power supply 23 is used to provide working voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and the external device, and the communication protocol it follows is any communication protocol that can be applied to the technical solution of the present application, and is not specifically limited here; the input and output interface 25 is used to obtain external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs and is not specifically limited here.

[0103] In addition, the memory 22, as a carrier for storing resources, can be a read-only memory, a random access memory, a disk or an optical disk, etc. The resources stored thereon can include an operating system 221, a computer program 222, etc., and the storage method can be temporary storage or permanent storage.

[0104] The operating system 221 is used to manage and control the hardware devices and computer program 222 on the electronic device 20, which can be Windows Server, Netware, Unix, Linux, etc. In addition to including a computer program that can be used to complete the self-encoding network training method performed by the electronic device 20 disclosed in any of the aforementioned embodiments, the computer program 222 can further include a computer program that can be used to complete other specific tasks.

[0105] Furthermore, the present application also discloses a computer-readable storage medium for storing a computer program; wherein, when the computer program is executed by a processor, the aforementioned disclosed autoencoder network training method is implemented. The specific steps of the method can refer to the corresponding contents disclosed in the aforementioned embodiments, and will not be repeated here.

[0106] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0107] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0108] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0109] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0110] The technical solution provided by the present application is introduced in detail above. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea. At the same time, for general technicians in this field, according to the idea of ​​the present application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A method for training an autoencoder network, characterized in that: include: Obtaining a plurality of first audio representations obtained after at least two large models process the song samples; Performing a preset process on each of the first audio representations to obtain a second audio representation; the preset process includes a masking operation and a splicing operation on the audio representation; Inputting the second audio representation into the autoencoder network to be trained, so as to train the autoencoder network to be trained using the target loss function, and obtaining a trained autoencoder network for generating a song fusion representation corresponding to the song to be processed; The target loss function includes a loss function constructed based on the second audio representation and a decoded audio representation, and the decoded audio representation is an audio representation obtained after the decoder in the autoencoder network to be trained decodes the second audio representation.

2. The autoencoder network training method according to claim 1, characterized in that: The obtaining of a plurality of first audio representations obtained after processing the song sample by at least two large models includes: Slicing the song sample based on the preset song segment duration to obtain a number of song segments; Determine the size relationship between the input sample duration supported by each of the large models and the preset song segment duration, and process the song segment through each of the large models and based on the corresponding size relationship to obtain corresponding first audio representations.

3. The autoencoder network training method according to claim 2, characterized in that: The song segments are processed by each of the large models based on the corresponding size relationship to obtain corresponding first audio representations, including: If the duration of the first input sample supported by the first large model of the at least two large models is consistent with the duration of the preset song segment, the song segment to be processed is directly input into the corresponding first large model for processing to obtain a corresponding first audio representation; If the second input sample duration supported by the second largest model of the at least two large models is less than the preset song segment duration, the song segment to be processed is cropped at least twice to obtain at least two cropped segments, and the at least two cropped segments are input into the second largest model, so that the second largest model extracts and averages the audio representations of each of the cropped segments to obtain the corresponding first audio representation; The duration of each of the cropped segments is consistent with the duration of the second input sample, and the contents of the at least two cropped segments include all the contents of the corresponding song segments.

4. The autoencoder network training method according to claim 1, characterized in that: The performing preset processing on each of the first audio representations to obtain a second audio representation includes: Performing random masking operations on the first audio representations according to a preset masking ratio to obtain corresponding masked audio representations; A splicing operation is performed on the plurality of masked audio representations to obtain a second audio representation corresponding to the song sample.

5. The autoencoder network training method according to claim 1, characterized in that: The performing preset processing on each of the first audio representations to obtain a second audio representation includes: Performing a splicing operation on the plurality of the first audio representations to obtain a spliced ​​audio representation; A random masking operation is performed on the concatenated audio representation to obtain a second audio representation corresponding to the song sample.

6. The autoencoder network training method according to any one of claims 1 to 5, characterized in that: The target loss function is a loss function constructed based on a first loss between the second audio representation and the decoded audio representation; Accordingly, the training of the autoencoder network to be trained by using the target loss function includes: Calculating the loss between the second audio representation and the decoded audio representation using the target loss function to obtain a corresponding loss calculation result; The autoencoder network to be trained is adjusted based on the loss calculation result.

7. The autoencoder network training method according to any one of claims 1 to 5, characterized in that: The target loss function is a loss function constructed based on a first loss between the second audio representation and the decoded audio representation and a second loss between the third audio representation and the decoded audio representation; wherein the third audio representation is an audio representation obtained by directly concatenating the plurality of first audio representations; Accordingly, the training of the autoencoder network to be trained by using the target loss function includes: Calculating a first loss between the second audio representation and the decoded audio representation and a second loss between the third audio representation and the decoded audio representation using the target loss function to obtain corresponding loss calculation results; The autoencoder network to be trained is adjusted based on the loss calculation result.

8. The autoencoder network training method according to claim 7, characterized in that: Also includes: Based on the requirements of model generalization performance and model convergence speed, the weight values ​​corresponding to the first loss and the second loss in the objective loss function are adjusted.

9. A method for generating song fusion representation, characterized in that: include: Acquire several initial audio representations obtained after processing the song to be processed by at least two large models, and perform preset processing on each of the initial audio representations to obtain a processed audio representation; the preset processing includes a masking operation and a splicing operation on the audio representation; Inputting the processed audio representation into the trained autoencoder network to obtain a song fusion representation corresponding to the song to be processed generated by the trained autoencoder network; The trained autoencoder network is a trained autoencoder network obtained by using the autoencoder network training method according to any one of claims 1 to 8.

10. The song fusion representation generation method according to claim 9, characterized in that: The process of generating a song fusion representation corresponding to the song to be processed by the trained autoencoder network includes: The encoder in the trained autoencoder network processes the received processed audio representation to obtain compressed initial fusion representations corresponding to the song to be processed output by the encoder; wherein different initial fusion representations correspond to different processed audio representations; The average of each of the initial fusion representations is calculated to obtain a song fusion representation corresponding to the song to be processed.

11. A song recommendation method, characterized in that: include: Obtain song fusion representation corresponding to the song to be processed; The song fusion representation is a song fusion representation obtained by using the song fusion representation generation method according to claim 9; Calculating the spatial distance between the song fusion representation and a plurality of preset reference fusion representations to obtain corresponding spatial distances; The reference fusion representations are fusion representations obtained by processing the historical songs that the target user sets as liked and disliked by using the trained autoencoder network; Based on the size relationship between a preset spatial distance threshold and the spatial distance, it is determined whether to recommend the song to be processed corresponding to the song fusion representation to the target user.

12. An electronic device, characterized in that: include: Memory, used to store computer programs; A processor, configured to execute the computer program to implement the method according to any one of claims 1 to 11.

13. A computer-readable storage medium, characterized in that: Used to store a computer program, which implements the method according to any one of claims 1 to 11 when executed by a processor.

14. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the method according to any one of claims 1 to 11 is implemented.

Images