Network training, song recommendation, and fusion representation generation methods, devices, media, and products

Abstract

This application discloses a method, device, medium, and product for network training, song recommendation, and fusion representation generation, relating to the field of audio processing technology. The method includes: acquiring several first audio representations obtained after processing song samples using at least two large models; performing pre-processing on each first audio representation to obtain a second audio representation; the pre-processing includes masking and concatenation operations on the audio representations; and inputting the second audio representations into an autoencoder network to be trained, using a target loss function to train the autoencoder network, resulting in a trained autoencoder network used to generate a fusion representation of the song to be processed. In this way, this application reduces the dimensionality of the audio representations through masking and concatenation operations, thereby improving training efficiency; and by using a loss function constructed based on the second audio representations and the decoded audio representations for training, it ensures that the autoencoder model has the ability to generate representations, improving model generalization.

Description

Technical Field

[0001] This invention relates to the field of audio processing technology, and in particular to a method, device, medium, and product for network training, song recommendation, and fusion representation generation. Background Technology

[0002] Several large-scale music understanding models are currently active in the industry, bringing better results to downstream music tasks. However, these models each have their own strengths and weaknesses; some perform better in genre classification, while others perform better in timbre tasks. No single large-scale music understanding model can dominate. For example, MERT (Music Understanding Model with Large-Scale Self-supervised Training), LAION-CLAP (Contrastive Language-Audio Pretraining, an open-source contrastive language-audio pretraining model from LAION), and MULE (Musicset Unsupervised Large Embedding) each have their own strengths in different aspects. Traditional feature fusion methods involve concatenating different features. Direct concatenation can lead to dimensionality issues. Many large-scale models have representation vectors with thousands of dimensions. Direct concatenation may result in excessive dimensionality and slow training for downstream tasks.

[0003] Therefore, it is evident that how to combine the advantages of different large music models to fuse the audio representations output by various models is a problem that needs to be solved in this field. Summary of the Invention

[0004] In view of this, the purpose of this invention is to provide a method, device, medium, and product for network training, song recommendation, and fusion representation generation. By reducing the dimensionality of audio representations through masking and concatenation operations, the training efficiency of autoencoder networks can be improved. Furthermore, by using a target loss function constructed based on a second audio representation and the decoded audio representation to train the autoencoder network, the ability to generate representations can be ensured, thus improving the model's generalization ability. The specific solution is as follows:

[0005] Firstly, this application provides a method for training an autoencoder network, including:

[0006] Obtain several first audio representations obtained after processing song samples using at least two large models;

[0007] Each of the first audio representations is subjected to a preset processing to obtain a second audio representation; the preset processing includes masking operations and splicing operations on the audio representations;

[0008] The second audio representation is input into the autoencoder network to be trained, and the autoencoder network to be trained is trained using the target loss function to obtain a trained autoencoder network for generating the song fusion representation corresponding to the song to be processed.

[0009] The target loss function includes a loss function constructed based on the second audio representation and the decoded audio representation, wherein the decoded audio representation is the audio representation obtained by the decoder in the autoencoder network to be trained after decoding the second audio representation.

[0010] Optionally, the acquisition of several first audio representations obtained after processing song samples using at least two large models includes:

[0011] Based on the preset song segment length, the song sample is sliced ​​to obtain several song segments;

[0012] The relationship between the duration of the input samples supported by each of the large models and the duration of the preset song segment is determined, and the song segment is processed by each of the large models based on the corresponding relationship to obtain several first audio representations.

[0013] Optionally, the process of processing the song segments using each of the large models and based on the corresponding size relationships to obtain several corresponding first audio representations includes:

[0014] If the duration of the first input sample supported by the first of the at least two large models is consistent with the duration of the preset song segment, then the song segment to be processed is directly input into the corresponding first large model for processing to obtain the corresponding first audio representation.

[0015] If the duration of the second input sample supported by the second largest model among the at least two large models is less than the duration of the preset song segment, then the song segment to be processed is cropped at least twice to obtain at least two cropped segments, and the at least two cropped segments are input into the second largest model so that the second largest model can extract the audio representation of each cropped segment and calculate the average to obtain the corresponding first audio representation.

[0016] The duration of each cropped segment is consistent with the duration of the second input sample, and the content of the at least two cropped segments includes all the content of the corresponding song segment.

[0017] Optionally, the step of performing preset processing on each of the first audio representations to obtain the second audio representation includes:

[0018] According to a preset masking ratio, a number of the first audio representations are randomly masked to obtain a number of masked audio representations.

[0019] The several masked audio representations are concatenated to obtain the second audio representation corresponding to the song sample.

[0020] Optionally, the step of performing preset processing on each of the first audio representations to obtain the second audio representation includes:

[0021] A splicing operation is performed on several of the first audio representations to obtain a spliced ​​audio representation;

[0022] A random masking operation is performed on the spliced ​​audio representation to obtain the second audio representation corresponding to the song sample.

[0023] Optionally, the target loss function is a loss function constructed based on the first loss between the second audio representation and the decoded audio representation;

[0024] Accordingly, training the autoencoder network to be trained using the target loss function includes:

[0025] The loss between the second audio representation and the decoded audio representation is calculated using the target loss function to obtain the corresponding loss calculation result;

[0026] The autoencoder network to be trained is adjusted based on the loss calculation results.

[0027] Optionally, the target loss function is a loss function constructed based on a first loss between the second audio representation and the decoded audio representation, and a second loss between the third audio representation and the decoded audio representation; wherein, the third audio representation is an audio representation obtained by directly concatenating the plurality of first audio representations;

[0028] Accordingly, training the autoencoder network to be trained using the target loss function includes:

[0029] The target loss function is used to calculate the first loss between the second audio representation and the decoded audio representation, and the second loss between the third audio representation and the decoded audio representation, so as to obtain the corresponding loss calculation results;

[0030] The autoencoder network to be trained is adjusted based on the loss calculation results.

[0031] Optionally, the method further includes:

[0032] The weight values ​​corresponding to the first loss and the second loss in the target loss function are adjusted based on the requirements of model generalization performance and model convergence speed.

[0033] Secondly, this application provides a method for generating song fusion representations, including:

[0034] Obtain several initial audio representations obtained after processing the song to be processed by at least two large models, and perform preset processing on each of the initial audio representations to obtain the processed audio representations; the preset processing includes masking operation and splicing operation on the audio representations.

[0035] The processed audio representation is input into a post-trained autoencoder network to obtain a song fusion representation generated by the post-trained autoencoder network corresponding to the song to be processed; the post-trained autoencoder network is a post-trained autoencoder network obtained using the autoencoder network training method described above.

[0036] Optionally, the process of generating a song fusion representation corresponding to the song to be processed by the trained autoencoder network includes:

[0037] The encoder in the post-trained autoencoder network processes the received processed audio representation to obtain each compressed initial fusion representation output by the encoder corresponding to the song to be processed; wherein, different initial fusion representations correspond to different processed audio representations.

[0038] The average value of each initial fusion representation is calculated to obtain the song fusion representation corresponding to the song to be processed.

[0039] Thirdly, this application provides a song recommendation method, including:

[0040] Obtain the song fusion representation corresponding to the song to be processed; the song fusion representation is the song fusion representation obtained by the above-mentioned song fusion representation generation method;

[0041] Spatial distance is calculated between the song fusion representation and several pre-set reference fusion representations to obtain the corresponding spatial distance; the several reference fusion representations are fusion representations obtained by processing historical songs set as likes and dislikes by the target user using a trained autoencoder network.

[0042] Based on the relationship between a preset spatial distance threshold and the spatial distance, it is determined whether to recommend the song to be processed corresponding to the song fusion representation to the target user.

[0043] Fourthly, this application provides an electronic device, comprising:

[0044] Memory, used to store computer programs;

[0045] A processor for executing the computer program to implement the method as described above.

[0046] Fifthly, this application provides a computer-readable storage medium for storing a computer program that, when executed by a processor, implements the method described above.

[0047] Sixthly, this application provides a computer program product, including a computer program / instructions that, when executed by a processor, implement the method described above.

[0048] Therefore, this application first obtains several first audio representations after processing song samples using at least two large models; then, it performs preset processing on each of the first audio representations to obtain second audio representations; the preset processing includes masking and concatenation operations on the audio representations; then, the second audio representations are input into the autoencoder network to be trained, and the autoencoder network to be trained is trained using a target loss function to obtain a trained autoencoder network for generating song fusion representations corresponding to the songs to be processed; wherein, the target loss function includes a loss function constructed based on the second audio representations and the decoded audio representations, and the decoded audio representations are the audio representations obtained by the decoder in the autoencoder network to be trained decoding the second audio representations. In this way, this application reduces the dimensionality of the audio representations through masking and concatenation operations, which can improve the representation learning ability of the autoencoder network and improve training efficiency; furthermore, using the target loss function constructed based on the second audio representations and the decoded audio representations to train the autoencoder network can ensure that the model has the ability to generate representations and improve the model's generalization; the autoencoder network trained in this way can output higher quality and more comprehensive fusion representations, which can better adapt to downstream music tasks. Attached Figure Description

[0049] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0050] Figure 1 This is a diagram illustrating the architecture of an autoencoder network training method disclosed in this application;

[0051] Figure 2 This is a flowchart of a specific autoencoder network training method disclosed in this application;

[0052] Figure 3 This is a flowchart of a specific song fusion representation generation method disclosed in this application;

[0053] Figure 4 This application discloses a specific song recommendation method flowchart;

[0054] Figure 5 This is a flowchart of another specific autoencoder network training method disclosed in this application;

[0055] Figure 6 This is a structural diagram of an electronic device disclosed in this application. Detailed Implementation

[0056] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0057] Existing large-scale music understanding models each have their strengths in different aspects. Traditional feature fusion methods involve concatenating different features, but direct concatenation introduces dimensionality issues. Many large models have representation vectors with thousands of dimensions, and direct concatenation can lead to excessive dimensionality and slow training for downstream tasks. To address this, this application provides an autoencoder network training method, device, storage medium, and program product. By reducing the dimensionality of audio representations through masking and concatenation operations, the training efficiency of the autoencoder network can be improved. Furthermore, by using a target loss function constructed based on the second audio representation and the decoded audio representation to train the autoencoder network, the model's ability to generate representations can be ensured, improving its generalization ability. The autoencoder model trained in this way can output higher-quality and more comprehensive fused representations, better adapting to downstream music tasks.

[0058] The system framework used in the autoencoder network training method of this application can be found in [reference needed]. Figure 1 As shown, it may specifically include: a backend server 01 and a client 02 that establishes a connection with the backend server 01.

[0059] In this application, the user terminal 02 is used to obtain song samples for training the autoencoder network and songs to be processed for audio representation fusion. The user terminal 02 can be a client, such as a mobile phone, tablet, or computer, or a web page; no limitation is made here.

[0060] Furthermore, the backend server 01 performs the steps of training the autoencoder network and generating the song fusion representation, including acquiring song samples and songs to be processed sent by the user terminal 02. For the song samples, at least two large models are used to process the samples to obtain a corresponding first audio representation. Then, the first audio representation is processed through masking and concatenation operations to obtain a second audio representation. The second audio representation is then input into the autoencoder network to be trained, using a target loss function to train the network, resulting in a trained autoencoder network used to generate the song fusion representation corresponding to the song to be processed. Correspondingly, for the music to be processed, several initial audio representations are obtained after processing the song using at least two large models. These initial audio representations are then masked and concatenated to obtain a processed audio representation. This processed audio representation is then input into the trained autoencoder network to obtain the song fusion representation generated by the trained autoencoder network corresponding to the song to be processed.

[0061] See Figure 2 As shown, this embodiment of the invention discloses a method for training an autoencoder network, comprising:

[0062] Step S11: Obtain several first audio representations obtained after processing the song samples by at least two large models.

[0063] In this embodiment of the application, the first step is to extract audio representations from the song samples using at least two existing large models. For a single song sample, at least two large models can extract a corresponding number of audio representations. For example, by processing a single song sample using three large models, three audio representations can be obtained.

[0064] In one specific embodiment, obtaining several first audio representations after processing song samples using at least two large models may include: slicing the song samples based on a preset song segment duration to obtain several song segments; determining the size relationship between the input sample duration supported by each of the large models and the preset song segment duration; and processing the song segments using each of the large models based on the corresponding size relationship to obtain several corresponding first audio representations. Specifically, since the input sample durations corresponding to the large models are different, a song segment duration applicable to most large models can be pre-set so that the relevant song samples can be sliced ​​based on this song segment duration, and a single song sample can be sliced ​​into several song segments. Further, during the process of extracting audio representations of the song segments corresponding to the song samples using the large models, it is necessary to determine the size relationship between the input sample duration supported by the currently used large model and the preset song segment duration, and then extract audio representations of the song segments corresponding to the song samples based on this size relationship.

[0065] In another specific embodiment, the process of processing the song segment using each of the large models and based on the corresponding size relationship to obtain several corresponding first audio representations may include: if the duration of the first input sample supported by the first of the at least two large models is consistent with the duration of the preset song segment, then the song segment to be processed is directly input into the corresponding first large model for processing to obtain the corresponding first audio representation; if the duration of the second input sample supported by the second of the at least two large models is less than the duration of the preset song segment, then the song segment to be processed is cropped at least twice to obtain at least two cropped segments, and the at least two cropped segments are input into the second large model so that the second large model extracts the audio representation of each cropped segment and calculates the average to obtain the corresponding first audio representation; wherein the duration of each cropped segment is consistent with the duration of the second input sample, and the content of the at least two cropped segments includes all the content of the corresponding song segment. Specifically, if the duration of the first input sample supported by the current large model is consistent with the preset song segment duration, it means that the song segments obtained from previously slicing the song samples can be directly input into the current large model, and processed by the large model to obtain the corresponding first audio representation. Conversely, if the duration of the second input sample supported by the current large model is less than the preset song segment duration, it means that the song segments obtained from previously slicing the song samples cannot be directly input into the current large model and need further trimming. It is understandable that the process of trimming song segments requires ensuring that the content of all the trimmed segments contains the entire content of the corresponding song segment, and that the duration of each trimmed segment is consistent with the duration of the second input sample, so that the trimmed segments can be input into the large model corresponding to the duration of the second input sample, and processed by the large model to obtain the corresponding first audio representation. It is understandable that the first audio representation obtained through the above steps is an audio representation corresponding to the song sample, which includes the audio representations of several song segments corresponding to a single song sample; for a single song sample, at least two first audio representations can be obtained through processing by at least two large models.

[0066] Step S12: Perform preset processing on each of the first audio representations to obtain the second audio representation; the preset processing includes masking operation and splicing operation on the audio representations.

[0067] In this embodiment, the above steps can obtain several first audio representations corresponding to the song sample through processing with several large models; then, each first audio representation can be processed through masking and concatenation operations. Masking can reduce the representation dimensionality, and concatenation can merge the audio representations processed by the various large models together; by performing masking and concatenation operations on each audio representation, a second audio representation corresponding to the song sample can finally be obtained. It should be noted that the second audio representation obtained here is the audio representation obtained after masking and concatenating several first audio representations corresponding to a single song sample; it includes audio representations of several song segments corresponding to the song sample.

[0068] In one specific embodiment, the step of pre-processing each of the first audio representations to obtain a second audio representation may include: performing random masking operations on several of the first audio representations according to a preset masking ratio to obtain several masked audio representations; and concatenating the several masked audio representations to obtain the second audio representation corresponding to the song sample. Specifically, in the process of processing each of the first audio representations to obtain the second audio representation, several first audio representations may first be randomly masked according to a preset masking ratio, that is, the first audio representations output by each of the major models may be randomly masked to obtain several masked audio representations; then, the masked audio representations may be concatenated; thus, by masking and concatenating the first audio representations corresponding to the song sample, the second audio representation corresponding to the song sample can be obtained.

[0069] In another specific embodiment, the step of performing preset processing on each of the first audio representations to obtain the second audio representation may include: concatenating several of the first audio representations to obtain a concatenated audio representation; and performing a random masking operation on the concatenated audio representation to obtain the second audio representation corresponding to the song sample. Specifically, in the process of processing each of the first audio representations to obtain the second audio representation, the first audio representations may first be concatenated to obtain the corresponding concatenated audio representation, and then the concatenated audio representation may be uniformly randomized to finally obtain the second audio representation corresponding to the song sample.

[0070] Step S13: Input the second audio representation into the autoencoder network to be trained, and train the autoencoder network to be trained using the target loss function to obtain a trained autoencoder network for generating the song fusion representation corresponding to the song to be processed; wherein, the target loss function includes a loss function constructed based on the second audio representation and the decoded audio representation, and the decoded audio representation is the audio representation obtained by the decoder in the autoencoder network to be trained after decoding the second audio representation.

[0071] In this embodiment, during the training of the autoencoder network, a second audio representation is input into the autoencoder network to be trained, so that the target loss function can be used to train the autoencoder network. It is understood that the autoencoder network to be trained includes an encoder and a decoder. The encoder compresses the second audio representation obtained in the above steps, and then the decoder decodes the compressed audio representation to obtain the decoded audio representation corresponding to the second audio representation. It should be noted that the target loss function used in the training of the autoencoder network can be a loss function constructed based on the second audio representation and the decoded audio representation. The encoder in the autoencoder network is specifically trained using this loss function, ultimately training a post-trained autoencoder network to process the song to be processed and generate the corresponding song fusion representation.

[0072] In one specific embodiment, the target loss function is a loss function constructed based on a first loss between the second audio representation and the decoded audio representation. Correspondingly, training the autoencoder network to be trained using the target loss function may include: calculating the loss between the second audio representation and the decoded audio representation using the target loss function to obtain a corresponding loss calculation result; and adjusting the autoencoder network to be trained based on the loss calculation result. Specifically, the target loss function used to train the autoencoder network can be a loss function constructed based on the first loss between the second audio representation and the decoded audio representation. Specifically, the loss between the second audio representation and the decoded audio representation is calculated, and the loss calculation result between the second audio representation and the decoded representation can be obtained through the target loss function; then, the autoencoder network to be trained is trained based on this loss calculation result.

[0073] In another specific embodiment, the target loss function is a loss function constructed based on a first loss between the second audio representation and the decoded audio representation, and a second loss between the third audio representation and the decoded audio representation; wherein, the third audio representation is an audio representation obtained by directly concatenating the plurality of first audio representations; correspondingly, training the autoencoder network to be trained using the target loss function may include: calculating the first loss between the second audio representation and the decoded audio representation, and the second loss between the third audio representation and the decoded audio representation, using the target loss function to obtain the corresponding loss calculation results; and adjusting the autoencoder network to be trained based on the loss calculation results. Specifically, the target loss function used in the training process of the autoencoder network to be trained can also be a loss function constructed based on the first loss between the second audio representation and the decoded audio representation, and the second loss between the third audio representation and the decoded audio representation; the third audio representation used here is an audio representation obtained by directly concatenating the first audio representations corresponding to the song samples. Furthermore, during network training, the first loss between the second audio representation and the decoded audio representation, and the second loss between the third audio representation and the decoded audio representation are calculated using the target loss function to obtain the corresponding loss calculation results. Then, based on these loss calculation results, the autoencoder network to be trained is adjusted, ultimately resulting in a trained autoencoder network used to generate the song fusion representation of the song to be processed.

[0074] It should be noted that, in specific embodiments, the method may further include adjusting the weight values ​​corresponding to the first loss and the second loss in the target loss function based on the model's generalization performance requirements and convergence speed requirements. In other words, during the training of the autoencoder network to be trained based on the first loss and the second loss, the weight values ​​of the first loss and the second loss in the autoencoder network to be trained can be adjusted considering the network's generalization and the loss convergence speed. The first loss ensures that the model has the ability to generate representations, which can improve the model's generalization. The second loss ensures that the model has the ability to reconstruct the original representations. Ultimately, a trained autoencoder network can be obtained to generate a fused song representation for the song to be processed.

[0075] Therefore, this application reduces the dimensionality of audio representations through masking and concatenation operations, which can improve the representation learning ability of autoencoder networks and increase training efficiency. Furthermore, during training, the loss between the audio representations before and after the masking operation and the final decoded audio representation can be used to learn the autoencoder network, which can improve the generalization of representation fusion and the ability to restore the original representation. In this way, the trained autoencoder model can output higher quality and more comprehensive fusion representations, which can better adapt to downstream music tasks.

[0076] The above embodiments describe the training process of an autoencoder network, performing masking and concatenation operations on the first audio representations of song samples obtained from multiple large models. Reducing the dimensionality of the audio representations through masking and concatenation operations improves the training efficiency of the autoencoder network. The obtained second audio representations are then used to train the autoencoder network to be trained, ultimately resulting in a post-trained autoencoder network used to process the songs and generate corresponding song fusion representations. Using the loss between the audio representations before and after masking and the final decoded audio representation to train the autoencoder network improves the generalization of representation fusion and the ability to reconstruct the original representations. Thus, the post-trained autoencoder model can output higher-quality and more comprehensive fusion representations, better adapting to downstream music tasks. Furthermore, the following embodiments will specifically describe the process of generating song fusion representations using the post-trained autoencoder network, such as... Figure 3 As shown in the figure, this application discloses a method for generating song fusion representations, including:

[0077] Step S21: Obtain several initial audio representations obtained after processing the song to be processed by at least two large models, and perform preset processing on each of the initial audio representations to obtain the processed audio representations; the preset processing includes masking operation and splicing operation on the audio representations.

[0078] In this embodiment, after obtaining the trained autoencoder network through the autoencoder network training method, audio representation processing can be performed on the song to be processed. Specifically, firstly, several initial audio representations are obtained by extracting audio representations from at least two large models of the song to be processed. Then, masking and concatenation operations are performed on these initial audio representations to obtain the processed audio representation. It can be understood that the process of masking and concatenating the initial audio representations can be either masking each initial audio representation first, and then concatenating the corresponding masked audio representations to obtain the processed audio representation; or concatenating each initial audio representation first, and then masking the corresponding concatenated audio representation to obtain the processed audio representation.

[0079] Step S22: Input the processed audio representation into a post-trained autoencoder network to obtain a song fusion representation generated by the post-trained autoencoder network corresponding to the song to be processed; the post-trained autoencoder network is a post-trained autoencoder network obtained using an autoencoder network training method.

[0080] In this embodiment, the processed audio representation corresponding to the song to be processed can be obtained through the above steps. The process of obtaining the processed audio representation is similar to the processing of song samples during the training of the autoencoder network, both involving masking and concatenation operations to obtain the corresponding audio representation. Further, the process of generating the song fusion representation through the trained autoencoder network involves inputting the processed audio representation obtained in the above steps into the trained autoencoder network, and then compressing the processed audio representation using the encoder in the trained autoencoder network, ultimately obtaining the song fusion representation corresponding to the song to be processed.

[0081] In one specific embodiment, the process of the post-trained autoencoder network generating a song fusion representation corresponding to the song to be processed may include: the encoder in the post-trained autoencoder network processes the received processed audio representation to obtain compressed initial fusion representations output by the encoder corresponding to the song to be processed; wherein different initial fusion representations correspond to different processed audio representations; and the average of each initial fusion representation is calculated to obtain a song fusion representation corresponding to the song to be processed. Specifically, since the processed audio representation corresponds to the initial audio representation of the song to be processed, and a single processed audio representation is obtained by masking and concatenating several initial audio representations output by several large models, a single song to be processed will correspond to multiple processed audio representations; then, the post-trained autoencoder network compresses each processed audio representation to obtain compressed initial fusion representations corresponding to a single song to be processed; further, for a single song to be processed, the average of its related initial fusion representations can be calculated to obtain a song fusion representation containing all audio representations corresponding to the single song to be processed. The song fusion representation obtained through the above steps can then be used for music-related downstream tasks.

[0082] Therefore, this application can utilize the encoder of the post-trained autoencoder network to compress the audio representation corresponding to the song to be processed, thereby obtaining a compressed representation with smaller dimensions, making the final output song fusion representation more flexible and usable. Furthermore, the post-trained autoencoder network is trained using the loss between the audio representation before and after the masking operation and the final decoded audio representation, which has good representation fusion generalization and the ability to restore the original representation. This allows for the output of a higher quality and more comprehensive fusion representation, which can better adapt to downstream music tasks.

[0083] The above examples illustrate the process of using a post-trained autoencoder network to process the songs to be processed and obtain the corresponding song fusion representations. The resulting song fusion representations can then be used for music-related downstream tasks. The following examples will specifically illustrate the process of using song fusion representations for downstream tasks, such as song recommendation. Figure 4 As shown, this application provides a song recommendation method, including:

[0084] Step S31: Obtain the song fusion representation corresponding to the song to be processed; the song fusion representation is the song fusion representation obtained by using the song fusion representation generation method.

[0085] In this embodiment, the song fusion representation obtained by the post-trained autoencoder network after processing the song to be processed can be directly obtained. This song fusion representation can be applied to the downstream song recommendation task. It is understood that the original audio representation of the input post-trained autoencoder network corresponding to the song fusion representation is also the (initial) audio representation obtained by the song fusion representation generation method of the above embodiment. Furthermore, the post-trained autoencoder network here is the autoencoder network trained using the above autoencoder network training method.

[0086] Step S32: Calculate the spatial distance between the song fusion representation and several pre-set reference fusion representations to obtain the corresponding spatial distance; the several reference fusion representations are fusion representations obtained by processing historical songs set as likes and dislikes by the target user using a trained autoencoder network.

[0087] Furthermore, in the song recommendation process, several historical songs set as liked and disliked by the target user need to be prepared in advance. These historical songs are then processed by a trained autoencoder network to obtain audio representations of the target user's liked and disliked historical songs. Correspondingly, spatial distances can be calculated between the fused song representation of the song to be processed and the audio representations of the two types of historical songs corresponding to the target user. This yields the spatial distances between the fused song representation of the song to be processed and the audio representations of the target user's liked historical songs, as well as between the fused song representation of the song to be processed and the audio representations of the target user's disliked historical songs.

[0088] Step S33: Based on the relationship between the preset spatial distance threshold and the spatial distance, determine whether to recommend the song to be processed corresponding to the song fusion representation to the target user.

[0089] In this embodiment, the song to be processed can be judged based on the spatial relationship between the song fusion representation corresponding to the song to be processed and the audio representations corresponding to two historical songs. Specifically, a first distance is first determined between the song fusion representation corresponding to the song to be processed and the audio representation corresponding to a historical song liked by the target user, and a second distance is determined between the song fusion representation and the audio representation corresponding to a historical song disliked by the target user. If the first distance is less than a first preset spatial distance threshold and the second distance is greater than a second preset spatial distance threshold, the song to be processed corresponding to the song fusion representation can be determined as a song that the target user may like, so as to recommend the song to the target user.

[0090] Therefore, in this embodiment, a post-trained autoencoder network (PSC) can be obtained through the above embodiments, and the PSSC can be used to process the songs to be processed to obtain corresponding song fusion representations. Then, in the downstream song recommendation task, the song fusion representation can be used to determine whether to recommend relevant songs to the target user. During this process, the PSSC needs to process the target user's previously set favorite and disliked historical songs to calculate the spatial distance between the audio representations of the historical songs and the song fusion representations of the songs to be processed. Based on this spatial distance, it is determined whether to recommend the song corresponding to the song fusion representation to the target user. The PSSC here possesses good representation fusion generalization ability and the ability to restore the original representation. It also combines the advantages of different large models processing audio representations, enabling it to output more comprehensive audio representations, improve the accuracy of the extracted audio representations, and enhance the song recommendation effect of the downstream task.

[0091] like Figure 5 As shown in the figure, this application discloses an autoencoder network training method, which specifically includes:

[0092] In this embodiment, an arbitrary song is first selected from the music library as a song sample. Then, each song sample is sliced ​​sequentially to obtain several 15-second segments corresponding to a single song sample (since most music representation models input segments shorter than 15 seconds, 15 seconds can be used as the preset song segment length in the above embodiment). Each song segment is then input into multiple different music representation models, such as the MULE model, LAION-CLAP model, and MERT model, to obtain the first audio representations (representation 1, representation 2, and representation 3) obtained by the large model processing the song segment. It is understood that commonly used audio representation models have their own characteristics and advantages due to differences in structure and training. For example, the MULE model, LAION-CLAP model, and MERT model each have their strengths in different aspects. Correspondingly, the first audio representations corresponding to the song samples obtained by processing the song samples using these audio representation models also have different characteristics; for example, the first audio representation obtained by processing a certain audio representation model performs better on timbre tasks or on music genre classification.

[0093] It's important to note that different large models accept different input lengths and output dimensions. Firstly, for different input lengths, if a large model supports 15-second inputs, such as the MERT model, then a 15-second audio segment is directly input and its representation embedding is extracted. If the model supports inputs shorter than 15 seconds, such as LAION-CLAP and MULE which support 10-second inputs, then the embeddings of two segments, 0-10 seconds and 5-15 seconds, can be extracted and averaged to serve as the representation embedding for the 15-second segment. This multi-segment averaging method can be used for all large models that support inputs shorter than 15 seconds.

[0094] Furthermore, since different large-scale representation models output different representation dimensions—for example, MULE outputs 1536 dimensions, LAION-CLAP outputs 512, and MERT outputs 1024—this embodiment can use multi-segment proportional random masking to mask the audio representations output by different large-scale models. Specifically, assuming a masking ratio of 40%, 1536*0.4=614 feature points are randomly selected and set to 0 in the audio representation output by MULE (representation 1), 512*0.4=204 feature points are selected and set to 0 in the audio representation output by LAION-CLAP (representation 2), and 1024*0.4=409 feature points are selected and set to 0 in the audio representation output by MERT (representation 3). The masked audio representations are then concatenated to obtain a masked concatenated representation, which is then input into a convolutional autoencoder network (or a fully connected autoencoder network). The encoder compresses the representation, and the compression size can be controlled by controlling the number of convolutional kernels in the last network layer of the encoder. The decoder restores the compressed representation to obtain the decoded representation. The output of the last network layer of the decoder has the same scale as the input of the coding layer.

[0095] Furthermore, when training an autoencoder network, mean squared error loss can be used. The network loss can be composed of concatenated representations (obtained by directly concatenating representations 1, 2, and 3) and decoding representations, mask concatenation representations, and decoding representations.

[0096] ;

[0097] , This represents the decoder output representation corresponding to the i-th song segment in the current batch (a batch of samples). Indicates splicing representation, The loss function represents the masked concatenation representation, where M is the number of segments in the current batch. 'a' is a weight parameter used to control the proportion of the concatenated representation and the masked concatenation representation in the loss function. A larger 'a' results in a smaller weight for the masked representation, less randomness in the model, and poorer generalization performance; conversely, a larger weight for the masked representation increases randomness, making training more difficult to converge, but improving generalization. Therefore, 'a' is an adjustable parameter. This loss function can be used to train the autoencoder network until the MSE (Mean Square Error) loss converges. The final autoencoder network is then saved, resulting in the trained autoencoder network.

[0098] In the process of generating a song fusion representation of a song to be processed using a post-trained autoencoder network, the song is sliced, the corresponding spliced ​​representation is obtained and then input into the encoder of the post-trained autoencoder network. The initial fusion representation is obtained from the encoder output. For multiple segments of a song, the average of the obtained initial fusion representations is calculated to obtain the compressed song fusion representation corresponding to a song to be processed. The song fusion representation can then be used for related downstream tasks.

[0099] Therefore, this application reduces the dimensionality of audio representations through masking and concatenation operations, which can improve the representation learning ability of the autoencoder network and increase training efficiency. Subsequently, the encoder of the trained autoencoder network can be used to compress the audio representation corresponding to the song to be processed, resulting in a compressed representation with smaller dimensions, making the final output song fusion representation more flexible and usable. Furthermore, the trained autoencoder network is trained using the loss between the audio representations before and after the masking operation and the final decoded audio representation, which has good representation fusion generalization and the ability to restore the original representation. Thus, the trained autoencoder model can output higher quality and more comprehensive fusion representations, which can better adapt to downstream music tasks.

[0100] Furthermore, embodiments of this application also disclose an electronic device, Figure 6 This is a structural diagram of an electronic device 20 according to an exemplary embodiment. The content of the diagram should not be construed as limiting the scope of this application.

[0101] Figure 6 This is a schematic diagram of the structure of an electronic device 20 provided in an embodiment of this application. Specifically, the electronic device 20 may include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. The memory 22 stores a computer program, which is loaded and executed by the processor 21 to implement the relevant steps in the autoencoder network training method disclosed in any of the foregoing embodiments. Alternatively, the electronic device 20 in this embodiment may specifically be an electronic computer.

[0102] In this embodiment, the power supply 23 is used to provide operating voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and external devices, and the communication protocol it follows can be any communication protocol applicable to the technical solution of this application, and is not specifically limited here; the input / output interface 25 is used to acquire external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs, and is not specifically limited here.

[0103] In addition, the memory 22, as a carrier for resource storage, can be a read-only memory, random access memory, disk or optical disk, etc. The resources stored thereon can include operating system 221, computer program 222, etc., and the storage method can be temporary storage or permanent storage.

[0104] The operating system 221 is used to manage and control the various hardware devices on the electronic device 20 and the computer program 222, which may be Windows Server, Netware, Unix, Linux, etc. In addition to including a computer program capable of performing the self-encoding network training method executed by the electronic device 20 as disclosed in any of the foregoing embodiments, the computer program 222 may further include a computer program capable of performing other specific tasks.

[0105] Furthermore, this application also discloses a computer-readable storage medium for storing a computer program; wherein, when the computer program is executed by a processor, it implements the aforementioned disclosed autoencoder network training method. Specific steps of this method can be found in the corresponding content disclosed in the foregoing embodiments, and will not be repeated here.

[0106] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to in the method section.

[0107] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0108] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented directly by hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0109] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0110] The technical solutions provided in this application have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A method for training an autoencoder network, characterized in that, include: Obtain several first audio representations obtained after processing song samples using at least two large models; Each of the first audio representations is subjected to a preset processing to obtain a second audio representation; the preset processing includes masking operations and splicing operations on the audio representations; The second audio representation is input into the autoencoder network to be trained, and the autoencoder network to be trained is trained using the target loss function to obtain a trained autoencoder network for generating the song fusion representation corresponding to the song to be processed. The target loss function includes a loss function constructed based on the second audio representation and the decoded audio representation, wherein the decoded audio representation is the audio representation obtained by the decoder in the autoencoder network to be trained after decoding the second audio representation.

2. The autoencoder network training method according to claim 1, characterized in that, The acquisition of several first audio representations obtained after processing song samples using at least two large models includes: Based on the preset song segment length, the song sample is sliced ​​to obtain several song segments; The relationship between the duration of the input samples supported by each of the large models and the duration of the preset song segment is determined, and the song segment is processed by each of the large models based on the corresponding relationship to obtain several first audio representations.

3. The autoencoder network training method according to claim 2, characterized in that, The process of processing the song segments using the various large models and based on the corresponding size relationships to obtain several corresponding first audio representations includes: If the duration of the first input sample supported by the first of the at least two large models is consistent with the duration of the preset song segment, then the song segment to be processed is directly input into the corresponding first large model for processing to obtain the corresponding first audio representation. If the duration of the second input sample supported by the second largest model among the at least two large models is less than the duration of the preset song segment, then the song segment to be processed is cropped at least twice to obtain at least two cropped segments, and the at least two cropped segments are input into the second largest model so that the second largest model can extract the audio representation of each cropped segment and calculate the average to obtain the corresponding first audio representation. The duration of each cropped segment is consistent with the duration of the second input sample, and the content of the at least two cropped segments includes all the content of the corresponding song segment.

4. The autoencoder network training method according to claim 1, characterized in that, The step of performing preset processing on each of the first audio representations to obtain the second audio representation includes: According to a preset masking ratio, a number of the first audio representations are randomly masked to obtain a number of masked audio representations. The several masked audio representations are concatenated to obtain the second audio representation corresponding to the song sample.

5. The autoencoder network training method according to claim 1, characterized in that, The step of performing preset processing on each of the first audio representations to obtain the second audio representation includes: A splicing operation is performed on several of the first audio representations to obtain a spliced ​​audio representation; A random masking operation is performed on the spliced ​​audio representation to obtain the second audio representation corresponding to the song sample.

6. The autoencoder network training method according to any one of claims 1 to 5, characterized in that, The target loss function is a loss function constructed based on the first loss between the second audio representation and the decoded audio representation; Accordingly, training the autoencoder network to be trained using the target loss function includes: The loss between the second audio representation and the decoded audio representation is calculated using the target loss function to obtain the corresponding loss calculation result; The autoencoder network to be trained is adjusted based on the loss calculation results.

7. The autoencoder network training method according to any one of claims 1 to 5, characterized in that, The target loss function is a loss function constructed based on the first loss between the second audio representation and the decoded audio representation, and the second loss between the third audio representation and the decoded audio representation; wherein, the third audio representation is an audio representation obtained by directly concatenating the plurality of first audio representations; Accordingly, training the autoencoder network to be trained using the target loss function includes: The target loss function is used to calculate the first loss between the second audio representation and the decoded audio representation, and the second loss between the third audio representation and the decoded audio representation, so as to obtain the corresponding loss calculation results; The autoencoder network to be trained is adjusted based on the loss calculation results.

8. The autoencoder network training method according to claim 7, characterized in that, Also includes: The weight values ​​corresponding to the first loss and the second loss in the target loss function are adjusted based on the requirements of model generalization performance and model convergence speed.

9. A method for generating song fusion representations, characterized in that, include: Obtain several initial audio representations obtained after processing the song to be processed by at least two large models, and perform preset processing on each of the initial audio representations to obtain the processed audio representations; the preset processing includes masking operation and splicing operation on the audio representations. The processed audio representation is input into a post-trained autoencoder network to obtain a song fusion representation generated by the post-trained autoencoder network that corresponds to the song to be processed. The post-trained autoencoder network is a post-trained autoencoder network obtained using the autoencoder network training method as described in any one of claims 1 to 8.

10. The song fusion representation generation method according to claim 9, characterized in that, The process of generating a song fusion representation corresponding to the song to be processed by the trained autoencoder network includes: The encoder in the post-trained autoencoder network processes the received processed audio representation to obtain each compressed initial fusion representation output by the encoder corresponding to the song to be processed; wherein, different initial fusion representations correspond to different processed audio representations. The average value of each initial fusion representation is calculated to obtain the song fusion representation corresponding to the song to be processed.

11. A song recommendation method, characterized in that, include: Obtain the song fusion representation corresponding to the song to be processed; The song fusion representation is a song fusion representation obtained using the song fusion representation generation method described in claim 9; Spatial distance is calculated between the song fusion representation and several pre-set reference fusion representations to obtain the corresponding spatial distance; The aforementioned reference fusion representations are fusion representations obtained by processing historical songs set as likes and dislikes by the target user using a post-trained autoencoder network. Based on the relationship between a preset spatial distance threshold and the spatial distance, it is determined whether to recommend the song to be processed corresponding to the song fusion representation to the target user.

12. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor for executing the computer program to implement the method as claimed in any one of claims 1 to 11.

13. A computer-readable storage medium, characterized in that, Used to store a computer program, which, when executed by a processor, implements the method as described in any one of claims 1 to 11.

14. A computer program product comprising a computer program / instructions, characterized in that, When the computer program / instructions are executed by the processor, they implement the method described in any one of claims 1 to 11.

Images