Training methods for song representation output models and song processing methods

Abstract

This application relates to a training method for a song representation output model and a song processing method. The method includes: acquiring song training data for multiple sample pairs; for the current sample pair, based on the textual differences between the song description text in the current sample pair and the song description text in each of the other sample pairs, selecting multiple song description texts with relatively large textual differences from each other sample pair as negative samples of the song audio in the current sample pair; determining a first representation distance between the song audio and the song description text in the current sample pair, and determining a second representation distance between the song audio and the multiple negative samples in the current sample pair; and performing comparative learning training on the song representation output model based on the first and second representation distances until the trained song representation output model is obtained. This method can overcome the problem of semantic similarity between negative and positive samples, effectively improving the model training effect and model performance.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a training method for a song representation output model, a song processing method, a computer device, a computer-readable storage medium, and a computer program product. Background Art

[0002] A large-scale cross-modal song representation model based on contrastive learning can utilize the textual and musical information of a song to enable the model to learn the interaction of cross-modal information within the representation space.

[0003] In related technologies, positive samples and randomly selected negative samples are typically used to train a large-scale cross-modal song representation model based on contrastive learning. However, during model training, there may be semantic relationships between the textual information of the songs in the randomly selected negative samples and the textual information of the songs in the positive samples. The greater the semantic relationship, the more likely the use of semantically related negative samples will cause confusion during model training, leading to poor training results. Summary of the Invention

[0004] Therefore, it is necessary to provide a training method, song processing method, computer equipment, computer-readable storage medium, and computer program product for a song representation output model that can improve the training effect of the model, in response to the above-mentioned technical problems.

[0005] Firstly, this application provides a method for training a song representation output model. The method includes:

[0006] Acquire song training data for multiple sample pairs; each sample pair includes the song audio and song description text of the same song;

[0007] For the current sample pair, based on the text differences between the song description text in the current sample pair and the song description text in each of the other sample pairs, select multiple song description texts with relatively large text differences from each of the other sample pairs as negative samples of the song audio in the current sample pair; wherein the current sample pair is any sample pair among the multiple sample pairs, and the other sample pairs are sample pairs other than the current sample pair among the multiple sample pairs.

[0008] Determine a first representation distance between the song audio and the song description text in the current sample pair, with the song description text in the current sample pair serving as a positive sample of the song audio in the current sample pair, and determine a second representation distance between the song audio in the current sample pair and multiple negative samples;

[0009] Based on the first representation distance and the second representation distance, the song representation output model is trained by comparative learning until a trained song representation output model is obtained; the trained song representation output model is used to output the song audio representation and the song description text representation.

[0010] In one embodiment, the step of selecting multiple song description texts with relatively large text differences from each of the other sample pairs as negative samples of the song audio in the current sample pair, based on the text differences between the song description text in the current sample pair and the song description text in each of the other sample pairs, includes:

[0011] The song description text in the current sample pair is input into the text encoding model in the song representation output model to obtain the first text representation; the song description text in each of the other sample pairs is input into the text encoding model to obtain the second text representation.

[0012] Determine the representation distance between the first text representation and each of the second text representations;

[0013] Based on the sorting of multiple representation distances, select the representation distance with the highest sorting order and filter it by a preset number. Then, use the song description text of the other sample pairs corresponding to the selected representation distance as the negative sample of the song audio in the current sample pair.

[0014] In one embodiment, determining the first representation distance between the song audio and the song description text in the current sample pair includes:

[0015] The song description text in the current sample pair is input into the text encoding model in the song representation output model to obtain the first text representation;

[0016] The song audio from the current sample pair is input into the audio encoding model in the song representation output model to obtain the audio representation;

[0017] The first text representation and the audio representation are dimensionally transformed respectively, and the representation distance between the first text representation after dimensional transformation and the audio representation after dimensional transformation is taken as the first representation distance.

[0018] In one embodiment, determining the second representation distance between the song audio in the current sample pair and the plurality of negative samples includes:

[0019] The song audio from the current sample pair is input into the audio encoding model in the song representation output model to obtain the audio representation;

[0020] The song description text in each of the other sample pairs is input into the text encoding model in the song representation output model to obtain the second text representation of each negative sample;

[0021] The second text representation and the audio representation of each negative sample are subjected to dimensional transformation respectively. The representation distance between the second text representation after dimensional transformation and the audio representation after dimensional transformation is taken as the second representation distance.

[0022] In one embodiment, the step of performing comparative learning training on the song representation output model based on the first representation distance and the second representation distance until a trained song representation output model is obtained includes:

[0023] The target loss value is calculated by combining the ratio of the first representation distance to the second representation distance with a weighting parameter; the weighting parameter is determined based on the average distance between the first text representation of the positive sample and the second text representation of multiple negative samples.

[0024] Based on the target loss value, comparative learning is performed until the loss function converges, resulting in the trained song representation output model.

[0025] In one embodiment, determining the second representation distance between the song audio in the current sample pair and the plurality of negative samples includes:

[0026] In each iteration of the loss process, a portion of negative samples are randomly selected from the multiple negative samples, and a second representation distance is determined between the song audio in the current sample pair and the randomly selected negative samples each time.

[0027] The step of performing comparative learning training on the song representation output model based on the first representation distance and the second representation distance includes:

[0028] During each iteration of the loss process, the song representation output model is trained by comparison based on the first representation distance and the second representation distance determined each time.

[0029] Secondly, this application provides a song processing method. The method includes:

[0030] Obtain the audio and description text of the song to be processed;

[0031] The song audio and the song description text are input into the trained song representation output model to obtain the song audio representation and song description text representation of the song to be processed; the song audio representation and song description text representation of the song to be processed are close to each other in the representation space;

[0032] The trained song representation output model is obtained by training the song representation output model according to any one of the above methods.

[0033] Thirdly, this application also provides a computer device. The computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the steps of the training method for the song representation output model as described in the first aspect, and / or the steps of the song processing method as described in the second aspect.

[0034] Fourthly, this application also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program thereon, which, when executed by a processor, implements the steps of the training method for the song representation output model as described in the first aspect, and / or the steps of the song processing method as described in the second aspect.

[0035] Fifthly, this application also provides a computer program product. The computer program product includes a computer program that, when executed by a processor, implements the steps of the training method for the song representation output model as described in the first aspect, and / or the steps of the song processing method as described in the second aspect.

[0036] The aforementioned training method, song processing method, computer device, computer-readable storage medium, and computer program product for a song representation output model acquire song training data from multiple sample pairs. Each sample pair includes the song audio and song description text of the same song. For the current sample pair, based on the text differences between the song description text in the current sample pair and the song description text in each of the other sample pairs, multiple song description texts with relatively large text differences are selected as negative samples of the song audio in the current sample pair. Here, the current sample pair is any sample pair among the multiple sample pairs, and the other sample pairs are sample pairs other than the current sample pair. Then, the song audio in the current sample pair is determined to be negative samples of the song audio. The first representation distance between the song description text and the song description text in the current sample pair is used as a positive sample of the song audio in the current sample pair. The second representation distance between the song audio in the current sample pair and multiple negative samples is determined. Then, based on the first and second representation distances, the song representation output model is trained by contrastive learning until the trained song representation output model is obtained. The trained song representation output model is used to output the song audio representation and the song description text representation. This realizes the optimization of the processing of negative samples during the training process of the song representation output model, which can overcome the problem of semantic similarity between negative samples and positive samples, so that the model can be trained stably and effectively improve the model training effect and model performance. Attached Figure Description

[0037] Figure 1 This is a schematic diagram of the comparison learning process in one embodiment;

[0038] Figure 2 This is a flowchart illustrating the training method of a song representation output model in one embodiment;

[0039] Figure 3 This is a schematic diagram illustrating the comparative learning training process of a song representation output model in one embodiment.

[0040] Figure 4 This is a flowchart illustrating a song processing method in one embodiment;

[0041] Figure 5 This is a flowchart illustrating the training method for the song representation output model in another embodiment;

[0042] Figure 6 This is a structural block diagram of a training device for a song representation output model in one embodiment;

[0043] Figure 7 This is a structural block diagram of a song processing device in one embodiment;

[0044] Figure 8 is a diagram of the internal structure of a computer device in one embodiment;

[0045] Figure 9 FIG. 4 is a diagram showing the internal structure of a computer device in another embodiment. Detailed Implementation

[0046] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0047] like Figure 1 The illustrated traditional song-based contrastive learning model processing flow is a dual-tower structure. The model can receive input from both audio and text modalities; for example, it can accept the audio of song A and a text description (description of the song's content). The song audio can be used by a music understanding model to extract audio representations (e.g.,...). Figure 1 (mid-audio embedding), text descriptions can be extracted into text representations through language processing models (such as...) Figure 1 (Chinese text embedding), because the representation dimensions output by the music understanding model and the language processing model are different, the model output representation can be processed by two different MLP (Multilayer Perceptron) modules (such as... Figure 1MLP module 1 and MLP module 2), and after mapping, representations of the same dimension can be obtained (e.g. Figure 1 (Audio embedding 1 and text embedding 2) are used for comparative learning.

[0048] In the computation process of comparative learning, audio files of N songs and their corresponding text descriptions can be selected. However, when processing the current song, the text descriptions of the other songs are randomly selected, and they may be similar to the current song's text description in terms of text semantics, which may affect the subsequent training effect.

[0049] Large-scale music representation models are crucial for music understanding and processing, serving as a foundation for downstream song tasks. Obtaining robust song representations significantly improves processing performance for these tasks, making the training of such foundational models essential. Since song information primarily comprises audio and related textual descriptions, cross-modal music representation models based on contrastive learning align well with musical contexts. Traditional methods, however, often neglect negative sample filtering during training. The semantic similarity between negative and positive samples can confuse training. This application proposes a training method for a song representation output model that optimizes the training process to address the semantic similarity between negative and current samples, thereby enhancing model performance and providing more reliable song representations for downstream music tasks.

[0050] In an exemplary embodiment, Figure 2 As shown, a training method for a song representation output model is provided. This embodiment illustrates the application of this method to a terminal. It is understood that this method can also be applied to a server, or to a system including both a terminal and a server, and can be implemented through the interaction between the terminal and the server. The server can be a standalone server or a server cluster consisting of multiple servers; the terminal can be, but is not limited to, various personal computers, laptops, smartphones, tablets, and portable wearable devices.

[0051] In this embodiment, the method includes the following steps S201 to S204. Wherein:

[0052] In step S201, multiple sample pairs of song training data are obtained; each sample pair includes the song audio and song description text of the same song.

[0053] As an example, song description text can be used to describe the content associated with a song. Song description text can include descriptive tags associated with the song, such as song name, artist name, song region, song language, song genre, etc.

[0054] In practical applications, for training the song representation output model, multiple sample pairs can be obtained from the music library to construct song training data. Each sample pair can be an audio and text pair, such as the song audio and song description text of the same song.

[0055] In step S202, for the current sample pair, based on the text differences between the song description text in the current sample pair and the song description text in each other sample pair, several song description texts with relatively large text differences are selected from each other sample pair as negative samples of the song audio in the current sample pair.

[0056] The current sample pair can be any of the multiple sample pairs, and the other sample pairs can be sample pairs other than the current sample pair.

[0057] In specific implementations, such as Figure 3 As shown, for the current sample pair (such as...) Figure 3 This includes audio and text pairs of song audio 1 and text description 1. It can take sample pairs other than the current sample pair from multiple sample pairs in the song training data as other sample pairs (e.g., Figure 3 This includes audio-text pairs of song audio 2 and text description 2 (...including audio-text pairs of song audio K and text description K), and can calculate the difference between the song description text in the current sample pair and the song description text in each other sample pair, such as calculating... Figure 3 The text similarity scores sim12, sim13, ..., sim1K are obtained by comparing the text representation embedding of text 1 with the text representation embeddings of text 2, text 3, ..., text K respectively. These text similarity scores are then ranked, and song description texts with lower similarity scores are selected as negative samples of the song audio in the current sample pair (e.g., ...). Figure 3 The M text descriptions with the lowest similarity are selected, which are the text descriptions with the largest degree of difference.

[0058] In step S203, a first representation distance between the song audio and the song description text in the current sample pair is determined, the song description text in the current sample pair is used as a positive sample of the song audio in the current sample pair, and a second representation distance between the song audio in the current sample pair and multiple negative samples is determined.

[0059] For example, for the current sample pair, by obtaining the audio representation of its song audio and the text representation of its song description text, the Euclidean distance between the audio representation and the text representation in the representation space can be used as the first representation distance.

[0060] After obtaining the negative samples of the song audio in the current sample pair, during the contrastive loss calculation for the current sample pair, a subset of negative samples can be extracted from multiple negative samples. For example, m negative samples can be randomly selected from M negative samples (M>m), and then the negative samples of the song audio in the current sample pair (e.g., ...) can be extracted. Figure 3 The audio representation of the song audio 1) and the text representation of the song description text in m negative samples are selected. The Euclidean distance between the audio representation and the multiple text representations in the representation space can be used as the second representation distance.

[0061] In step S204, the song representation output model is trained by contrastive learning based on the first representation distance and the second representation distance until the trained song representation output model is obtained.

[0062] The trained song representation output model can be used to output both the song audio representation and the song description text representation. The training objective of the song representation output model is to make the song audio and text description of the same song closer in the representation space. By utilizing a large number of song audio recordings and their corresponding text descriptions, a song representation output model based on song contrastive learning can be constructed.

[0063] In one example, regarding the training process of a song representation output model using contrastive learning, this embodiment calculates the difference between the song description text in the current sample pair and the song description text in each of the other sample pairs. Based on the other sample pairs with the lowest text similarity, multiple negative samples of the song audio in the current sample pair are obtained. This avoids semantically similar negative samples from the selected negative samples. Subsequently, some negative samples can be extracted from multiple negative samples to perform contrastive learning training on the song representation output model. This optimizes the negative sample processing process, enabling the song representation output model based on song contrastive learning to be trained stably, achieving better model training results and helping to provide better song representations for downstream tasks.

[0064] In the training method of the above-mentioned song representation output model, by acquiring song training data of multiple sample pairs, for the current sample pair, based on the text differences between the song description text in the current sample pair and the song description text in each of the other sample pairs, multiple song description texts with relatively large text differences are selected as negative samples of the song audio in the current sample pair. Then, the first representation distance between the song audio and the song description text in the current sample pair is determined, and the song description text in the current sample pair is used as the positive sample of the song audio in the current sample pair. The second representation distance between the song audio in the current sample pair and multiple negative samples is determined. Then, based on the first and second representation distances, the song representation output model is trained by comparative learning until the trained song representation output model is obtained. This realizes the optimization of the processing of negative samples in the training process of the song representation output model, which can overcome the problem of semantic similarity between negative samples and positive samples, so that the model can be trained stably, effectively improving the model training effect and model performance.

[0065] In an exemplary embodiment, in step S202, for the current sample pair, based on the text differences between the song description text in the current sample pair and the song description text in each of the other sample pairs, selecting multiple song description texts with relatively large text differences from each of the other sample pairs as negative samples of the song audio in the current sample pair may include the following steps:

[0066] Input the song description text in the current sample pair into the text encoding model in the song representation output model to obtain the first text representation. Input the song description text in each other sample pair into the text encoding model to obtain the second text representation. Determine the representation distance between the first text representation and each second text representation. Select the representation distance with the highest ranking according to the size of the multiple representation distances and use the song description text of the other sample pairs corresponding to the selected representation distance as the negative sample of the song audio in the current sample pair.

[0067] As an example, the text encoding model can be a pre-trained language model based on the Transformer architecture, or it can use other language processing models.

[0068] In practical applications, the song description text in the current sample pair, as well as the song description text in each other sample pair, can be input into the text encoding model in the song representation output model (e.g., Figure 3 The BERT model is used to process the text to obtain the first text representation (such as...). Figure 3The system uses text 1 embedding and second text representations (such as text 2 embedding, ..., text K embedding) to calculate the Euclidean distance (i.e., representation distance) between text 1 embedding and other text 2 embeddings ... text K embeddings for the current sample pair. Figure 3 In the sim12...sim1k range, you can select the M text embeddings (i.e., other sample pairs) that are farthest apart in Euclidean distance (i.e., the preset number of filters) to obtain the negative samples of the song audio in the current sample pair.

[0069] For example, Euclidean distance is a distance metric that can be used to measure the straight-line distance between two points in a multidimensional space. In embedding space, the similarity or difference between two vectors can be measured by calculating the straight-line distance between them.

[0070] In this embodiment, the song description text in the current sample pair is input into the text encoding model in the song representation output model to obtain the first text representation. The song description text in each other sample pair is input into the text encoding model to obtain the second text representation. Then, the representation distance between the first text representation and each second text representation is determined. The representation distances that are ranked first according to the size of the multiple representation distances are selected according to a preset number of filtering steps. The song description text of other sample pairs corresponding to the selected representation distances are used as negative samples of the song audio in the current sample pair. This can filter out the negative samples with the lowest text similarity, providing data support for further model training based on negative samples.

[0071] In an exemplary embodiment, determining the first representation distance between the song audio and the song description text in the current sample pair in step S203 may include the following steps:

[0072] Input the song description text in the current sample pair into the text encoding model in the song representation output model to obtain the first text representation; input the song audio in the current sample pair into the audio encoding model in the song representation output model to obtain the audio representation; perform dimensional transformation on the first text representation and the audio representation respectively, and take the representation distance between the dimensionally transformed first text representation and the dimensionally transformed audio representation as the first representation distance.

[0073] As an example, the audio coding model can be a self-supervised music understanding model, or other music understanding models can be used.

[0074] In one example, the song audio in the current sample pair (e.g.) Figure 31) Select the song audio and input it into the audio coding model in the song representation output model (e.g., Figure 3 The MERT model is used to process the data to obtain the audio representation of the current sample pair (e.g., ...). Figure 3 (Mid-audio embedding). Because the output representation dimensions of the MERT and BERT models are inconsistent, they need to be mapped to the same dimensional space for loss calculation. Therefore, the text embedding and audio embedding of the current sample pair can be input into two different fully connected modules (e.g., ...). Figure 1 The fully connected module (MLP1 and MLP2) can be used to perform dimensional transformation on text representation and audio representation. By calculating the representation distance based on the transformed text representation and audio representation, the Euclidean distance (i.e., the first representation distance) between the audio representation of the song audio and the text representation of the song description text can be obtained for the current sample pair.

[0075] In an alternative embodiment, other networks may also be used to perform dimensionality transformation on the text and audio representations.

[0076] In this embodiment, the first text representation is obtained by inputting the song description text in the current sample pair into the text encoding model in the song representation output model, and the audio of the song in the current sample pair is input into the audio encoding model in the song representation output model to obtain the audio representation. Then, the first text representation and the audio representation are transformed in dimension, and the representation distance between the first text representation after dimension transformation and the audio representation after dimension transformation is taken as the first representation distance, which provides data support for further model comparison learning and training.

[0077] In an exemplary embodiment, determining the second representation distance between the song audio in the current sample pair and multiple negative samples in step S203 may include the following steps:

[0078] The audio of the song in the current sample pair is input into the audio encoding model in the song representation output model to obtain the audio representation; the song description text in each other sample pair is input into the text encoding model in the song representation output model to obtain the second text representation of each negative sample; the second text representation and audio representation of each negative sample are transformed in dimension, and the representation distance between the second text representation after dimension transformation and the audio representation after dimension transformation is taken as the second representation distance.

[0079] In a practical implementation, the audio representation of the current sample pair can be obtained similarly (e.g., Figure 3The audio embedding of the current sample pair and the text embeddings of multiple negative samples are then input into two different fully connected modules to obtain the Euclidean distance (i.e., the second representation distance) between the audio representation of the current sample pair and the multiple text representations.

[0080] In this embodiment, the audio of the song in the current sample pair is input into the audio encoding model in the song representation output model to obtain the audio representation. The text description of the song in each other sample pair is input into the text encoding model in the song representation output model to obtain the second text representation of each negative sample. Then, the second text representation and audio representation of each negative sample are transformed in dimension. The representation distance between the second text representation after dimension transformation and the audio representation after dimension transformation is used as the second representation distance, which provides data support for further model comparison learning and training.

[0081] In an exemplary embodiment, in step S204, the song representation output model is trained through contrastive learning based on the first representation distance and the second representation distance until the trained song representation output model is obtained. This may include the following steps:

[0082] The target loss value is calculated by combining the ratio of the first representation distance to the second representation distance with the weighting parameters. The weighting parameters are determined based on the average distance between the first text representation of positive samples and the second text representation of multiple negative samples. Comparative learning is performed based on the target loss value until the loss function converges, resulting in the trained song representation output model.

[0083] Specifically, when processing based on the current sample pairs during model training, the weighted contrastive loss can be calculated as follows:

[0084]

[0085] in, This represents the Euclidean distance (i.e., the first representation distance) between the audio representation of the song audio and the text representation of the song description text in the current sample pair after dimensionality transformation by MLP. This represents the Euclidean distance (i.e., the second representation distance) between the audio representation of the song audio in the current sample pair and the text representations of the randomly selected m negative sample song descriptions, after dimensionality transformation by MLP. The goal of the loss function is to minimize the loss value, that is, to make the distance between the audio representation of the song and the text description of the same song as close as possible, and to make the distance between the audio representation of the song and the text description of other songs as far as possible.

[0086] In one example, the weighted contrastive loss uses the first representation distance as the numerator and the sum of the second representation distances based on m negative samples as the denominator. A weighting parameter p_am is introduced, which represents the mean of the Euclidean distances between the text representation of the song description text of the current sample pair and the text representations of the song description texts of the m negative samples. The larger the distance, the larger the weighted contrastive loss, and the greater the optimization strength that can be achieved. By training until the loss function converges, the trained song representation output model can be obtained, which can be applied to downstream music tasks. Thus, by combining negative sample selection and weighting parameters, the optimization processing of negative samples is achieved, which can flexibly control different text similarities to achieve different optimization strengths, effectively ensuring the stability of model training.

[0087] Compared to traditional methods, which do not perform additional screening of negative text samples during training, the technical solution in this embodiment addresses the shortcomings in negative sample construction. For the song description text of the current sample pair, the Euclidean distance between the song description text and the text representations of other song description texts in the training data is calculated. Then, the M farthest negative samples are selected. In subsequent comparative loss calculations based on the current sample pair, m negative samples (M > m) are randomly selected from these M negative samples each time, and the loss function is weighted according to the text representation distance of the song description text. Through this negative sample screening and weighting strategy, training combinations with semantically similar positive and negative samples are avoided, ensuring that semantically similar negative samples do not hinder model learning during comparative learning. This effectively solves the problem of selecting negative text samples and improves the training stability of cross-modal comparative learning models.

[0088] In this embodiment, the target loss value is calculated by combining the ratio of the first representation distance to the second representation distance with weighting parameters. Then, comparative learning is performed based on the target loss value until the loss function converges, resulting in a trained song representation output model. This allows the model to be trained stably, effectively improving the model training effect and model performance.

[0089] In an exemplary embodiment, determining the second representation distance between the song audio in the current sample pair and multiple negative samples in step S203 may include the following steps:

[0090] In each iteration of the loss process, a portion of negative samples are randomly selected from multiple negative samples, and the second representation distance between the song audio in the current sample pair and the randomly selected negative samples is determined.

[0091] Based on the first representation distance and the second representation distance, the song representation output model is trained by comparative learning, including: in each iteration of loss, the song representation output model is trained by comparative learning based on the first representation distance and the second representation distance determined each time.

[0092] Optionally, in the process of calculating the contrastive loss based on the current sample pair, for each iteration, m negative samples can be randomly selected from the M negative samples, where M>m. This ensures that the loss will not use the same negative samples in each iteration when the song audio in the current sample pair is used to perform multiple iterations with the M negative samples, thus avoiding model overfitting.

[0093] In an exemplary embodiment, Figure 4 As shown, a song processing method is provided. This embodiment illustrates the application of this method to a server. It is understood that this method can also be applied to a terminal, or to a system including both a terminal and a server, and is implemented through the interaction between the terminal and the server. The server can be a standalone server or a server cluster consisting of multiple servers; the terminal can be, but is not limited to, various personal computers, laptops, smartphones, tablets, and portable wearable devices.

[0094] In this embodiment, the method includes the following steps S401 to S402. Wherein:

[0095] In step S401, the audio of the song to be processed and the description text of the song are obtained;

[0096] In step S402, the song audio and song description text are input into the trained song representation output model to obtain the song audio representation and song description text representation of the song to be processed; the song audio representation and song description text representation of the song to be processed are close to each other in the representation space.

[0097] The trained song representation output model is obtained by training the song representation output model of any one of the terms.

[0098] In practical applications, the audio and description text of the song to be processed can be obtained. The audio and description text can be input into the trained song representation output model to obtain the audio and description text representations that are close to each other in the representation space, thus providing a more reliable song representation for downstream music tasks.

[0099] In one embodiment, such as Figure 5 The diagram illustrates a flowchart of another method for training a song representation output model. In this embodiment, the method includes the following steps:

[0100] In step S501, song training data for multiple sample pairs are acquired; each sample pair includes the song audio and song description text of the same song. In step S502, for the current sample pair, the song description text is input into the text encoding model in the song representation output model to obtain a first text representation; the song description text of each other sample pair is input into the text encoding model to obtain a second text representation. In step S503, the representation distance between the first text representation and each second text representation is determined. In step S504, the representation distances ranked highest according to the size of the multiple representation distances are selected, and the song description text of the other sample pairs corresponding to the selected representation distances are used as negative samples of the song audio in the current sample pair. In step S505, the first representation distance between the song audio and the song description text in the current sample pair is determined, and the second representation distance between the song audio and multiple negative samples in the current sample pair is determined. In step S506, the ratio of the first representation distance to the second representation distance is used to calculate the target loss value, which is determined based on the average distance between the first text representation of positive samples and the second text representation of multiple negative samples. In step S507, contrastive learning is performed based on the target loss value until the loss function converges, resulting in the trained song representation output model. It should be noted that the specific limitations of the above steps can be found in the specific limitations of the training method for a song representation output model described above, and will not be repeated here.

[0101] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0102] Based on the same inventive concept, this application also provides a training device for a song representation output model to implement the training method for the song representation output model described above. The solution provided by this device is similar to the implementation described in the above method. Therefore, the specific limitations of one or more training device embodiments for song representation output models provided below can be found in the limitations of the training method for song representation output models described above, and will not be repeated here.

[0103] In one embodiment, such as Figure 6 As shown, a training device for a song representation output model is provided, comprising:

[0104] The training data acquisition module 601 is used to acquire song training data for multiple sample pairs; each sample pair includes the song audio and song description text of the same song;

[0105] The negative sample determination module 602 is used to, for the current sample pair, select multiple song description texts with relatively large text differences from each of the other sample pairs as negative samples of the song audio in the current sample pair, based on the text differences between the song description text in the current sample pair and the song description text in each of the other sample pairs; wherein the current sample pair is any one of the multiple sample pairs, and the other sample pairs are sample pairs other than the current sample pair among the multiple sample pairs.

[0106] The representation distance determination module 603 is used to determine the first representation distance between the song audio and the song description text in the current sample pair, wherein the song description text in the current sample pair is used as a positive sample of the song audio in the current sample pair, and to determine the second representation distance between the song audio in the current sample pair and the multiple negative samples.

[0107] The model training module 604 is used to perform comparative learning training on the song representation output model based on the first representation distance and the second representation distance until the trained song representation output model is obtained; the trained song representation output model is used to output the song audio representation and the song description text representation.

[0108] In one embodiment, the negative sample determination module 602 includes:

[0109] The text representation submodule is used to input the song description text in the current sample pair into the text encoding model in the song representation output model to obtain the first text representation, and to input the song description text in each of the other sample pairs into the text encoding model to obtain the second text representation;

[0110] The text representation distance determination submodule is used to determine the representation distance between the first text representation and each of the second text representations;

[0111] The negative sample filtering submodule is used to select the top-ranked representation distance according to the size of multiple representation distances, and use the song description text of the other sample pairs corresponding to the selected representation distance as negative samples of the song audio in the current sample pair.

[0112] In one embodiment, the representation distance determination module 603 includes:

[0113] The first text representation acquisition submodule is used to input the song description text in the current sample pair into the text encoding model in the song representation output model to obtain the first text representation.

[0114] The audio representation acquisition submodule is used to input the song audio in the current sample pair into the audio coding model in the song representation output model to obtain the audio representation;

[0115] The first representation distance acquisition submodule is used to perform dimensional transformation on the first text representation and the audio representation respectively, and to take the representation distance between the first text representation after dimensional transformation and the audio representation after dimensional transformation as the first representation distance.

[0116] In one embodiment, the representation distance determination module 603 includes:

[0117] The audio representation determination submodule is used to input the song audio in the current sample pair into the audio coding model in the song representation output model to obtain the audio representation;

[0118] The second text representation acquisition submodule is used to input the song description text in each of the other sample pairs into the text encoding model in the song representation output model to obtain the second text representation of each negative sample.

[0119] The second representation distance acquisition submodule is used to perform dimensional transformation on the second text representation and the audio representation of each negative sample, and to take the representation distance between the dimensionally transformed second text representation and the dimensionally transformed audio representation as the second representation distance.

[0120] In one embodiment, the model training module 604 includes:

[0121] The loss calculation submodule is used to calculate the target loss value by combining the ratio of the first representation distance to the second representation distance with a weighting parameter; the weighting parameter is determined based on the average distance between the first text representation of the positive sample and the second text representation of multiple negative samples.

[0122] The training submodule is used to perform comparative learning training based on the target loss value until the loss function converges, thereby obtaining the trained song representation output model.

[0123] In one embodiment, the representation distance determination module 603 includes:

[0124] The negative sample selection submodule is used to randomly select a portion of negative samples from the multiple negative samples during each iteration of loss, and to determine the second representation distance between the song audio in the current sample pair and the randomly selected negative samples each time.

[0125] The model training module 604 includes:

[0126] The iterative submodule is used to perform comparative learning training on the song representation output model based on the first representation distance and the second representation distance determined each time during each iteration loss process.

[0127] Each module in the training device for the aforementioned song representation output model can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the operations corresponding to each module.

[0128] Based on the same inventive concept, this application also provides a song processing apparatus for implementing the song processing method described above. The solution provided by this apparatus is similar to the implementation described in the above method; therefore, the specific limitations in one or more song processing apparatus embodiments provided below can be found in the limitations of the song processing method described above, and will not be repeated here.

[0129] In one embodiment, such as Figure 7 As shown, a song processing device is provided, comprising:

[0130] The unprocessed song acquisition module 701 is used to acquire the song audio and song description text of the unprocessed song;

[0131] The song representation acquisition module 702 is used to input the song audio and the song description text into the trained song representation output model to obtain the song audio representation and song description text representation of the song to be processed; the song audio representation and song description text representation of the song to be processed are close to each other in the representation space;

[0132] The trained song representation output model is obtained by training the song representation output model according to any one of the above methods.

[0133] Each module in the aforementioned song processing device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the corresponding operations of each module.

[0134] In one embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 8 As shown, the computer device includes a processor, memory, input / output interface, communication interface, display unit, and input device. The processor, memory, and input / output interface are connected via a system bus, and the communication interface, display unit, and input device are also connected to the system bus via the input / output interface. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The input / output interface is used for exchanging information between the processor and external devices. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, NFC (Near Field Communication), or other technologies. When the computer program is executed by the processor, it implements a training method for a song representation output model.

[0135] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 9 As shown, this computer device includes a processor, memory, input / output interfaces (I / O), and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The database stores song processing data. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communicating with external terminals via a network connection. When the computer program is executed by the processor, it implements a song processing method.

[0136] Those skilled in the art will understand that Figure 8 and Figure 9 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0137] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to perform the following steps:

[0138] Acquire song training data for multiple sample pairs; each sample pair includes the song audio and song description text of the same song;

[0139] For the current sample pair, based on the text differences between the song description text in the current sample pair and the song description text in each of the other sample pairs, select multiple song description texts with relatively large text differences from each of the other sample pairs as negative samples of the song audio in the current sample pair; wherein the current sample pair is any sample pair among the multiple sample pairs, and the other sample pairs are sample pairs other than the current sample pair among the multiple sample pairs.

[0140] Determine a first representation distance between the song audio and the song description text in the current sample pair, with the song description text in the current sample pair serving as a positive sample of the song audio in the current sample pair, and determine a second representation distance between the song audio in the current sample pair and multiple negative samples;

[0141] Based on the first representation distance and the second representation distance, the song representation output model is trained by comparative learning until a trained song representation output model is obtained; the trained song representation output model is used to output the song audio representation and the song description text representation.

[0142] In one embodiment, the processor also performs the steps described in the other embodiments when executing the computer program.

[0143] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to perform the following steps:

[0144] Obtain the audio and description text of the song to be processed;

[0145] The song audio and the song description text are input into the trained song representation output model to obtain the song audio representation and song description text representation of the song to be processed; the song audio representation and song description text representation of the song to be processed are close to each other in the representation space;

[0146] The trained song representation output model is obtained by training the song representation output model according to any one of the above methods.

[0147] In one embodiment, the processor also performs the steps described in the other embodiments when executing the computer program.

[0148] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program performing the following steps when executed by a processor:

[0149] Acquire song training data for multiple sample pairs; each sample pair includes the song audio and song description text of the same song;

[0150] For the current sample pair, based on the text differences between the song description text in the current sample pair and the song description text in each of the other sample pairs, select multiple song description texts with relatively large text differences from each of the other sample pairs as negative samples of the song audio in the current sample pair; wherein the current sample pair is any sample pair among the multiple sample pairs, and the other sample pairs are sample pairs other than the current sample pair among the multiple sample pairs.

[0151] Determine a first representation distance between the song audio and the song description text in the current sample pair, with the song description text in the current sample pair serving as a positive sample of the song audio in the current sample pair, and determine a second representation distance between the song audio in the current sample pair and multiple negative samples;

[0152] Based on the first representation distance and the second representation distance, the song representation output model is trained by comparative learning until a trained song representation output model is obtained; the trained song representation output model is used to output the song audio representation and the song description text representation.

[0153] In one embodiment, the computer program, when executed by a processor, also implements the steps described in the other embodiments above.

[0154] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program performing the following steps when executed by a processor:

[0155] Obtain the audio and description text of the song to be processed;

[0156] The song audio and the song description text are input into the trained song representation output model to obtain the song audio representation and song description text representation of the song to be processed; the song audio representation and song description text representation of the song to be processed are close to each other in the representation space;

[0157] The trained song representation output model is obtained by training the song representation output model according to any one of the above methods.

[0158] In one embodiment, the computer program, when executed by a processor, also implements the steps described in the other embodiments above.

[0159] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, performs the following steps:

[0160] Acquire song training data for multiple sample pairs; each sample pair includes the song audio and song description text of the same song;

[0161] For the current sample pair, based on the text differences between the song description text in the current sample pair and the song description text in each of the other sample pairs, select multiple song description texts with relatively large text differences from each of the other sample pairs as negative samples of the song audio in the current sample pair; wherein the current sample pair is any sample pair among the multiple sample pairs, and the other sample pairs are sample pairs other than the current sample pair among the multiple sample pairs.

[0162] Determine a first representation distance between the song audio and the song description text in the current sample pair, with the song description text in the current sample pair serving as a positive sample of the song audio in the current sample pair, and determine a second representation distance between the song audio in the current sample pair and multiple negative samples;

[0163] Based on the first representation distance and the second representation distance, the song representation output model is trained by comparative learning until a trained song representation output model is obtained; the trained song representation output model is used to output the song audio representation and the song description text representation.

[0164] In one embodiment, the computer program, when executed by a processor, also implements the steps described in the other embodiments above.

[0165] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, performs the following steps:

[0166] Obtain the audio and description text of the song to be processed;

[0167] The song audio and the song description text are input into the trained song representation output model to obtain the song audio representation and song description text representation of the song to be processed; the song audio representation and song description text representation of the song to be processed are close to each other in the representation space;

[0168] The trained song representation output model is obtained by training the song representation output model according to any one of the above methods.

[0169] In one embodiment, the computer program, when executed by a processor, also implements the steps described in the other embodiments above.

[0170] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0171] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments described above. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0172] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0173] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A training method for a song representation output model, characterized in that, The method includes: Acquire song training data for multiple sample pairs; each sample pair includes the song audio and song description text of the same song; For the current sample pair, based on the text differences between the song description text in the current sample pair and the song description text in each of the other sample pairs, multiple song description texts are selected as negative samples of the song audio in the current sample pair. This includes: determining the representation distance between a first text representation and each second text representation, where the first text representation is used to represent the song description text in the current sample pair, and each second text representation is used to represent the song description text in each of the other sample pairs; selecting the representation distance with the highest ranking according to the size of the multiple representation distances, and using the song description text of the other sample pair corresponding to the selected representation distance as a negative sample of the song audio in the current sample pair; wherein the current sample pair is any one of the multiple sample pairs, and the other sample pairs are sample pairs other than the current sample pair among the multiple sample pairs. Determine a first representation distance between the song audio and the song description text in the current sample pair, with the song description text in the current sample pair serving as a positive sample of the song audio in the current sample pair, and determine a second representation distance between the song audio in the current sample pair and multiple negative samples; Based on the first representation distance and the second representation distance, the song representation output model is trained by comparative learning until a trained song representation output model is obtained; the trained song representation output model is used to output the song audio representation and the song description text representation.

2. The method according to claim 1, characterized in that, The method further includes: The song description text in the current sample pair is input into the text encoding model in the song representation output model to obtain the first text representation. The song description text in each of the other sample pairs is input into the text encoding model to obtain the second text representation.

3. The method according to claim 1, characterized in that, Determining the first representation distance between the song audio and the song description text in the current sample pair includes: The song description text in the current sample pair is input into the text encoding model in the song representation output model to obtain the first text representation; The song audio from the current sample pair is input into the audio encoding model in the song representation output model to obtain the audio representation; The first text representation and the audio representation are dimensionally transformed respectively, and the representation distance between the first text representation after dimensional transformation and the audio representation after dimensional transformation is taken as the first representation distance.

4. The method according to claim 1, characterized in that, Determining the second representation distance between the song audio in the current sample pair and the plurality of negative samples includes: The song audio from the current sample pair is input into the audio encoding model in the song representation output model to obtain the audio representation; The song description text in each of the other sample pairs is input into the text encoding model in the song representation output model to obtain the second text representation of each negative sample; The second text representation and the audio representation of each negative sample are subjected to dimensional transformation respectively. The representation distance between the second text representation after dimensional transformation and the audio representation after dimensional transformation is taken as the second representation distance.

5. The method according to claim 1, characterized in that, The step of performing comparative learning training on the song representation output model based on the first representation distance and the second representation distance until a trained song representation output model is obtained includes: The target loss value is calculated by combining the ratio of the first representation distance to the second representation distance with a weighting parameter; the weighting parameter is determined based on the average distance between the first text representation of the positive sample and the second text representation of multiple negative samples. Based on the target loss value, comparative learning is performed until the loss function converges, resulting in the trained song representation output model.

6. The method according to claim 1, characterized in that, Determining the second representation distance between the song audio in the current sample pair and the plurality of negative samples includes: In each iteration of the loss process, a portion of negative samples are randomly selected from the multiple negative samples, and a second representation distance is determined between the song audio in the current sample pair and the randomly selected negative samples each time. The step of performing comparative learning training on the song representation output model based on the first representation distance and the second representation distance includes: During each iteration of the loss process, the song representation output model is trained by comparison based on the first representation distance and the second representation distance determined each time.

7. A song processing method, characterized in that, The method includes: Obtain the audio and description text of the song to be processed; The song audio and the song description text are input into the trained song representation output model to obtain the song audio representation and song description text representation of the song to be processed; the song audio representation and song description text representation of the song to be processed are close to each other in the representation space; The trained song representation output model is obtained by training the song representation output model according to any one of claims 1-6.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 7.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

Images