Music review generation method, device and computer readable storage medium

Abstract

The embodiment of the present application discloses a method, device and computer-readable storage medium for generating music reviews, wherein the method comprises: obtaining a fusion feature vector of a target music, the fusion feature vector being composed of a concatenation of a lyrics feature vector of the target music and an audio feature vector of the target music; obtaining a preset vocabulary, the preset vocabulary including a plurality of preset vocabulary characters; determining the probability of each vocabulary character in the preset vocabulary generating each comment character in the comment text of the target music according to the fusion feature vector of the target music; obtaining the comment text of the target music based on the probability of each vocabulary character generating each comment character in the comment text. The present application can improve the quality of generating music reviews.

Description

Technical Field

[0001] The present application relates to the field of audio processing technology, and in particular to a music review generation method, device and computer-readable storage medium. Background Art

[0002] With the rapid development of Internet technology and audio processing technology, users can access a vast amount of music resources through various channels. Generally speaking, the music resources that meet user needs are a small number of popular music (or popular music), which often receive more publicity and exposure, while most long-tail music (or niche music) cannot be effectively used because it is difficult to identify the quality of the content. Therefore, how to fully tap the economic benefits of long-tail music is an urgent problem to be solved.

[0003] The commonly used technical means at present is to generate high-quality music reviews related to long-tail music and push the reviews to users, so as to stimulate users' interest in listening to long-tail music. Generating music reviews can be achieved by retrieving and extracting historical reviews. However, due to the unpopularity of long-tail music, this method cannot match the historical reviews that meet the requirements, and when reconstructing new reviews based on historical reviews, the quality of the generated reviews is poor. Summary of the invention

[0004] The embodiments of the present application provide a music review generation method, device and computer-readable storage medium, which can improve the quality of generated music reviews.

[0005] In a first aspect, an embodiment of the present application provides a method for generating music reviews, the method comprising:

[0006] Obtain a fused feature vector of the target music, where the fused feature vector is formed by concatenating the lyrics feature vector of the target music and the audio feature vector of the target music;

[0007] Obtaining a preset vocabulary, the preset vocabulary including a plurality of preset vocabulary characters;

[0008] Determine the probability of each character in the preset vocabulary generating each comment character in the comment text of the target music according to the fusion feature vector of the target music;

[0009] The probability of each comment character in the comment text is generated based on each vocabulary character to obtain the comment text of the target music.

[0010] Based on the method described in the first aspect, the present application can integrate the two modal information of the target music (audio information and lyrics information) to process the target music to generate a commentary text (or music review text) for the target music. The generated music review text has a stronger correlation with the target music, thereby improving the quality of the generated music review text.

[0011] In a possible implementation, the above-mentioned determination of the probability of each character in the preset vocabulary generating each comment character in the comment text of the target music based on the fusion feature vector of the target music includes:

[0012] According to the fused feature vector of the target music, determine the initial feature vector corresponding to each comment character in the comment text of the target music;

[0013] Input the initial feature vector corresponding to each comment character into the self-attention layer of the trained language model to obtain the target feature vector corresponding to each comment character;

[0014] Based on the mapping rules of the preset vocabulary, the target feature vectors corresponding to the various comment characters are mapped to obtain the probability of each vocabulary character in the preset vocabulary generating each comment character.

[0015] Based on this approach, the present application may use the self-attention layer of the language model to process the fused feature vector to obtain the probability of each vocabulary character in the preset vocabulary generating each comment character.

[0016] In a possible implementation manner, the comment text includes N comment characters; the above-mentioned determination of the initial feature vectors corresponding to the respective comment characters in the comment text of the target music based on the fused feature vector of the target music includes:

[0017] When N is equal to 1, the fusion feature vector of the target music is determined as the initial feature vector corresponding to the first comment character in the comment text;

[0018] When N is greater than 1, the character vector of the N-1th comment character is used to update the fused feature vector; the updated fused feature vector is determined as the initial feature vector corresponding to the Nth comment character in the comment text.

[0019] Based on this method, the present application can update the fused feature vector corresponding to each comment character, and the character vector of the previous comment character is used during the update. This method improves the semantic relevance between the various comment characters.

[0020] In a possible implementation, the method further includes: determining sampling parameters corresponding to each comment character in the comment text of the target music;

[0021] The above generates the probability of each comment character in the comment text based on each word list character, and obtains the comment text of the target music, including:

[0022] For each comment character, according to the sampling parameters corresponding to the comment character and the probability of each word table character corresponding to the comment character, a word table subset corresponding to the comment character is selected from the preset word table;

[0023] For each comment character, select a word list character from the word list subset corresponding to the comment character as the comment character;

[0024] Arrange the comment characters in order to obtain a text as the comment text of the target music.

[0025] Based on this approach, the present application can determine each comment character in the comment text according to the sampling parameters corresponding to each comment character and the probability corresponding to each vocabulary character.

[0026] In one possible implementation, the sampling parameters include a cumulative probability threshold P and a probability quantity threshold K;

[0027] For each comment character, according to the sampling parameters corresponding to the comment character and the probability of each word table character corresponding to the comment character, a word table subset corresponding to the comment character is selected from the preset word table, including:

[0028] For each comment character, select the first K characters with the highest probability in the preset vocabulary to form a first subset of the vocabulary to be selected;

[0029] Determine multiple second subsets of vocabulary to be selected from the first subset of vocabulary to be selected; wherein the sum of the probabilities corresponding to the vocabulary characters in the second subset of vocabulary to be selected is greater than or equal to P, or the vocabulary characters in the second subset of vocabulary to be selected are the same as the vocabulary characters in the subset of vocabulary to be selected and the sum of the probabilities corresponding to the vocabulary characters in the subset of vocabulary to be selected is less than P;

[0030] The second candidate vocabulary subset containing the least number of vocabulary characters is used as the vocabulary subset corresponding to the comment character.

[0031] Based on this method, the present application can use the TOP-K sampling method and the TOP-P sampling method in series to obtain a vocabulary subset corresponding to each comment character. Compared with a single TOP-K sampling method or TOP-P sampling, this method can make the obtained vocabulary subset more accurate and of higher quality.

[0032] In one possible implementation, the comment text includes first-category comment characters and second-category comment characters, the first-category comment characters are located before the second-category comment characters in the comment text, and the value of the sampling parameter corresponding to the first-category comment characters is greater than the value of the sampling parameter corresponding to the second-category comment characters.

[0033] Based on this method, the present application can dynamically adjust the sampling parameters corresponding to the TOP-K sampling method and the TOP-P sampling method to obtain more fluent and diverse review texts.

[0034] In a possible implementation, the above-mentioned generation of the probability of each comment character in the comment text based on each word list character to obtain the comment text of the target music includes:

[0035] Generate the probability of the first comment character in the comment text based on each vocabulary character, and obtain multiple first comment characters;

[0036] Generate optional comment text corresponding to each first comment character respectively;

[0037] An optional comment text is selected from multiple optional comment texts as the comment text of the target music.

[0038] Based on this approach, when determining the first character in a comment text, the present application reserves multiple vocabulary characters as the first character, and generates optional comment texts based on each first character, which can increase the diversity of the generated comment texts.

[0039] In a possible implementation manner, the comment text includes N comment characters; the above-mentioned generating the optional comment text corresponding to each first comment character includes:

[0040] For each first comment character, generate the optional comment text corresponding to each first comment character according to the following steps;

[0041] Set N to 1;

[0042] Use the character vector of the Nth comment character to update the fused feature vector, input the updated fused feature vector into the self-attention layer of the trained language model, and obtain the target feature vector corresponding to the N+1th comment character;

[0043] Based on the mapping rules of the preset vocabulary, the target feature vector corresponding to the N+1th comment character is mapped to obtain the probability of each vocabulary character in the preset vocabulary generating the N+1th comment character;

[0044] Determine the N+1th comment character based on the probability of generating the N+1th comment character in the comment text based on each vocabulary character;

[0045] If the condition for stopping the generation of the optional comment text is not met, performing an operation of adding 1 to N, and returning to the step of updating the fused feature vector using the character vector of the Nth comment character;

[0046] If the condition for stopping the generation of the optional comment text is met, the first comment character and each of the determined N+1th comment characters are sorted to obtain the optional comment text corresponding to the first comment character.

[0047] Based on this method, when generating optional comment text for each first comment character, the present application can adopt an autoregressive generation method to increase the semantic connection of each comment character in the optional comment text.

[0048] In one possible implementation, the method further includes:

[0049] Obtain lyrics text and audio data of target music;

[0050] Call the embedding layer of the trained language model to vector encode the lyrics text of the target music to obtain the lyrics feature vector of the target music;

[0051] The trained audio coding model is called to perform vector encoding on the audio data of the target music to obtain the audio feature vector of the target music.

[0052] Based on this method, the present application can respectively call the trained language model and the trained audio encoding model to obtain the vector representations corresponding to the lyrics and audio, so that the self-attention layer in the trained language model can subsequently process the fused feature vector obtained by concatenating these two vectors.

[0053] In one possible implementation, the method further includes:

[0054] splicing the training lyrics and training comments of the training music to obtain the training text;

[0055] Calling the embedding layer of the language model to perform vector encoding on the training text to obtain a first text feature vector;

[0056] Calling the self-attention layer of the language model to determine the probability of each character in the preset vocabulary generating each comment character in the training comment based on the first text feature vector and a mapping rule of the preset vocabulary;

[0057] Determining a first cumulative likelihood function value based on the probability of each character in the training comment in the training text being generated by each character in the vocabulary;

[0058] The parameters in the language model are trained and updated based on the first cumulative likelihood function value to obtain a trained language model.

[0059] Based on this method, when training a language model, the present application can reflect the probability of generating training comments based on training lyrics through the first cumulative likelihood probability value. The first cumulative likelihood function focuses on the part with training comments as the model output, so that the effect of generating comments after language model training is better. In addition, compared with the method of training models in the prior art, the language model is a model that has learned a large number of common features through large-scale corpus in advance. Therefore, the present application can use a small number of training samples to train the language model, and the training method is more convenient and efficient.

[0060] In one possible implementation, the method further includes:

[0061] Obtain training audio data for training music;

[0062] Call the audio coding model to perform vector encoding on the training audio data to obtain the training audio feature vector;

[0063] Calling the embedding layer of the trained language model to perform vector encoding on the training text to obtain a second text feature vector;

[0064] Concatenate the second text feature vector with the training audio feature vector to obtain a training fusion feature vector;

[0065] Calling the self-attention layer of the trained language model, based on the training fusion feature vector and the mapping rule of the preset vocabulary, determines the probability of each vocabulary character in the preset vocabulary generating each comment character in the training comment;

[0066] Determining a second cumulative likelihood function value based on the probability of each character in the training comment in the training text being generated by each character in the vocabulary;

[0067] The parameters in the audio coding model are updated based on the second cumulative likelihood function value to obtain a trained audio coding model.

[0068] Based on this, the present application can integrate the audio information of music for training when training and generating comment text. And when updating parameters, all parameters of the trained language model are frozen, so that the gradient will only act on the parameters of the updated audio coding model, ensuring that the parameters in the trained language model are not polluted or damaged.

[0069] In a second aspect, an embodiment of the present application provides a music review generating device, the device comprising:

[0070] In a third aspect, an embodiment of the present application provides a terminal device, comprising: a memory, a processor; the memory is used to store a computer program; the processor is used to call the computer program from the memory, so that the terminal device executes any one of the methods in the above-mentioned first aspect.

[0071] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, in which computer-readable instructions are stored. When the computer-readable instructions are executed on the terminal device of the second aspect, the terminal device executes any one of the methods in the first aspect.

[0072] In a fifth aspect, an embodiment of the present application provides a computer program or a computer program product, including codes or instructions. When the codes or instructions are executed on a computer, the computer executes any one of the methods in the first aspect above.

[0073] The beneficial effects of each possible implementation method in the second to fifth aspects can be found in the corresponding description in the first aspect and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0074] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following briefly introduces the drawings required for describing the embodiments.

[0075] Figure 1 It is a schematic diagram of a framework of a GPT model provided in an embodiment of the present application;

[0076] Figure 2a It is a flowchart of a model training provided in an embodiment of the present application;

[0077] Figure 2b It is a flow chart of a model application provided in an embodiment of the present application;

[0078] Figure 3 It is a flowchart of a music review generation method provided in an embodiment of the present application;

[0079] Figure 4 It is a schematic diagram of a model processing flow for generating music reviews provided in an embodiment of the present application;

[0080] Figure 5 It is a schematic diagram of a framework of an audio encoder based on a convolutional neural network provided in an embodiment of the present application;

[0081] Figure 6 is a schematic diagram of a process for determining an Nth vocabulary subset provided in an embodiment of the present application;

[0082] Figure 7 It is a schematic diagram of a process of dynamically adjusting sampling parameters provided in an embodiment of the present application;

[0083] Figure 8 It is a flow chart of a cluster sampling method provided in an embodiment of the present application;

[0084] Fig. 9 It is a flowchart of another music review generation method provided in an embodiment of the present application;

[0085] Fig.10 A flowchart of a training phase of a trained language model provided in an embodiment of the present application;

[0086] Fig.11 It is a flowchart of a training phase of a trained audio coding model provided in an embodiment of the present application;

[0087] Fig.12 It is a structural schematic diagram of a music review generating device provided in an embodiment of the present application;

[0088] Fig.13 It is a structural diagram of a terminal device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0089] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0090] The terms "first" and "second" and the like in the specification, claims and drawings of the present application are used to distinguish different objects, rather than to describe a specific order. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device comprising a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products or devices.

[0091] In order to better understand the solution of this application, the professional terms involved in this application are introduced below:

[0092] 1. Pre-trained language model (PLM)

[0093] The pre-trained language model is obtained by self-supervised pre-training on a large amount of unlabeled corpus. In the process of self-supervised pre-training, the pre-trained language model can learn a large number of general language representations, which can be better migrated to assist downstream tasks of various natural language processing, such as text classification, text generation, text translation, etc. When the pre-trained language model is applied to downstream tasks, it is only necessary to use a small data set to train the model again, without collecting a large number of data sets and training the model parameters from scratch, which improves the processing efficiency of downstream tasks. The model architecture of the pre-trained language model is mostly based on models such as machine translation models Transformers, including bidirectional encoding models (bidirectional encoder representation from transformers, BERT) based on Transformers models, generative pre-training (generative pre-training, GPT) models, etc. The language model used in the embodiment of the present application is a pre-trained language model, and the language model completed by training is a language model after the pre-trained language model is trained again using a small data set. The example can be a GPT model. The workflow of the GPT model in the task of generating text is introduced below.

[0094] 2. GPT Model

[0095] See also Figure 1 , is a schematic diagram of a GPT model provided in an embodiment of the present application. Figure 1 As shown, the GPT model includes an input layer, an intermediate layer and an output layer. The input layer is a vector embedding layer of the input characters, which can encode the input characters into vector representations (including character embedding vectors and character position vectors); the intermediate layer adopts the decoder structure of the Transformers model, which can be used to determine the probability distribution of generating text characters based on each vocabulary character in the preset vocabulary according to the vector representation of the input characters; the output layer decodes the probability distribution and determines a vocabulary character from the preset vocabulary as the generated text character.

[0096] It should be noted that the input layer here can also be called an embedding layer, and the middle layer can also be called a self-attention layer. The functions performed by the above output layer may not be integrated in the GPT model. After processing the input characters each time, the GPT model outputs the probability distribution of each vocabulary character to generate a text character, and inputs this probability distribution into a decoding module that decodes the probability distribution; this decoding module pre-sets a decoding strategy, and can determine a vocabulary character from the preset vocabulary as the generated text character after decoding the probability distribution based on the decoding strategy.

[0097] The GPT model uses an autoregressive generation method when generating predicted text. For example, Figure 1 As shown in the figure, the GPT model first generates the second character T1 based on the vector representation E1 of the first character, and returns T1 to the input layer and encodes it as a vector E2. The GPT model generates the third character T2 based on E1 and E2, and the generated T2 is also returned to the input layer and encoded as a vector E3. The GPT model generates the fourth character T3 based on E1, E2, and E3, and so on. The GPT model will generate a character sequence T1 to TN according to this process, and when generating each character in T2 to TN, the semantic understanding of the input character (the character corresponding to E1) and the aforementioned generated character is integrated into the intermediate layer, so that each character in the generated character sequence has semantic relevance.

[0098] 3. Decoding Strategy

[0099] In the text generation task, the decoding strategy refers to selecting a character from multiple optional characters as the generated character according to the probability distribution of the characters obtained by the language model. Depending on the selection rules, the decoding strategies include TOP-K sampling, TOP-P sampling, and beam sampling BeamSearch.

[0100] TOP-K sampling refers to selecting the top K characters with the highest probability from multiple optional characters according to the probability distribution corresponding to the characters, then renormalizing the probabilities of the selected K characters, and selecting an optional character from the K characters as the generated character based on the normalized probability.

[0101] TOP-P sampling refers to determining the minimum character set (containing the least number of optional characters) whose cumulative probability exceeds a threshold P from the probability distribution corresponding to the characters, then renormalizing the character probabilities in this minimum character set, and selecting an optional character as the generated character based on the normalized probability distribution.

[0102] When generating the first character in a character sequence, BeamSearch retains the first S optional characters with the highest probability; then, these S first characters are sent back to the input layer to generate the character sequence corresponding to each first character; finally, a character sequence is selected from the generated S character sequences as the final generated character sequence. BeamSearch retains more sampling selectivity and increases the probability of generating the best combination of character sequences.

[0103] 4. Multimodal Information of Music

[0104] The source or form of each information is called a modality. When the information source contains information in multiple forms, it can be called multimodal information. For music, in addition to audio information (such as ripple data), there is also text information related to it (such as comment data), picture information (album cover) or video information (such as MV data), etc. When at least two of the above information are used to extract features from music, multimodal music features can be obtained. The language model trained in the embodiment of the present application is processed using multimodal music features to obtain higher quality output results.

[0105] The present application embodiment provides a method for generating music reviews, which uses the above-mentioned trained language model to fuse multiple modal information of music to generate music reviews. The method flow specifically includes: Figure 2a The model training phase and Figure 2b The model application stage shown. The music review generation method can be implemented by a music review generation device, which can be a terminal device or a server.

[0106] Exemplarily, the terminal device may be a device with data processing functions, including but not limited to mobile phones, tablet computers, laptop computers, PDAs, mobile internet devices (MID), vehicles, roadside devices, aircraft, wearable devices, etc., and this application does not limit this. The server may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms.

[0107] like Figure 2a As shown, in the training phase, the music review generating device can obtain training music (including audio data, lyrics text and comment text of the training music) to train the language model and the audio coding model. Specifically, when training the language model, the music review generating device first vectorizes the lyrics text and comment text of the training music, and then performs self-attention learning on the vectorized result to obtain the cumulative likelihood function value of the pre-trained language model, which can be used to fine-tune (or update, adjust, etc.) the model parameters of the language model to obtain the trained language model, and the trained language model can be used in the application phase of the model. Then, when training the audio coding model, the music review generating device first vectorizes the audio data of the training music through the audio coding model, and vectorizes the lyrics text and comment text of the training music through the trained language model, and then splices the vectorized representation results of the two and then performs self-attention learning to obtain the cumulative likelihood function value. The cumulative likelihood function value here is only used to update the model parameters of the audio coding model to obtain the trained audio coding model. The model parameters in the trained language model are fixed and frozen during this update process.

[0108] like Figure 2b As shown, in the application stage, the music review generating device can first obtain the target music (including the audio data and lyrics text of the target music), and then input the audio data and lyrics text of the target music into the trained audio coding model and the trained language model respectively for vectorized representation, and then concatenate the vectorized representation results of the two and input them into the trained language model for self-attention learning to generate the review text of the target music.

[0109] It should be noted that in the above-mentioned training stage and application stage, self-attention learning refers to first obtaining the probability of each vocabulary character in the preset vocabulary generating each character in the training comments and the comment text of the target music, so that the cumulative likelihood function value or the character in the comment text of the target music can be determined based on the probability of each vocabulary character generating each character in the comment text of the training comments and the target music.

[0110] Based on the above process, the present application can integrate multiple modal information of the target music (for example, audio information and lyrics information) to model the target music to generate high-quality music review text, and the generated music review text has improved relevance to the target music. It should be noted that when the present application is applied to specific products and technologies, the relevant data of the music in the embodiments of the present application (such as the above-mentioned audio information, commentary information and lyrics information) need to obtain the permission or consent of the user / copyright holder before it can be obtained, and the collection, use and processing of these data need to comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0111] The following will be combined Figures 3 to 13 The illustrated embodiment provides a detailed description of the above process.

[0112] See also Figure 3 , is a flow chart of a music review generation method provided by an embodiment of the present application, the method mainly involves the application stage of the trained language model and the trained audio coding model (that is, how to use these two models to generate music review text), and the trained language model is obtained by sequentially connecting the embedding layer and the self-attention layer. The method can be applied to the above-mentioned music review generation device, including steps S301 to S303. Among them:

[0113] S301, obtaining a fusion feature vector of target music, where the fusion feature vector is formed by concatenating a lyrics feature vector of the target music and an audio feature vector of the target music.

[0114] In a specific implementation, the music review generating device can first obtain the lyrics feature vector and audio feature vector of the target music based on the following method, then determine the splicing method and splice the lyrics feature vector and audio feature vector of the target music based on the determined splicing method to obtain the fused feature vector of the target music.

[0115] In a possible implementation, the above method of obtaining the target music's lyrics feature vector and audio feature vector specifically includes: obtaining the target music's lyrics text and audio data; calling the trained language model's embedding layer to vector encode the target music's lyrics text to obtain the lyrics feature vector; calling the trained audio encoding model to vector encode the target music's audio data to obtain the audio feature vector. The lyrics feature vector and the audio feature vector are respectively the embedding vector representations of the lyrics text and the audio data.

[0116] Specifically, when the embedding layer of the trained language model performs vector embedding on the lyrics text, it can search for an embedding vector representation that matches each character in the lyrics text in a preset embedding dictionary matrix. Optionally, each row of the preset embedding dictionary matrix is ​​an embedding vector representation of a character.

[0117] Optionally, in order to make the trained language model understand the order of characters in the lyrics text, a position vector can also be used to represent the position of each character in the lyrics text. The vector dimension of the position vector is the same as the vector dimension of the embedding vector, and can be found in a preset position vector matrix. The size of the preset position vector matrix is ​​equal to the maximum number of characters that can be input into the trained language model. Based on this optional method, the lyrics feature vector can be obtained by adding the embedding vector corresponding to each character in the lyrics text to the position vector.

[0118] For example, Figure 4 It is a schematic diagram of a model processing flow for generating music reviews provided in an embodiment of the present application. Figure 4 The GPT-2 language model shown corresponds to Figure 2a and Figure 2b The trained language model (including embedding layer and self-attention layer) shown in Figure 4 The audio encoder shown corresponds to Figure 2a and Figure 2b The trained audio coding model shown in FIG. Lyrics text X = {x1, x2, x3, ... x l}, x l is the lth character in the lyrics text, then the lyrics text X can be embedded in the GPT-2 language model to obtain the corresponding lyrics feature vector L = {l1,l2,l3,...,l l}, l l is the vector representation corresponding to the lth character in the lyrics text, l l It can be represented by the embedding vector l corresponding to the lth character l,1 The position vector l corresponding to the lth character l,2 The audio data Y = {y1, y2, y3, ... y a},ya is the ath segment of audio data (for example, a complete audio data can be divided into a frames, then y a is the ath audio frame), then the audio feature vector obtained by the audio data Y through the audio encoder can be represented by A = {a1, a2, a3, ..., a a} means that a a It is the vector representation corresponding to the a-th segment of data in the audio data.

[0119] It should be noted that when the dimension of the audio feature vector is different from that of the lyrics feature vector, a linear mapping method is required to convert the dimension of the audio feature vector into the dimension of the lyrics feature vector so that the pre-trained language model can fuse the two feature vectors.

[0120] For example, the trained audio coding model can be as follows: Figure 5 The audio encoder based on convolutional neural networks (CNN) shown in FIG. When processing audio data, the audio encoder can directly process the input audio waveform data, and the CNN network automatically extracts abstract features similar to the audio mel spectrum features without the need for manual design and extraction of features. Figure 5 In the embodiment, the audio encoder is divided into a front-end processing part and a back-end processing part. The front-end processing part can map the input audio data (audio waveform data) into the latent space to obtain the primary abstract features of the audio data, and the back-end processing part further processes the primary abstract features to obtain the audio feature vector in the embodiment of the present application.

[0121] As an example, Figure 5The front-end processing part in adopts seven layers of processing modules for iterative processing. The first layer of processing modules consists of 3x1 CNN filters and batch normalization layers. The second to seventh layers of processing modules are all composed of 3x1 CNN filters, batch normalization layers and maximum pooling layers. Exemplarily, the number of CNN filters in the seven layers is 64, 64, 64, 128, 128, 128, 256 respectively. The back-end processing part consists of three CNN layers (exemplarily, each layer has 512 CNN filters and batch normalization layers), two pooling layers and one dense layer (including batch normalization layers). Among them, residual connections are used between the three CNN layers to realize the nonlinear mapping between input and output. The CNN filters used here can extract all possible features in the audio data. In addition, the pooling layer used in the back-end processing part is a global pooling after averaging and maximum statistics of the input. The global pooling strategy allows variable-length input of the network, so the back-end processing part can receive variable-length input from the front-end processing part. Finally, the output after maximum pooling is connected to the output layer by a dense layer (also known as a fully connected layer) to obtain an audio feature vector. In actual application, the model architecture of the audio encoder can also adopt other designs, such as including more or fewer CNN filters, not using residual connections, using the Transformer-based WavLM speech pre-training framework, etc. This application does not limit the model architecture of the audio encoder.

[0122] In a possible implementation, the above-mentioned determination of the splicing method, and splicing the target music's lyrics feature vector and audio feature vector based on the determined splicing method to obtain the target music's fusion feature vector can adopt a splicing method in which the audio feature vector is in front and the lyrics feature vector is in the back, or the audio feature vector is in front and the lyrics feature vector is in the back. Compared with directly performing subsequent processing on the two feature vectors, this method of splicing the lyrics feature vector and the audio feature vector and then performing subsequent processing can effectively avoid the special situation that the audio feature vector and the lyrics feature vector are not in the same feature space.

[0123] For example, for the lyrics feature vector L = {l1,l2,l3,...,l l} and the audio feature vector A={a1,a2,a3,...,a a}, the fused feature vector obtained based on the first concatenation method can be {a1,a2,a3,...,a a ,l1,l2,l3,...,l l}, the fused feature vector obtained based on the second concatenation method is {l1,l2,l3,...,l l ,a1,a2,a3,...,a a}.

[0124] Optionally, a preset fusion feature vector template may be used to concatenate the lyrics feature vector and the audio feature vector.

[0125] For example, the fused feature vector template is {l1,l2,l3,...,l l ,b1,b2,b3,a1,a2,a3,...,a a}, where {b1, b2, b3} are auxiliary semantic connection vectors. The fused feature vector is obtained by splicing the auxiliary semantic connection vectors to achieve semantic completeness. For example, {b1, b2, b3} can be the feature vector corresponding to "the lyrics of the audio data are: ". In this case, the semantic content included in the fused feature vector is "the content of the audio data + the lyrics of the audio data are: + the lyrics content".

[0126] S302: Obtain a preset vocabulary, where the preset vocabulary includes a plurality of preset vocabulary characters.

[0127] The preset vocabulary is a character set including multiple optional vocabulary characters, and the type of vocabulary characters can be Chinese, numbers, letters, symbols, etc. In tasks such as generating text (such as generating comments in this application), any vocabulary character can be selected from the preset vocabulary as a character in the generated text (such as comment text).

[0128] Specifically, the comment text includes N comment characters, N is a positive integer, and the Nth character can be an end identifier (such as a period) or the last character reaching the preset total number of characters N. In the process of generating each comment character in the comment text, the probability of each word table character in the preset word table generating each character in the comment text can be determined based on the fusion feature vector obtained in step S301 and the preset word table obtained in step S302 (this step refers to the specific implementation of S303 below), and then the comment text is determined according to the probability of each word table character in the preset word table generating each character in the comment text (this step refers to the specific implementation of S304 below). In addition, when generating the comment text, the first comment character, the second comment character...the Nth comment character in the comment text are generated in sequence.

[0129] S303: Determine the probability of each character in the preset vocabulary generating each comment character in the comment text of the target music according to the fusion feature vector of the target music.

[0130] In one possible implementation, the specific implementation process of step S303 includes: determining the initial feature vector corresponding to each comment character in the comment text of the target music based on the fused feature vector of the target music; inputting the initial feature vector corresponding to each comment character into the self-attention layer of the trained language model to obtain the target feature vector corresponding to each comment character; based on the mapping rules of a preset vocabulary, mapping the target feature vector corresponding to each comment character to obtain the probability of each vocabulary character in the preset vocabulary generating each comment character.

[0131] Among them, the above-mentioned determination of the initial feature vectors corresponding to each comment character in the comment text of the target music based on the fused feature vector of the target music specifically includes: when N is equal to 1, determining the fused feature vector of the target music as the initial feature vector corresponding to the first comment character in the comment text; when N is greater than 1, using the character vector of the N-1th comment character to update the fused feature vector; and determining the updated fused feature vector as the initial feature vector corresponding to the Nth comment character in the comment text.

[0132] In other words, the initial feature vector corresponding to the first comment character in the above comment text is the fused feature vector obtained in step S301, and the fused feature vector corresponding to the remaining comment characters in the comment text except the first comment character is the updated vector of the fused feature vector. The character vector of the N-1th comment character here refers to the vector obtained by calling the embedding layer of the trained language model to perform vector encoding on the N-1th comment character, and the update may refer to splicing the character vector of the N-1th comment character to the fused feature vector to obtain the updated fused feature vector. It should be noted that, for the above-mentioned process of updating the fused feature vector, each time a new round of updating is performed on the fused feature vector, the fused feature vector to be updated may refer to the fused feature vector that has undergone the previous round of update.

[0133] For example, the initial feature vector corresponding to the second comment character is the vector obtained by concatenating the fused feature vector (i.e., the fused feature vector obtained by concatenating the lyrics feature vector and the audio feature vector of the target music, which is also the initial feature vector corresponding to the first comment character in the comment text) with the character vector of the first (i.e., the first) comment character; the initial feature vector corresponding to the third comment character is the vector obtained by concatenating the initial feature vector corresponding to the second character (i.e., the fused feature vector updated with the character vector of the first comment character) with the character vector of the second comment character, and so on. Based on this method, when the self-attention layer processes the initial feature vector of each comment character, it can simultaneously consider the semantic correlation between the fused feature vector and the character vectors corresponding to the first N-1 characters (when N is 1, only the semantics of the fused feature vector is considered), so that each character in the comment text generated by this step and subsequent steps has a close semantic connection.

[0134] Furthermore, after determining the initial feature vector corresponding to each comment character based on the above process, the initial feature vector corresponding to each comment character can be input into the self-attention layer of the trained language model to obtain the target feature vector corresponding to each comment character. The target feature vector here can represent the result of the self-attention layer's semantic understanding and self-attention scoring of each vector component in the initial feature vector. The self-attention scoring here is based on the principles of linguistics and the semantic association between contexts after semantic understanding.

[0135] For example, in a language system, a sentence often begins with a noun or a pronoun. When generating the first comment character in a comment text, the target feature vector corresponding to the first comment character can indicate that the component corresponding to the noun character in the initial feature vector has a higher self-attention score, which means that the noun character has a stronger semantic connection with the first character.

[0136] Furthermore, after determining the target feature vector corresponding to each comment character based on the above process, the target feature vector corresponding to each comment character can be mapped based on the mapping rules of the preset vocabulary to obtain the probability of each vocabulary character in the preset vocabulary generating each comment character. Here, the probability of each vocabulary character in the preset vocabulary generating each comment character can be called the vocabulary probability distribution. This method of mapping to probability can more intuitively determine each comment character in the comment text and improve the efficiency of determining the comment text.

[0137] For example, the mapping rule can be: vocabulary probability distribution = W*target feature vector + b, W is the weight matrix, b is the bias. When the target feature vector is a 768-dimensional vector representation and the preset vocabulary includes 20,000 characters, the formula can convert the target feature vector into a preset vocabulary of 20,000 dimensions, and each vocabulary character is used as the probability of a comment character.

[0138] It should be noted that the above-mentioned vocabulary probability distribution can also be softmax normalized so that the sum of the probabilities corresponding to all vocabulary characters is 1. And the number of self-attention layers can be one or more layers, exemplarily including 12 layers in a smaller language model and 24 layers in a larger language model, which is not limited in this application.

[0139] S304: Generate the probability of each comment character in the comment text based on each word list character to obtain the comment text of the target music.

[0140] In a possible implementation, the music review generating device can also determine the sampling parameters corresponding to each comment character in the comment text of the target music. The specific implementation of step S304 includes: for each comment character, based on the sampling parameters corresponding to the comment character and the probability of each vocabulary character corresponding to the comment character, select a vocabulary subset corresponding to the comment character from a preset vocabulary; for each comment character, select a vocabulary character from the vocabulary subset corresponding to the comment character as the comment character; and arrange the comment characters in order to obtain a text as the comment text of the target music.

[0141] Among them, the vocabulary characters in the vocabulary subset corresponding to each comment character are comment characters selected from a preset vocabulary. This possible implementation method first selects some vocabulary characters from the preset vocabulary to form a vocabulary subset through the sampling parameters corresponding to each comment character and each vocabulary probability (also called vocabulary probability distribution), and then selects (exemplarily randomly, or selects the vocabulary characters with the largest probability, or randomly selects from the vocabulary characters with probabilities exceeding a preset value) a vocabulary character from the corresponding vocabulary subset as the comment character in the comment text.

[0142] In a possible implementation, the sampling parameters include a cumulative probability threshold P and a probability quantity threshold K. The method of selecting a subset of vocabulary characters corresponding to a comment character from a preset vocabulary according to the sampling parameters corresponding to the comment character and the probabilities of each vocabulary character corresponding to the comment character specifically includes: for each comment character, selecting the top K vocabulary characters with the highest probabilities in the preset vocabulary to form a first candidate subset of vocabulary characters; determining multiple second candidate subsets of vocabulary characters from the first candidate subset of vocabulary characters; wherein, the sum of the probabilities of the vocabulary characters corresponding to the second candidate subset of vocabulary characters is greater than or equal to P, or, the vocabulary characters corresponding to the second candidate subset of vocabulary characters are the same as the vocabulary characters in the candidate subset of vocabulary characters and the sum of the probabilities of the vocabulary characters in the candidate subset of vocabulary characters is less than P; taking the second candidate subset of vocabulary characters with the smallest number of vocabulary characters as the subset of vocabulary characters corresponding to the comment character.

[0143] Wherein, P is a parameter involved in TOP-P sampling, and K is a parameter involved in TOP-K sampling. That is to say, this method first uses the TOP-K method to determine multiple second candidate subsets of vocabulary characters from the preset vocabulary, and then uses the TOP-P method to select the second candidate subset of vocabulary characters that meets the selection rules and has the smallest number of vocabulary characters from the multiple second candidate subsets of vocabulary characters as the subset of vocabulary characters corresponding to the comment character.

[0144] The following takes determining the vocabulary characters corresponding to a comment character as an example for illustration. Exemplarily, as Figure 6 shown, the preset vocabulary includes 10 vocabulary characters: "了 (le), 是 (shi), 这 (zhe), 曲 (qu), 好 (hao), 少 (shao), 的 (de), 我 (wo), 里 (li), 歌 (ge)", and the probabilities corresponding to each vocabulary character are "0.24, 0.22, 0.16, 0.13, 0.11, 0.05, 0.03, 0.025, 0.025, 0.01". When K is set to 7, the top 7 vocabulary characters with the highest probabilities can be determined from the preset vocabulary to form a first candidate subset of vocabulary characters: "了 (le), 是 (shi), 这 (zhe), 曲 (qu), 好 (hao), 少 (shao), 的 (de)". Then, when P is set to 0.9 and characters are selected from the first candidate subset of vocabulary characters, two second candidate subsets of vocabulary characters that satisfy the sum of the probabilities of the vocabulary characters being not less than 0.9 can be obtained. The total probabilities of the vocabulary characters in the two second candidate subsets of vocabulary characters are 0.91 (corresponding to "了 (le), 是 (shi), 这 (zhe), 曲 (qu), 好 (hao), 少 (shao)") and 0.94 (corresponding to "了 (le), 是 (shi), 这 (zhe), 曲 (qu), 好 (hao), 少 (shao), 的 (de)") respectively. Since the second candidate subset with the total probability of vocabulary characters being 0.91 includes fewer vocabulary characters, it is taken as the subset of vocabulary characters corresponding to this comment character: "了 (le), 是 (shi), 这 (zhe), 曲 (qu), 好 (hao), 少 (shao)".

[0145] Further, after determining the vocabulary subset corresponding to each comment character according to the above process, the music review generating device can determine the comment character according to the determined vocabulary subset. In this process, the probability of the vocabulary characters in the vocabulary subset corresponding to the comment character can be re-normalized so that the sum of the probabilities of all vocabulary characters in the vocabulary subset corresponding to the comment character is 1. Then, the normalized vocabulary subset can be randomly sampled, and the sampled vocabulary characters are used as the corresponding comment characters. Exemplarily, random sampling can be used to correspond all vocabulary characters in the vocabulary subset to each sub-interval in the [0, 1] interval according to the probability (normalized) (the size of the probability is reflected in the length of the sub-interval). Each time sampling is performed, a target value in the [0, 1] interval is randomly generated according to a uniform distribution. The vocabulary character corresponding to the sub-interval to which the target value falls is used as the corresponding comment character.

[0146] It should be noted that if the sum of the probabilities of all vocabulary characters in the first candidate vocabulary subset is less than the cumulative probability threshold value P, the first candidate vocabulary subset can be directly used as the second candidate vocabulary subset, and further as a vocabulary subset. In addition, if there are multiple second candidate vocabulary subsets with the least number of vocabulary characters in multiple second candidate vocabulary subsets, any second candidate vocabulary subset with the least number of vocabulary characters can be used as a vocabulary subset. Based on the above method, this scheme can use TOP-K sampling and TOP-P sampling in serial connection to further narrow the scope of random sampling of vocabulary characters and improve the accuracy of generating comment characters.

[0147] After each comment character in the comment text is determined based on the above process, the comment characters can be arranged in sequence according to the determined order, and the text composed after the arrangement is the comment text corresponding to the target music.

[0148] In a possible implementation, the comment text generated by the music review generating device according to the above steps S301 to S304 includes the first category comment characters and the second category comment characters, the first category comment characters are located before the second category comment characters in the comment text, and the value of the sampling parameter corresponding to the first category comment characters is greater than the value of the sampling parameter corresponding to the second category comment characters. It should be noted that the comment text may also include the third category comment characters (located after the second category comment characters), the fourth category comment characters (located after the third category comment characters), and so on; the number of categories is not limited in this application. Furthermore, the number of characters included in each category is also not limited, and the number of characters included in each category may be the same or different. Relatively speaking, the sampling parameters (for example, P, K) of the comment characters located at the front position in the comment text may be greater than the sampling parameters of the comment characters located at the back position in the comment text. Based on this method, when the music review generating device generates comment characters in sequence, it can dynamically adjust the sampling parameters corresponding to each comment character, so that the comment text finally generated is more diverse and fluent.

[0149] For sampling parameters P and K, the larger their values ​​are, the larger the character range available for sampling in the word list subset corresponding to the comment characters is, and the more random the generated comment text is (i.e., the more diverse it is); conversely, the smaller their values ​​are, the more precise the character range available for sampling in the comment character word list subset is, and the higher the semantic fluency of the generated comment text is. Therefore, the sampling parameters (for example, P and K) of comment characters located at the front of the comment text can be greater than the sampling parameters of comment characters located at the back of the comment text, so that in the early stage of generating the comment text, the diversity of the comment text is emphasized, and as the number of generated comment characters increases, the requirements for the fluency of the comment text are increased.

[0150] For example, Figure 7 is a schematic diagram of a process for dynamically adjusting sampling parameters provided in an embodiment of the present application, such as Figure 7As shown, in the first 3 steps of generating the review text (corresponding to generating the 1st to 3rd review characters), the music review generation device uses larger values of P and K (such as P1 and K1) to ensure the diversity at the beginning of the review text. For example, in the 1st step, P1 and K1 are used to generate the review character "This"; for the 4th to 10th steps of generating the review text (corresponding to generating the 4th to 10th review characters), the values of P and K are slightly reduced, and the values are modified to P2 and K2 (P2 < P1, K2 < K1) to ensure the unity of the diversity and fluency of the review text. For example, in the 4th step, P2 and K2 are used to generate the review character "song"; for the 11th step and later of generating the review text (corresponding to generating the 11th to Nth review characters), the values of P and K are further reduced, and the values are modified to P3 and K3 (P3 < P2, K3 < K2) to ensure the fluency of the main content of the review text. For example, in the 11th step, the review character "le" is generated. It should be noted that when adjusting the dynamic parameters, there may be other adjustment methods (such as modifying the values of the sampling parameters for generating each review character), etc., and this application does not limit this.

[0151] For the above step S304, the present application also proposes another possible implementation manner. In another possible implementation manner, the specific implementation process of the above step S304 includes: obtaining multiple first review characters based on the probabilities of each vocabulary character being the first review character in the review text; respectively generating the optional review texts corresponding to each first review character; and selecting one optional review text from the multiple optional review texts as the review text of the target music.

[0152] Specifically, in this possible implementation manner, the music review generation device uses multiple vocabulary characters as the optional characters for the first review character (the 1st review character) in the review text. Then, the music review generation device repeats the operation of determining the review text in the above first possible implementation manner for each first review character to determine the optional review text corresponding to each first review character. Finally, one review text is selected from the multiple optional review texts as the review text of the target music. This method retains more possibilities for generating the review text and can improve the diversity and richness of the review text. It should be noted that the number of the first review characters can be set in advance, and the example is S, so there are S final generated optional review texts.

[0153] In specific implementation, the above-mentioned method of generating optional comment text corresponding to each first comment character includes: for each first comment character, generating optional comment text corresponding to each first comment character according to the following steps; setting N to 1; using the character vector of the Nth comment character to update the fused feature vector, inputting the updated fused feature vector into the self-attention layer of the trained language model, and obtaining the target feature vector corresponding to the N+1th comment character; based on the mapping rules of the preset vocabulary, mapping the target feature vector corresponding to the N+1th comment character, and obtaining the probability of each vocabulary character in the preset vocabulary generating the N+1th comment character; determining the N+1th comment character based on the probability of each vocabulary character generating the N+1th comment character in the comment text; if the condition for stopping the generation of the optional comment text is not met, performing an operation of adding 1 to N, and returning to the step of updating the fused feature vector using the character vector of the Nth comment character; if the condition for stopping the generation of the optional comment text is met, sorting the first comment character and the determined N+1th comment characters to obtain the optional comment text corresponding to the first comment character.

[0154] Among them, the Nth comment character and the N+1th comment character refer to the Nth comment character and the N+1th comment character in the comment text. In the process of generating the optional comment text corresponding to each first comment character: first call the self-attention layer to process the initial feature vector corresponding to the N+1th comment character, determine the target feature vector corresponding to the N+1th comment character, and then determine the vocabulary probability distribution corresponding to the N+1th comment character by the mapping rule of the target feature vector and the preset vocabulary, and finally determine one vocabulary character each time (the example is the vocabulary character with the largest probability) as the N+1th comment character based on the vocabulary probability distribution; repeat this process until the stop generation condition is met, and arrange all the determined comment characters in order to obtain the optional comment text. The stop generation condition here can refer to the value of N reaching the preset threshold, or the N+1th comment character is the end identifier (period).

[0155] Optionally, based on the obtained multiple optional comment texts, the music review generating device can select the optional comment text with the largest sum of probabilities corresponding to comment characters among all the optional comment texts as the comment text of the target music.

[0156] For example, when the preset vocabulary includes 5 vocabulary characters and the number S of the first comment character is 2, when the music review generating device generates the first comment character, it will first obtain the vocabulary probability distribution corresponding to the first comment character according to the above step S303 (that is, the probability of each vocabulary character in the preset vocabulary being the first comment character), and determine the first two vocabulary characters with the largest probability from the 5 vocabulary characters as optional characters for the first comment character according to the vocabulary probability distribution. For example, {w 1,1 ,w1,2}, these two optional characters are respectively sent back to the trained pre-trained language model to participate in generating two optional comment texts.

[0157] In the generated 1,1 Use w for optional comment text for the first comment character. 1,1 The character vector of step S303 is used to update the fused feature vector to obtain the vocabulary probability distribution corresponding to the second comment character, and according to the vocabulary probability distribution corresponding to the second comment character, the vocabulary character with the largest probability is determined from the five vocabulary characters as the second comment character, for example, w 2,1 , and so on, until the optional comment text {w 1，1 ,w 2,1 ,...,w N,1}.

[0158] Similarly, when generating 1,1 Use w for optional comment text for the first comment character. 1,1 The character vector of update step S303 is used to obtain the vocabulary probability distribution corresponding to the second comment character, and the vocabulary character with the largest probability is determined from the five vocabulary characters as the second comment character according to the vocabulary probability distribution corresponding to the second comment character, for example, w 2,2 , and so on, until the optional comment text {w 1,2 ,w 2,2 ,...,w N,2}.

[0159] Finally, count the two optional comment texts {w 1,1 ,w 2,1 ,...,w N,1} and {w 1,2 ,w 2,2 ,...,w N,2 The sum of the cumulative probabilities of the comment characters in each optional comment text in}, and the optional comment text with the largest sum of probabilities, for example, {w 1,2 ,w 2,2 ,...,w N,2} as the comment text of the target music.

[0160] Below Figure 8 The example shown further illustrates the above process. Figure 8As shown, the preset vocabulary includes 6 vocabulary characters: "了 (le)", "是 (shi)", "这 (zhe)", "曲 (qu)", "好 (hao)", and "听 (ting)", and the value of the number S of the first comment character is 2. When generating the first comment character in the first step, it is determined that the vocabulary probability distribution corresponding to the first comment character is "0.16, 0.15, 0.24, 0.22, 0.14, 0.09". The "这 (zhe)" corresponding to the maximum probability of "0.24" and the "曲 (qu)" corresponding to "0.22" are used as the optional characters for the first comment character.

[0161] Then, in the second step, when determining the vocabulary probability distribution corresponding to the second comment character with "这 (zhe)" and "曲 (qu)" as the first comment character respectively, the examples are "0.08, 0.25, 0.15, 0.30, 0.17, 0.05" and "0.08, 0.14, 0.17, 0.15, 0.24, 0.22" respectively. The "曲 (qu)" corresponding to "0.30" and the "好 (hao)" corresponding to "0.24" are determined from these two vocabulary probability distributions as the second comment characters for "这 (zhe)" and "曲 (qu)". That is to say, in the second step, two texts, "这曲 (zhe qu)" and "这好 (zhe hao)", can be determined.

[0162] Similarly, through the third step and the fourth step, the two optional review texts are determined to be "这曲好听 (zhe qu hao ting)" and "曲好听了 (qu hao ting le)" respectively, and the cumulative probability corresponding to "这曲好听 (zhe qu hao ting)" is 0.24 + 0.30 + 0.30 + 0.28 = 1.12, and the cumulative probability corresponding to "曲好听了 (qu hao ting le)" is 0.22 + 0.24 + 0.28 + 0.25 = 0.99. Then finally, "这曲好听 (zhe qu hao ting)" can be used as the review text of the target music.

[0163] Based on Figure 3 the described embodiments, the present application can fuse two modal information (audio information and lyric information) of the target music to process the target music to generate a music review text, and the generated music review text has a stronger correlation with the target music, improving the quality of the generated music review text.

[0164] Referring to Fig. 9 , which is a schematic flowchart of another music review generation method provided by the embodiments of the present application. This method mainly involves the training stage of a trained language model and a trained audio encoding model, and can be applied to the above music review generation device, including steps S901 to S905. Among them:

[0165] S901. Concatenate the training review and the training lyric to obtain a training text.

[0166] Optionally, the training comments here may include one or more high-quality comment texts related to the training music. The high-quality comment texts may be comment texts with a high number of likes or comments. The training lyrics may be the lyrics texts that have been preprocessed in the lyrics texts corresponding to the training music. The preprocessing may filter out the repeated lyrics in the lyrics texts (retaining only one place), or retain only the most popular lyrics in the lyrics texts. Accordingly, if the training music has preprocessed the lyrics texts, when processing the target music, the lyrics texts may also be preprocessed accordingly to improve the accuracy of the model-generated comments.

[0167] Specifically, when concatenating the training comments and training lyrics, the concatenation order is adopted, that is, the training lyrics are in front and the training comments are in the back, so as to meet the semantic logic requirements of the language model to predict the training comments based on the training lyrics. Fig.10 As shown, Fig.10 This is a flow chart of the training phase of a trained language model provided in an embodiment of the present application. The model to be trained here is a GPT-2 model. Among them, "Take me flying..." is the training lyrics part, and "Youth..." is the training comment part. The training text after splicing them is "Take me flying...Youth...".

[0168] S902: Call the embedding layer of the language model to perform vector encoding on the training text to obtain a first text feature vector.

[0169] When the embedding layer of the language model performs vector encoding on the training text to obtain the first text feature vector, the embedding vector matching each character in the training text can be searched in the preset embedding dictionary matrix, and each row of the preset embedding dictionary matrix is ​​the embedding vector representation of one character.

[0170] Optionally, a position vector can be used to represent the position of each character in the training text. The position vector can be found in a preset position vector matrix. The embedding vector corresponding to each character is added to the position vector to obtain the vector component corresponding to each character in the first text feature vector (such as the "I" in "Take me flying...boy...").

[0171] S903, calling the self-attention layer of the language model, based on the first text feature vector and the mapping rule of the preset vocabulary, determining the probability of each vocabulary character in the preset vocabulary generating each comment character in the training comment.

[0172] Among them, when training the language model in this application, a masking method is adopted for the characters in the training comment part of the training text. The masking here means that when the language model determines the vocabulary probability distribution corresponding to any character in the training comment (that is, the probability of each vocabulary character in the preset vocabulary generating any comment character), it can only process the vector components corresponding to the characters before any character in the text feature vector, without considering the vector components corresponding to any character and the characters after any character. After determining the vocabulary probability distribution corresponding to any character in the training comment, when determining the vocabulary probability distribution corresponding to the character after any character, the masking state of the any character is lifted, so that the vector components corresponding to the characters before the character after the any character in the first text feature vector can be processed. This method can simulate the process of predicting the training comment based on the training lyrics.

[0173] Specifically, the self-attention layer of the language model first learns the semantic information of the training lyrics in the first text feature vector and generates a target feature vector based on the learned semantic information. After passing through the mapping rule of the preset vocabulary, the vocabulary probability distribution corresponding to the first character in the training comment can be determined. Further, the self-attention layer learns the semantic information of the training lyrics and the first character in the training comment in the first text feature vector, and generates a new target feature vector based on the learned semantic information to determine the vocabulary probability distribution corresponding to the second character in the training comment; still further, the self-attention layer learns the semantic information of the training lyrics and the first and second characters in the training comment in the first text feature vector, and generates a new target feature vector based on the learned semantic information to determine the vocabulary probability distribution corresponding to the third character in the training comment, and so on, until the vocabulary probability distribution corresponding to each character in the training comment is determined.

[0174] Exemplarily, in Fig.10 , after obtaining the first text feature vector of "Take me flying... Teenager...", the GPT-2 model will first mask the training comment part of "Teenager..." in "Take me flying... Teenager...". At this time, the self-attention layer determines the probability of generating "teenager" according to the vector representation corresponding to "Take me flying..." (that is, the probability of "teenager" as the first comment character in the vocabulary probability distribution). Then, the self-attention layer obtains the probability of generating "year" according to "Take me flying... Teenager", and repeats the above process to obtain the probability corresponding to each character in the training comment.

[0175] For the specific implementation method of this step, reference can be made to the specific implementation method of step S303 in the corresponding embodiment above, and details are not described here. Figure 3

[0176] ​S904: Generate the probability of each comment character in the training comment in the training text based on each vocabulary character, and determine a first cumulative likelihood function value.

[0177] Specifically, the probability of generating a training comment from the entire training lyrics can be obtained by the sum of the probabilities of each character in the training comment generated from the training lyrics, and the probability of each character except the first character in the training comment generated from the training lyrics is also related to the characters with determined probabilities in the training comment. It can be seen that the present application can use the sum of the probabilities of each character in the training comment generated from the training lyrics as the first cumulative likelihood function value.

[0178] For example, if the training lyrics include L characters, the training comments include C characters, and the first text feature vector is {h1,h2,h3,...,h L ,...,h L+c}, h L represents the vector representation corresponding to the Lth character in the training text, h1,h2,h3,...,h L is the vector representation corresponding to the training lyrics, h L+1 ,...,h L+c is the vector representation corresponding to the training comment. Then, the present application can determine the first cumulative likelihood function value based on Formula 1 according to the above principle, and determine the target loss function value of the language model training based on the first cumulative likelihood function value and Formula 2:

[0179]

[0180]

[0181] Among them, P(h i |h ＜i ) represents the probability of generating the i-th character based on the first i-1 characters in the training text, that is, the vocabulary probability distribution corresponding to the i-th character. For example, when i is L+1, P(h i |h ＜i ) represents the probability of predicting the first character in the training comment according to the training lyrics in the training text. When i is L+2, P(h i |h ＜i ) indicates the probability of predicting the second character in the training comment based on the training lyrics in the training text and the first character in the training comment, and so on. Based on this, the larger the value of the first cumulative likelihood function, The smaller the value of , the better the effect of the trained language model in generating comments.

[0182] S905: Train and update the parameters in the language model based on the first cumulative likelihood function value to obtain a trained language model.

[0183] Specifically, after obtaining the above target loss function value, the present application will train and update the parameters in the embedding layer and self-attention layer of the language model according to the gradient back propagation algorithm (for example, in Fig.10 The dashed arrow in the figure indicates the update process of gradient backpropagation. After completing one update, the next batch of training music will be received and processed according to the process of steps S901 to S905 to complete the second training update. Repeat this process. When the first cumulative likelihood function value is greater than the preset cumulative likelihood function threshold or When the loss function value is less than the preset value, the language model training is completed and the trained language model can be applied to the above Figure 3 The corresponding embodiment.

[0184] In one possible implementation, after obtaining the trained language model, the music review generating device will train the audio coding model to obtain the trained audio coding model, and the specific training method is: obtain the training audio data of the training music; call the audio coding model to perform vector encoding on the training audio data to obtain the training audio feature vector; call the embedding layer of the trained language model to perform vector encoding on the training text to obtain a second text feature vector; concatenate the second text feature vector with the training audio feature vector to obtain a training fusion feature vector; call the self-attention layer of the trained language model, and determine the probability of each vocabulary character in the preset vocabulary generating each comment character in the training comment based on the training fusion feature vector and the mapping rules of the preset vocabulary; determine the second cumulative likelihood function value based on the probability of each vocabulary character generating each comment character in the training comment in the training text; update the parameters in the audio coding model based on the second cumulative likelihood function value to obtain the trained audio coding model.

[0185] The training audio data is an audio data segment corresponding to the training lyrics. When obtaining the training audio data, the training audio data can be preprocessed, including framing, sampling, quantization, etc., to convert the analog waveform of the training audio data into a digital representation. Then, the preprocessing result can be input into the audio coding model to encode the training audio data to obtain the training audio feature vector, an example of which is {h1,h2,h3,...,h A Then, the training audio feature vector can be compared with the second text feature vector {h1,h2,h3,...,h L ,...,h L+c} to obtain a training fusion feature vector {h1,h2,h3,...,hA ,...,h A+L ,...,h A+L+c Here, the second text feature vector {h1,h2,h3,...,h L ,...,h L+c The training text is encoded using the embedding layer of the trained language model. The concatenation of the training audio feature vector and the second text feature vector can be found in Figure 3 The corresponding description of the illustrated embodiment is not repeated here. Further, the training fusion feature vector is input into the self-attention layer of the trained language model to obtain the probability (likelihood function value) of generating each character in the training comment.

[0186] Specifically, the second cumulative likelihood function value can be determined according to the following formula 3, and the target loss function value of the audio coding model training can be determined based on the second cumulative likelihood function value and formula 4.

[0187]

[0188]

[0189] Finally, the trained language model will convert the second cumulative likelihood function value (or ) is sent back and the parameters in the model are updated according to the gradient back-propagation algorithm. During the update process, the gradient will only be used to update the parameters in the audio coding model, and the parameters of the trained language model will be frozen to ensure that the determined parameters in the pre-trained language model are not contaminated.

[0190] For example, Fig.11 is a flowchart of a training phase of a trained audio coding model provided in an embodiment of the present application, where Figure 5 The audio encoder shown is used as a training model, Fig.11 The gradient update represented by the dashed arrow only works on the audio encoder, not on the GPT-2 model. Similarly, after completing this training, the audio data of the next batch of training music will be used to train the audio encoding model, and when the second cumulative likelihood function value is greater than the preset cumulative likelihood function threshold or the target loss function value If the loss is less than a preset loss threshold, the audio coding model training is completed, and a trained audio coding model is obtained.

[0191] It should be noted that the language model in this embodiment refers to a pre-trained language model obtained by self-supervised pre-training using a large amount of unlabeled corpus. Therefore, when the language model is re-trained in this application, the training comments and training lyrics of a small data set can be used to update the parameters in the language model. This method can improve the efficiency of training the language model.

[0192] based on Fig. 9 In the described embodiment, the present application can train the language model and the audio coding model in sequence, and the loss function adopted during training focuses on the training comment part, which can improve the effect and efficiency of the training of the two models. In addition, the language model used in the present application has learned a large number of common features through large-scale corpus in advance, so only a small number of training samples are needed to train the language model, and the training method is more convenient.

[0193] See also Fig.12 , is a schematic diagram of the structure of a music review generating device provided in an embodiment of the present application, the device 12 includes an acquisition unit 1201 and a processing unit 1202. Among them:

[0194] An acquisition unit 1201 is used to acquire a fusion feature vector of a target music, where the fusion feature vector is formed by concatenating a lyrics feature vector of the target music and an audio feature vector of the target music;

[0195] The acquisition unit 1201 is further used to acquire a preset vocabulary, wherein the preset vocabulary includes a plurality of preset vocabulary characters;

[0196] Processing unit 1202, for determining the probability of each character in the preset vocabulary generating each comment character in the comment text of the target music according to the fusion feature vector of the target music;

[0197] The processing unit 1202 is further configured to generate the probability of each comment character in the comment text based on each vocabulary character, so as to obtain the comment text of the target music.

[0198] In a possible implementation manner, the processing unit 1202, when used to determine the probability of each word character in the preset word list generating each comment character in the comment text of the target music according to the fusion feature vector of the target music, specifically includes:

[0199] According to the fused feature vector of the target music, determine the initial feature vector corresponding to each comment character in the comment text of the target music;

[0200] Input the initial feature vector corresponding to each comment character into the self-attention layer of the trained language model to obtain the target feature vector corresponding to each comment character;

[0201] Based on the mapping rules of the preset vocabulary, the target feature vectors corresponding to the various comment characters are mapped to obtain the probability of each vocabulary character in the preset vocabulary generating each comment character.

[0202] In a possible implementation, the comment text includes N comment characters; the processing unit 1202, when used to determine the initial feature vector corresponding to each comment character in the comment text of the target music according to the fusion feature vector of the target music, specifically includes:

[0203] When N is equal to 1, the fusion feature vector of the target music is determined as the initial feature vector corresponding to the first comment character in the comment text;

[0204] When N is greater than 1, the character vector of the N-1th comment character is used to update the fused feature vector; the updated fused feature vector is determined as the initial feature vector corresponding to the Nth comment character in the comment text.

[0205] In a possible implementation, the processing unit 1202 is further configured to: determine sampling parameters corresponding to each comment character in the comment text of the target music;

[0206] The processing unit 1201, when used to generate the probability of each comment character in the comment text based on each word list character to obtain the comment text of the target music, specifically includes:

[0207] For each comment character, according to the sampling parameters corresponding to the comment character and the probability of each word table character corresponding to the comment character, a word table subset corresponding to the comment character is selected from the preset word table;

[0208] For each comment character, select a word list character from the word list subset corresponding to the comment character as the comment character;

[0209] Arrange the comment characters in order to obtain a text as the comment text of the target music.

[0210] In a possible implementation, the sampling parameters include a cumulative probability threshold value P and a probability quantity threshold value K. When the processing unit 1202 is used to select a vocabulary subset corresponding to the comment character from a preset vocabulary according to the sampling parameters corresponding to the comment character and the probabilities of each vocabulary character corresponding to the comment character for each comment character, the processing unit 1202 specifically includes:

[0211] For each comment character, select the first K characters with the highest probability in the preset vocabulary to form a first subset of vocabulary to be selected;

[0212] Determine multiple second subsets of vocabulary to be selected from the first subset of vocabulary to be selected; wherein the sum of the probabilities corresponding to the vocabulary characters in the second subset of vocabulary to be selected is greater than or equal to P, or the vocabulary characters in the second subset of vocabulary to be selected are the same as the vocabulary characters in the subset of vocabulary to be selected and the sum of the probabilities corresponding to the vocabulary characters in the subset of vocabulary to be selected is less than P;

[0213] The second candidate vocabulary subset containing the least number of vocabulary characters is used as the vocabulary subset corresponding to the comment character.

[0214] In one possible implementation, the comment text includes first-category comment characters and second-category comment characters, the first-category comment characters are located before the second-category comment characters in the comment text, and the value of the sampling parameter corresponding to the first-category comment characters is greater than the value of the sampling parameter corresponding to the second-category comment characters.

[0215] In a possible implementation manner, the processing unit 1202, when used to generate the probability of each comment character in the comment text based on each word list character to obtain the comment text of the target music, specifically includes:

[0216] Generate the probability of the first comment character in the comment text based on each vocabulary character, and obtain multiple first comment characters;

[0217] Generate optional comment text corresponding to each first comment character respectively;

[0218] An optional comment text is selected from multiple optional comment texts as the comment text of the target music.

[0219] In a possible implementation manner, the comment text includes N comment characters; when the processing unit 1202 is used to generate the optional comment text corresponding to each first comment character, it specifically includes:

[0220] For each first comment character, generate the optional comment text corresponding to each first comment character according to the following steps;

[0221] Set N to 1;

[0222] Use the character vector of the Nth comment character to update the fused feature vector, input the updated fused feature vector into the self-attention layer of the trained language model, and obtain the target feature vector corresponding to the N+1th comment character;

[0223] Based on the mapping rules of the preset vocabulary, the target feature vector corresponding to the N+1th comment character is mapped to obtain the probability of each vocabulary character in the preset vocabulary generating the N+1th comment character;

[0224] Determine the N+1th comment character based on the probability of generating the N+1th comment character in the comment text based on each vocabulary character;

[0225] If the condition for stopping the generation of the optional comment text is not met, performing an operation of adding 1 to N, and returning to the step of updating the fused feature vector using the character vector of the Nth comment character;

[0226] If the condition for stopping the generation of the optional comment text is met, the first comment character and each of the determined N+1th comment characters are sorted to obtain the optional comment text corresponding to the first comment character.

[0227] In a possible implementation, the acquisition unit 1201 is further used to acquire lyrics text and audio data of the target music;

[0228] The processing unit 1202 is further used to: call the embedding layer of the trained language model to perform vector encoding on the lyrics text of the target music to obtain a lyrics feature vector of the target music;

[0229] The trained audio coding model is called to perform vector encoding on the audio data of the target music to obtain the audio feature vector of the target music.

[0230] In one possible implementation, the processing unit 1202 is further configured to:

[0231] splicing the training lyrics and training comments of the training music to obtain the training text;

[0232] Calling the embedding layer of the language model to perform vector encoding on the training text to obtain a first text feature vector;

[0233] Calling the self-attention layer of the language model to determine the probability of each character in the preset vocabulary generating each comment character in the training comment based on the first text feature vector and a mapping rule of the preset vocabulary;

[0234] Determining a first cumulative likelihood function value based on the probability of each character in the training comment in the training text being generated by each character in the vocabulary;

[0235] The parameters in the language model are trained and updated based on the first cumulative likelihood function value to obtain a trained language model.

[0236] In a possible implementation, the acquisition unit 1201 is further used to acquire training audio data of the training music;

[0237] The processing unit 1202 is further used to: call the audio coding model to perform vector coding on the training audio data to obtain a training audio feature vector;

[0238] Calling the embedding layer of the trained language model to perform vector encoding on the training text to obtain a second text feature vector;

[0239] Concatenate the second text feature vector with the training audio feature vector to obtain a training fusion feature vector;

[0240] Calling the self-attention layer of the trained language model, based on the training fusion feature vector and the mapping rule of the preset vocabulary, determines the probability of each vocabulary character in the preset vocabulary generating each comment character in the training comment;

[0241] Determining a second cumulative likelihood function value based on the probability of each character in the training comment in the training text being generated by each character in the vocabulary;

[0242] The parameters in the audio coding model are updated based on the second cumulative likelihood function value to obtain a trained audio coding model.

[0243] It should be noted that the functions of each unit module of the music review generation device in the embodiment of the present application can be specifically implemented according to the method in the above-mentioned method embodiment. The specific implementation process can refer to the relevant description of the above-mentioned method embodiment, which will not be repeated here.

[0244] See also Fig.13 , is a schematic diagram of the structure of a terminal device provided in an embodiment of the present application. The terminal device 13 may include: one or more processors 1301, a memory 1302, and a transceiver 1303. The processor 1301, the memory 1302, and the transceiver 1303 are connected via a bus 1304. The memory 1302 is used to store a computer program, which includes program instructions. The processor 1301 and the transceiver 1303 are used to execute the program instructions stored in the memory 1302 to perform the following operations:

[0245] Obtain a fused feature vector of the target music, where the fused feature vector is formed by concatenating the lyrics feature vector of the target music and the audio feature vector of the target music;

[0246] Obtaining a preset vocabulary, the preset vocabulary including a plurality of preset vocabulary characters;

[0247] Determine the probability of each character in the preset vocabulary generating each comment character in the comment text of the target music according to the fusion feature vector of the target music;

[0248] The probability of each comment character in the comment text is generated based on each vocabulary character to obtain the comment text of the target music.

[0249] It should be understood that in some feasible implementations, the processor 1301 may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc. The memory 1302 may include a read-only memory and a random access memory, and provide instructions and data to the processor 1301. A portion of the memory 1302 may also include a non-volatile random access memory. For example, the memory 1302 may also store information about the device type.

[0250] In a specific implementation, the above terminal device can execute the above-mentioned functions through its built-in functional modules. Figure 3 or Fig. 9 For the implementation methods provided in each step, please refer to the implementation methods provided in the above steps for details, which will not be repeated here.

[0251] The embodiment of the present application further provides a computer-readable storage medium, and the computer-readable storage medium stores computer-readable instructions executed by the aforementioned audio synthesis device, and the computer-readable instructions include program instructions. When the processor executes the above program instructions, it can execute the above Figure 3 , Fig. 9 The method in the corresponding embodiment, therefore, will not be repeated here. In addition, the description of the beneficial effects of the same method will not be repeated. For technical details not disclosed in the computer-readable storage medium embodiment involved in this application, please refer to the description of the method embodiment of this application. As an example, the program instructions can be deployed on a computer device, or executed on multiple computer devices located in one location, or, executed on multiple computer devices distributed in multiple locations and interconnected by a communication network, and multiple computer devices distributed in multiple locations and interconnected by a communication network can constitute a blockchain system.

[0252] According to one aspect of the present application, a computer program product or a computer program is provided, the computer program product or the computer program comprising computer instructions, the computer instructions being stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device can perform the above-mentioned Figure 3 , Fig. 9 The method in the corresponding embodiment is therefore not described again here.

[0253] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing related hardware through a computer program, and the above-mentioned program can be stored in a computer-readable storage medium, and when the program is executed, it can include the processes of the embodiments of the above-mentioned methods. The above-mentioned storage medium can be a disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM), etc.

[0254] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed by the present invention, and these modifications or replacements should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. A method for generating music reviews, characterized in that: The method comprises: Obtaining a fusion feature vector of a target music, wherein the fusion feature vector is formed by concatenating a lyrics feature vector of the target music and an audio feature vector of the target music; Obtaining a preset vocabulary, wherein the preset vocabulary includes a plurality of preset vocabulary characters; According to the fused feature vector of the target music, the initial feature vector corresponding to each comment character in the comment text of the target music is determined; wherein the comment text includes N comment characters, and when N is equal to 1, the fused feature vector of the target music is determined as the initial feature vector corresponding to the first comment character in the comment text; when N is greater than 1, the character vector of the N-1th comment character is used to update the fused feature vector, and the updated fused feature vector is determined as the initial feature vector corresponding to the Nth comment character in the comment text; Inputting the initial feature vector corresponding to each comment character into the self-attention layer of the trained language model to obtain the target feature vector corresponding to each comment character; Based on the mapping rules of the preset vocabulary, the target feature vectors corresponding to the respective comment characters are mapped to obtain the probability of each word character in the preset vocabulary generating each comment character in the comment text of the target music; The probability of each comment character in the comment text is generated based on each of the vocabulary characters to obtain the comment text of the target music.

2. The method according to claim 1, characterized in that Also includes: Determine sampling parameters corresponding to each comment character in the comment text of the target music; The generating the probability of each comment character in the comment text based on each of the vocabulary characters to obtain the comment text of the target music includes: For each of the comment characters, according to the sampling parameters corresponding to the comment characters and the probabilities of the respective vocabulary characters corresponding to the comment characters, a vocabulary subset corresponding to the comment characters is selected from the preset vocabulary; For each of the comment characters, selecting a word list character from the word list subset corresponding to the comment character as the comment character; Arrange the comment characters in order to obtain a text as the comment text of the target music.

3. The method according to claim 2, characterized in that The sampling parameters include a cumulative probability threshold P and a probability quantity threshold K; The step of selecting, for each comment character, a vocabulary subset corresponding to the comment character from the preset vocabulary according to the sampling parameter corresponding to the comment character and the probability of each vocabulary character corresponding to the comment character comprises: For each of the comment characters, select the first K characters with the highest probability in the preset vocabulary to form a first subset of vocabulary to be selected; Determine a plurality of second subsets of vocabulary to be selected from the first subset of vocabulary to be selected; wherein the sum of the probabilities corresponding to the vocabulary characters in the second subset of vocabulary to be selected is greater than or equal to P, or the vocabulary characters in the second subset of vocabulary to be selected are the same as the vocabulary characters in the first subset of vocabulary to be selected and the sum of the probabilities corresponding to the vocabulary characters in the first subset of vocabulary to be selected is less than P; The second to-be-selected vocabulary subset containing the least number of vocabulary characters is used as the vocabulary subset corresponding to the comment character.

4. The method according to claim 2, characterized in that: The comment text includes first-category comment characters and second-category comment characters, the first-category comment characters are located before the second-category comment characters in the comment text, and the values ​​of sampling parameters corresponding to the first-category comment characters are greater than the values ​​of sampling parameters corresponding to the second-category comment characters.

5. The method according to claim 1, characterized in that The generating the probability of each comment character in the comment text based on each of the vocabulary characters to obtain the comment text of the target music includes: Generate a probability of the first comment character in the comment text based on each of the vocabulary characters to obtain a plurality of first comment characters; Generating optional comment text corresponding to each of the first comment characters respectively; An optional comment text is selected from the plurality of optional comment texts as the comment text of the target music.

6. The method according to claim 5, characterized in that The comment text includes N comment characters; The step of respectively generating the optional comment text corresponding to each of the first comment characters comprises: For each of the first comment characters, generate the optional comment text corresponding to each of the first comment characters according to the following steps; Set N to 1; Use the character vector of the Nth comment character to update the fused feature vector, input the updated fused feature vector into the self-attention layer of the trained language model, and obtain the target feature vector corresponding to the N+1th comment character; Based on the mapping rules of the preset vocabulary, the target feature vector corresponding to the N+1th comment character is mapped to obtain the probability of each vocabulary character in the preset vocabulary generating the N+1th comment character; Determine the N+1th comment character based on the probability of generating the N+1th comment character in the comment text for each of the vocabulary characters; If the condition for stopping the generation of the optional comment text is not met, performing an operation of adding 1 to N, and returning to the step of updating the fused feature vector using the character vector of the Nth comment character; If the condition for stopping generating the optional comment text is met, the first comment character and each determined N+1th comment character are sorted to obtain the optional comment text corresponding to the first comment character.

7. The method according to any one of claims 1 to 6, characterized in that The method further comprises: Acquire lyrics text and audio data of the target music; Calling the embedding layer of the trained language model to perform vector encoding on the lyrics text of the target music to obtain a lyrics feature vector of the target music; The trained audio coding model is called to perform vector encoding on the audio data of the target music to obtain the audio feature vector of the target music.

8. The method according to claim 7, characterized in that The method further comprises: splicing the training lyrics and training comments of the training music to obtain the training text; Calling the embedding layer of the language model to perform vector encoding on the training text to obtain a first text feature vector; calling the self-attention layer of the language model to determine the probability of each word character in the preset word table generating each comment character in the training comment based on the first text feature vector and the mapping rule of the preset word table; The probability of generating each comment character in the training comment in the training text based on each of the vocabulary characters is used to determine a first cumulative likelihood function value; and the parameters in the language model are trained and updated based on the first cumulative likelihood function value to obtain the trained language model.

9. The method according to claim 8, characterized in that The method further comprises: Acquire training audio data of the training music; Calling the audio coding model to perform vector coding on the training audio data to obtain a training audio feature vector; calling the embedding layer of the trained language model to perform vector coding on the training text to obtain a second text feature vector; The second text feature vector is concatenated with the training audio feature vector to obtain a training fusion feature vector; the self-attention layer of the trained language model is called to determine the probability of each character in the preset vocabulary generating each comment character in the training comment based on the training fusion feature vector and the mapping rule of the preset vocabulary; The probability of generating each comment character in the training comments in the training text based on each of the vocabulary characters is used to determine a second cumulative likelihood function value; and the parameters in the audio coding model are updated based on the second cumulative likelihood function value to obtain the trained audio coding model.

10. A terminal device, characterized in that: The terminal device includes a memory and a processor; The memory is used to store computer programs; The processor is used to call the computer program from the memory so that the terminal device executes the method according to any one of claims 1 to 9.

11. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-readable instructions, and when the computer-readable instructions are executed on a terminal device, the terminal device executes the method according to any one of claims 1 to 9.

Images