Lyric generation method, lyric generation model training method, device, and medium

Abstract

The application relates to a song lyric generation method, a song lyric generation model training method, computer equipment and a storage medium. After a feature processing module of a song lyric generation model extracts features of an image and a text of a preset style, the features are fused to obtain fused features, a song lyric generation processing module generates a plurality of groups of candidate song lyric texts based on the fused features, outputs a song lyric text meeting a preset matching degree condition according to a matching degree of the candidate song lyric text and a song lyric generation material, and obtains a target song lyric text matching the image and the text according to the output song lyric text. Compared with a traditional method of generating a song lyric by specifying a number of words, the present application generates a suitable song lyric by combining an image and a text of a preset style, so that a song lyric meeting an atmosphere and a scene of the image and the text is obtained, and the accuracy of the generated song lyric is improved.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a lyric generation method, a training method for a lyric generation model, a computer device, a storage medium, and a computer program product. Background Technology

[0002] With the development of computer technology, music creation can now be done using computer devices. Lyric writing is essential for music creation. Currently, lyric creation using computer devices involves specifying the number of words or keywords to generate corresponding lyrics. However, generating lyrics by specifying the number of words and keywords often fails to produce lyrics that match the creator's intentions, leading to a decrease in the accuracy of the generated lyrics.

[0003] Therefore, current lyric generation methods suffer from low accuracy. Summary of the Invention

[0004] Therefore, it is necessary to provide a lyric generation method, a lyric generation model training method, a computer device, a computer-readable storage medium, and a computer program product that can improve accuracy in addressing the above-mentioned technical problems.

[0005] Firstly, this application provides a lyrics generation method, the method comprising:

[0006] Acquire lyrics generation materials, which include images and text in a preset style;

[0007] The image and text are input into a trained lyrics generation model. The feature processing module of the lyrics generation model extracts the image features and the text features of the text. The image features and text features are then fused to obtain fused features. The lyrics generation processing module of the lyrics generation model generates multiple sets of candidate lyrics texts based on the fused features. The matching degree of each set of candidate lyrics texts with the lyrics generation material is obtained. Based on the matching degree, the candidate lyrics texts that meet the preset matching degree conditions from the multiple sets of candidate lyrics texts are output.

[0008] Based on the candidate lyric texts output by the trained lyric generation model, a target lyric text that matches the lyric generation material is obtained.

[0009] In one embodiment, the feature processing module of the lyrics generation model extracts image features of the image and text features of the text, including:

[0010] The feature processing module encodes the image using a visual geometry group convolutional neural network to obtain the image vector corresponding to the image, which serves as the image feature;

[0011] The text vector corresponding to the text is extracted using a bidirectional gated recurrent unit neural network and used as the text feature.

[0012] In one embodiment, the step of extracting the text vector corresponding to the text as the text feature using a bidirectional gated recurrent unit neural network includes:

[0013] The text is segmented into words to obtain multiple corresponding word groups;

[0014] A word sequence is generated based on the multiple word groups. The word sequence is then input into a bidirectional gated recurrent unit neural network. Each word in the word sequence is mapped to a word vector through the bidirectional gated recurrent unit neural network. The word sequence containing multiple word vectors is then encoded to obtain the text vector corresponding to the text.

[0015] In one embodiment, the lyrics generation model includes a multilayer perceptron;

[0016] The fusion of the image features and text features to obtain the fused features includes:

[0017] The text features and image features are concatenated, and the concatenated features are input into the multilayer perceptron. The multilayer perceptron performs fusion encoding on the concatenated features to obtain the corresponding fusion encoding vector, which is used as the fusion feature.

[0018] In one embodiment, the lyrics generation processing module includes a one-way gated loop unit decoder;

[0019] The lyrics generation processing module of the lyrics generation model generates multiple sets of candidate lyrics text based on the fusion features, including:

[0020] The fused features are input into the one-way gated loop unit decoder, and the fused features are decoded by the one-way gated loop unit decoder to obtain multiple sets of candidate lyrics text.

[0021] In one embodiment, the lyrics generation processing module further includes: a bidirectional gated loop encoder and a multilayer perceptron;

[0022] The step of obtaining the matching degree between each group of candidate lyric texts and the lyric generation material based on the candidate lyric texts and fusion features, and outputting the candidate lyric texts that meet the preset matching degree conditions from the multiple groups of candidate lyric texts based on the matching degree, includes:

[0023] The bidirectional gated loop unit encoder obtains multiple candidate lyric text vectors corresponding to the multiple candidate lyric texts.

[0024] For each set of candidate lyrics text vectors, the candidate lyrics text vector is concatenated with the fusion feature, and the multilayer perceptron generates the matching degree between the candidate lyrics text vector and the lyrics generation material based on the concatenated candidate lyrics text vector and the fusion feature.

[0025] The candidate lyrics text with the highest matching score among multiple matching scores is selected as the candidate lyrics text that meets the preset matching score conditions and output.

[0026] In one embodiment, obtaining the multiple sets of candidate lyrics text vectors corresponding to the multiple sets of candidate lyrics texts through the bidirectional gated loop encoder includes:

[0027] For each group of candidate lyrics text, the candidate lyrics text is segmented into words to obtain the candidate word group sequence corresponding to the candidate lyrics text;

[0028] The candidate word sequence is mapped to word vectors to obtain the corresponding candidate word vector sequence;

[0029] The candidate word vector sequence is encoded by the bidirectional gated loop encoder to obtain the corresponding candidate lyric text vector.

[0030] In one embodiment, the step of obtaining text of a preset style includes:

[0031] When a user enters a text keyword in the text input area, the system queries the text database based on the text keyword and displays multiple candidate texts corresponding to the text keyword.

[0032] In response to the user's trigger information for the target text among the plurality of candidate texts, the target text is used as the text of the preset style.

[0033] Secondly, this application provides a method for training a lyrics generation model, the method comprising:

[0034] Obtain lyrics generation material samples and corresponding lyrics samples, wherein the lyrics generation material samples include text samples and image samples of a preset style;

[0035] The text samples and image samples are input into the lyrics generation model to be trained. The feature processing module of the lyrics generation model extracts the text features of the text samples and the image features of the image samples, and fuses the text features and image features to obtain fused features. The lyrics generation processing module of the lyrics generation model generates multiple sets of candidate lyrics text prediction results based on the fused features. Based on the multiple sets of candidate lyrics text prediction results and the fused features, the matching degree of each set of candidate lyrics text prediction results with the lyrics generation material samples is obtained. Based on the matching degree, the lyrics text prediction results that meet the preset matching degree conditions among the multiple sets of candidate lyrics text prediction results are output.

[0036] Based on the matching degree between the predicted lyrics text and the lyrics generation material sample, the model parameters of the lyrics generation model to be trained are adjusted until a preset training termination condition is met, thus obtaining a trained lyrics generation model; the preset training termination condition includes the matching degree being greater than or equal to a first preset threshold.

[0037] In one embodiment, the method further includes:

[0038] Obtain text samples and corresponding lyric samples for a preset text style;

[0039] The text sample is input into the lyrics generation processing module to be trained, and the lyrics generation processing module outputs the corresponding lyrics text prediction result based on the text sample.

[0040] Based on the matching degree between the predicted lyrics text and the text sample, the model parameters of the lyrics generation processing module to be trained are adjusted until a preset training termination condition is met, thus obtaining a trained lyrics generation processing module; the preset training termination condition includes the matching degree being greater than or equal to a second preset threshold.

[0041] Thirdly, this application provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the above-described method.

[0042] Fourthly, this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the above-described method.

[0043] Fifthly, this application provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the above-described method.

[0044] The aforementioned lyric generation method, lyric generation model training method, computer equipment, storage medium, and computer program product extract features from images and text of a preset genre through the feature processing module of the lyric generation model. These features are then fused to obtain fused features. The lyric generation processing module generates multiple sets of candidate lyric texts based on these fused features. The lyric texts that meet the preset matching criteria are output based on their matching degree with the lyric generation material. The target lyric text matching the image and text is obtained from the output lyric texts. Compared to traditional methods of generating lyrics by specifying a number of characters, this solution generates suitable lyrics by combining images and text of a preset genre, thus satisfying the requirements of matching the atmosphere and context of the image and text, and improving the accuracy of the generated lyrics. Attached Figure Description

[0045] Figure 1 This is a flowchart illustrating a lyrics generation method in one embodiment;

[0046] Figure 2 This is a schematic diagram of the lyrics generation interface in one embodiment;

[0047] Figure 3 This is a flowchart illustrating the model generation steps in one embodiment;

[0048] Figure 4 This is a flowchart illustrating the image feature extraction steps in one embodiment;

[0049] Figure 5 This is a flowchart illustrating the lyrics generation method in another embodiment;

[0050] Figure 6 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0051] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0052] In one embodiment, such as Figure 1 As shown, a lyrics generation method is provided. This embodiment illustrates the application of this method to a terminal. It is understood that this method can also be applied to a server, or to a system including both a terminal and a server, and is implemented through the interaction between the terminal and the server. The method includes the following steps:

[0053] Step S202: Obtain lyrics generation materials, which include images and text in a preset style.

[0054] The lyrics generation materials can be used to generate lyrics, including but not limited to images and text in preset styles. The images and preset text can be user-inputted information, such as image and text input by the user in the lyrics generation software on the terminal, allowing the terminal to generate corresponding lyrics based on the images and preset text. Images can be input in various ways. For example, the terminal can acquire images uploaded locally by the user, take photos using the terminal's camera, or capture images by hand. The text can be text with a preset style, such as poems for festivals. This text can be obtained through user input on the terminal. Users can input text on the terminal in various ways.

[0055] For example, in one embodiment, when the terminal acquires the text of the aforementioned preset text style, the user can input complete text in the text input area of ​​the terminal, so that the terminal can use the complete text input by the user as the text of the aforementioned preset text style. In one embodiment, the terminal can also, when detecting text keywords input by the user in the text input area, query the text database based on the text keywords and display multiple candidate texts corresponding to the text keywords; the user can trigger a certain text among the multiple candidate texts, such as clicking on the corresponding candidate text, so that the terminal can respond to the user's trigger information for the target text among the multiple candidate texts and use the target text as the text of the preset text style, that is, the terminal can recommend text to the user through methods such as automatic completion or similar text fragments.

[0056] Step S204: Input the image and text into the trained lyrics generation model. The feature processing module of the lyrics generation model extracts the image features and the text features of the extracted text. The image features and text features are then fused to obtain fused features. The lyrics generation processing module of the lyrics generation model generates multiple sets of candidate lyrics text based on the fused features. The matching degree between each set of candidate lyrics text and the lyrics generation material is obtained based on the matching degree. Based on the matching degree, the candidate lyrics text that meets the preset matching degree condition among the multiple sets of candidate lyrics text is output.

[0057] The terminal can pre-train the lyrics generation model to be trained; in some embodiments, the training process can also be performed on a server. During model training, the terminal can use text samples, image samples, and corresponding lyrics samples of a preset genre for training.

[0058] For example, in one embodiment, a training method for a lyrics generation model is also provided. The method includes: a terminal acquiring lyrics generation material samples and corresponding lyrics samples, the lyrics generation material samples including text samples and image samples of a preset style, and inputting the text samples and image samples into a lyrics generation model to be trained. The lyrics generation model includes a feature processing module and a lyrics generation processing module. The feature processing module of the lyrics generation model extracts text features from the text samples and image features from the image samples, and fuses the text features and image features to obtain fused features. The lyrics generation processing module of the lyrics generation model generates multiple sets of candidate lyrics text prediction results based on the fused features, and obtains the matching degree between each set of candidate lyrics text prediction results and the lyrics generation material samples based on the multiple sets of candidate lyrics text prediction results and the fused features. Based on the matching degree, the lyrics text prediction results that meet the preset matching degree conditions among the multiple sets of candidate lyrics text prediction results are output. The matching degree can represent the matching degree between the generated candidate lyrics text and the atmosphere and context of the text samples and image samples. For example, whether the lyrics text contains descriptions of objects in the image, whether it contains text descriptions of a preset style, etc.

[0059] The terminal can obtain the matching degree between the predicted lyrics text that meets the preset matching degree conditions and the lyrics generation material samples, and adjust the model parameters of the lyrics generation model to be trained according to the matching degree until the preset training termination condition is met, thus obtaining the trained lyrics generation model. The preset training termination condition includes several factors, such as the matching degree being greater than or equal to a first preset threshold within a preset number of training iterations, or the preset number of training iterations being reached.

[0060] The lyrics generation processing module can be a pre-trained module, for example, a module trained using text samples of a preset genre. This module can be trained on a terminal or trained on a server and then sent to the terminal. For example, in one embodiment, the terminal can obtain text samples of a preset genre and corresponding lyrics samples, and input the text samples into the lyrics generation processing module to be trained. The module then outputs the corresponding lyrics text prediction result based on the text samples. The terminal can obtain the matching degree between the lyrics text prediction result and the text samples, and adjust the model parameters of the lyrics generation processing module to be trained based on this matching degree until a preset training termination condition is met, thus obtaining the trained lyrics generation processing module. The preset training termination condition can include various factors, such as the matching degree being greater than or equal to a second preset threshold within a preset number of training iterations, or the number of training iterations reaching a preset number.

[0061] Specifically, the text in the aforementioned preset genre can be a type of poetry, such as ancient poems related to festivals. In some embodiments, the text in the aforementioned preset genre can also be prose related to festivals. The aforementioned lyrics generation processing module can be a one-way GRU (Gate Recurrent Unit) decoder. The terminal pre-trains this decoder using a large number of ancient poems related to festivals. During the training of the aforementioned lyrics generation model, the parameters are loaded, thereby combining the pre-trained lyrics generation processing module to train the aforementioned lyrics generation model, resulting in a better ancient-style lyrics generator capable of generating festive atmospheres.

[0062] In the actual lyrics generation process, users can input corresponding images and preset text styles into the terminal. Specifically, for example... Figure 2 As shown, Figure 2 This is a schematic diagram of the lyrics generation interface in one embodiment. The terminal can display multiple areas on the lyrics creation page, including an image input area, a text input area, and a lyrics generation area. Users can upload an image in the image input area and input text as a reference for the lyrics in the text input area, such as lines or paragraphs of poetry related to a festival. The terminal can utilize the input image and text, combined with the aforementioned lyrics generation model, to display the generated lyrics text in the lyrics generation area. The generated lyrics can be presented line by line and can include descriptive phrases about the image, such as describing objects in the image, the scene the image represents, and the festival. The generated lyrics can also include related phrases to the input text in a preset style; for example, the generated lyrics can be similar in style to the input text, and the generated lyrics text can include phrases from the input text, as well as descriptions of the scene and festival represented by the input text. Furthermore, users can edit and save the generated lyrics in the lyrics generation area.

[0063] The lyrics generation model described above can be an encoding / decoding model, and the specific process of generating lyrics using this model can be as follows: Figure 3 As shown, Figure 3 This is a flowchart illustrating the model generation steps in one embodiment. The feature processing module in the lyrics generation model may include a text processing module, an image feature extraction module, and a feature fusion module; the lyrics generation processing module in the lyrics generation model may include a lyrics generation module and a lyrics result sorting module. For example... Figure 3As shown, the lyrics generation model extracts text features from the input text through a text processing module and image features through an image feature extraction module. The image and text features are then fused through a feature fusion module. Specifically, the feature fusion module performs multimodal representations of the text and image features, fuses the image and text features through the contextual hidden layer states, and generates the corresponding lyrics text through decoding the hidden layer states.

[0064] The lyrics generation processing module can generate multiple lyrics texts. The lyrics generation model can obtain the matching degree between each group of lyrics texts and the lyrics generation material through the lyrics generation processing module, and sort the lyrics texts based on the matching degree. The lyrics texts that meet the preset matching degree conditions are output, such as the lyrics texts with a matching degree greater than or equal to the preset matching degree threshold, so that the output lyrics texts are more in line with the scene and atmosphere represented by the input image and text.

[0065] Step S206: Based on the candidate lyric text output by the trained lyric generation model, obtain the target lyric text that matches the lyric generation material.

[0066] The terminal can display the lyrics text output by the lyrics generation model that meets preset matching conditions, and users can edit or save the lyrics text. When the terminal detects that the user has saved the output lyrics text, it can determine that the output lyrics text is the target lyrics text that matches the image and text.

[0067] If the terminal detects that the user has edited the output lyrics text, the terminal can generate modified lyrics text based on the user's editing information. After the user triggers the save operation, the modified lyrics text will be used as the target lyrics text that matches the input image and text.

[0068] In the above-described lyrics generation method, the feature processing module of the lyrics generation model extracts features from the image and the text of a preset genre, then fuses these features to obtain fused features. The lyrics generation processing module then generates multiple sets of candidate lyrics text based on these fused features. Based on the matching degree between the candidate lyrics text and the lyrics generation material, the lyrics text that meet the preset matching conditions are output. The target lyrics text that matches the image and text is then obtained from the output lyrics text. Compared to traditional methods of generating lyrics by specifying a number of characters, this solution generates suitable lyrics by combining the image and the text of a preset genre, thus satisfying the requirement of lyrics matching the atmosphere and context of the image and text, and improving the accuracy of the generated lyrics.

[0069] In one embodiment, the feature processing module of the lyrics generation model extracts image features and text features of the text, including: encoding the image by the feature processing module through a visual geometric group convolutional neural network to obtain the image vector corresponding to the image, which is used as the image feature; and extracting the text vector corresponding to the text through a bidirectional gated recurrent unit neural network, which is used as the text feature.

[0070] In this embodiment, the lyrics generation model described above can extract features from images and text respectively through a feature processing module. Specifically, for image features, the terminal can use the feature processing module in the lyrics generation model to encode the image through a VGGNet-16 (Visual Geometry Group Net-16) convolutional neural network to form the image vector corresponding to the image, thereby obtaining image features based on the image vector.

[0071] Specifically, the structure of the aforementioned visual geometry group convolutional neural network can be as follows: Figure 4 As shown, Figure 4 This is a flowchart illustrating the image feature extraction step in one embodiment. The lyrics generation model uses a VGGNet-16 convolutional neural network as the image encoder, encoding the image into corresponding image vectors. The VGGNet-16 convolutional neural network can include multiple layers, such as... Figure 4 As shown in the model structure, this convolutional neural network can extract image features layer by layer from the convolutional layers and finally output an image vector through multiple channels of the fully connected layer. Specifically, the image vector can be represented as vec. img =VGGNet(img). Where, vec img This represents an image vector, where `img` represents the input image, and `VGGNet` represents the aforementioned visual geometry group convolutional neural network. Specifically, as follows... Figure 4 The convolutional structure in a convolutional neural network uses convolution, max pooling, fully connected layers, and ReLU (Rectified Linear Unit) activation function to extract features from the input image layer by layer, gradually increasing the number of channels, and finally outputting an image vector.

[0072] For text features, the terminal can use the feature processing module in the lyrics generation model to extract the text vector corresponding to the input text through a bidirectional GRU (Gate Recurrent Unit) neural network. The terminal can then obtain the text features corresponding to the input text based on the text vector. Specifically, the terminal can extract the text vector of the input text through word segmentation. For example, in one embodiment, the terminal can segment the input text to obtain multiple word groups. The lyrics generation model uses a bidirectional GRU neural network to generate a word group sequence based on these multiple word groups, and inputs the word group sequence into the bidirectional GRU neural network. The bidirectional GRU neural network maps each word group in the word group sequence to a word vector, resulting in a word group sequence containing multiple word vectors. By encoding the word group sequence containing multiple word vectors, the text vector corresponding to the text can be obtained.

[0073] Specifically, taking poetry as an example, the terminal can encode the input text into a text vector using a bidirectional GRU neural network. A fragment of poetry can be represented as a sequence of word groups. Specifically, it can be represented as... Where text represents the input text. Let {emb} represent the j-th token in the text, where each token represents a phrase in the text. By mapping the above phrase sequence to word vectors, we can obtain a phrase sequence containing multiple word vectors, specifically represented as {emb}. tok0 emb tok1 , ...,emb tokn}. Among them, emb tokj Let represent the word vector corresponding to the j-th word group. After the terminal encodes the word group sequence containing multiple word vectors through the GRU network described above, it can obtain the corresponding text vector, which can be specifically represented as vec. text =biGRU({emb tok0 emb tok1 , ...,emb tokn}). Among them, vec text represents a text vector, and biGRU represents the bidirectional gated recurrent unit neural network described above.

[0074] Through the above embodiments, the terminal can utilize various neural networks in the lyrics generation model to extract corresponding features from the input image and text. The terminal then uses the extracted features to generate lyrics, ensuring that the generated lyrics match the context and atmosphere of the input image and text, thereby improving the accuracy of lyrics generation.

[0075] In one embodiment, fusing image features and text features to obtain fused features includes: concatenating text features and image features, inputting the concatenated features into a multilayer perceptron, and performing fusion encoding on the concatenated features through the multilayer perceptron to obtain a corresponding fusion encoding vector, which serves as the fused feature.

[0076] In this embodiment, the terminal can fuse image features and text features using a lyrics generation model. The lyrics generation model may include a Multilayer Perceptron (MLP) network, which concatenates extracted text and image features and inputs the concatenated features into the MLP. The MLP then performs fusion encoding on the concatenated features to obtain a corresponding fusion encoding vector. The terminal can then obtain the fused features based on this fusion encoding vector.

[0077] Specifically, the above image features can be represented as vec img The above text features can be represented as vec text Then the terminal will vec img and vec text In the input lyrics generation model, the feature fusion module within the feature processing module outputs a feature fusion encoding vector through a multilayer perceptron network, which can be specifically represented as: Enc ens =MLP([vec img vec text ]). Among them, Enc ens This represents the feature fusion encoding vector.

[0078] In this embodiment, the terminal can use the feature processing module to fuse image features and text features, thereby enabling the terminal to generate lyrics text that matches the scene and atmosphere described by the input image and text, thus improving the accuracy of lyrics generation.

[0079] In one embodiment, the lyrics generation processing module of the lyrics generation model generates multiple sets of candidate lyrics text based on fusion features, including: inputting the fusion features into a one-way gated loop unit decoder, and decoding the fusion features through the one-way gated loop unit decoder to obtain multiple sets of candidate lyrics text.

[0080] In this embodiment, the lyrics generation processing module may include a one-way GRU (Gate Recurrent Unit) decoder. When generating candidate lyrics text, the lyrics generation model can input the fused features obtained after fusion into the one-way gated recurrent unit decoder in the lyrics generation processing module. The one-way gated recurrent unit decoder decodes the fused features to obtain multiple sets of candidate lyrics text.

[0081] The aforementioned unidirectional gated recurrent unit decoder can be pre-trained using a large number of ancient poems about festivals. During the training of the multimodal lyrics generation model, this decoder is loaded into the lyrics generation model during the parameter initialization phase, enabling the model to generate lyrics that better match the scene, atmosphere, and festival information described by the input image and text. Specifically, the terminal can input the feature fusion encoding vector Enc into the unidirectional gated recurrent unit decoder. ens Enc is decoded step by step through a one-way gated loop unit decoder. ens This generates multiple sets of candidate lyrics. Each line of lyrics in each set of candidate lyrics can be represented as: lyric n =GRU dec (Enc ens , {lyric0, lyric1,…,lyric n-1}); =softmax(lyric n ). Among them, lyric j This represents the j-th line of lyrics. Each line of lyrics can be generated by combining previously generated lyrics. Represents the nth element in the candidate lyrics text, GRU dec This represents a one-way gated loop unit decoder. `softmax` represents the normalization function. Each group of candidate lyrics can be represented as `lyric`, specifically as `lyric = { lyric0, lyric1, ..., lyric}`. n}

[0082] In this embodiment, the terminal can decode the fused features through a one-way gated loop unit decoder and generate corresponding lyrics text, thereby obtaining lyrics text that better matches the scene and atmosphere described by the input image and text, thus improving the accuracy of lyrics generation.

[0083] In one embodiment, the matching degree between each group of candidate lyrics texts and the lyrics generation material is obtained, and candidate lyrics texts that meet the preset matching degree conditions are output based on the matching degree. This includes: obtaining multiple candidate lyrics text vectors corresponding to multiple groups of candidate lyrics texts through a bidirectional gated cyclic unit encoder; for each group of candidate lyrics text vectors, concatenating the candidate lyrics text vector with the fusion feature, and generating the matching degree between the candidate lyrics text vector and the lyrics generation material based on the concatenated candidate lyrics text vector and the fusion feature through a multilayer perceptron; and outputting the candidate lyrics text corresponding to the matching degree with the largest matching degree value among multiple matching degrees as the candidate lyrics text that meets the preset matching degree conditions.

[0084] In this embodiment, the lyrics generation processing module further includes a bidirectional GRU (Gate Recurrent Unit) encoder and a multilayer perceptron. Thus, the terminal can obtain multiple sets of candidate lyrics text vectors corresponding to the multiple sets of candidate lyrics texts generated by the lyrics generation processing module in the lyrics generation model through the bidirectional GRU encoder. Each candidate lyrics text can correspond one-to-one with a candidate lyrics text vector.

[0085] In this process, the terminal can acquire candidate lyric text vectors through word segmentation in the lyrics generation and processing module. For example, in one embodiment, for each set of candidate lyric texts, the terminal can perform word segmentation on the candidate lyric text using the lyrics generation and processing module to obtain the corresponding candidate word group sequence. Then, the candidate word group sequence is mapped to word vectors to obtain the corresponding candidate word vector sequence. The lyrics generation and processing module can then encode the candidate word vector sequence using a bidirectional gated cyclic encoder to obtain the corresponding candidate lyric text vector.

[0086] For each set of candidate lyric text vectors, the terminal's lyric generation processing module can concatenate the candidate lyric text vector with its corresponding fusion feature. The fusion feature represents the fusion feature corresponding to the image and text of the candidate lyric text that generated the set of candidate lyric text vectors. The lyric generation processing module then uses a multilayer perceptron based on the concatenated candidate lyric text vector and the fusion feature to generate the matching degree between the candidate lyric text vector and the corresponding lyric generation material. Here, the image and text corresponding to the candidate lyric text vector represent the image and text used to generate the candidate lyric text corresponding to that candidate lyric text vector.

[0087] The terminal can determine the matching degree of multiple sets of candidate lyric text vectors through the lyrics generation and processing module, thus obtaining multiple matching degrees. The terminal can then select the candidate lyric text with the highest matching degree from these multiple matching degrees and use it as the lyric text that meets the preset matching degree conditions. The lyric generation model can then output this lyric text that meets the preset matching degree conditions.

[0088] Specifically, the lyrics generation module described above can generate multiple sets of candidate lyric texts. The lyric generation model calculates and sorts the matching scores of the generated lyric texts, then selects the result with the highest score for output, thus generating lyrics. A set of candidate lyric texts can be represented as a sequence of elements, specifically 'lyric='. , wherein lyric represents a set of candidate lyric texts, and the terminal performs text processing on the above candidate lyric texts through a lyric generation model, for example, obtaining corresponding candidate lyric text sequences through word vector mapping, and the candidate lyric text sequences can be specifically expressed as . Wherein, represents the sequence element corresponding to the j-th element in the candidate lyric text. The terminal can input the above candidate lyric text vector into a bidirectional gated recurrent unit encoder, and obtain the corresponding text vector by encoding the candidate lyric text sequence . Wherein vec lyric represents the candidate lyric text vector, and biGRU represents the bidirectional gated recurrent unit encoder.

[0089] The terminal can, through the lyric generation model, generate the candidate lyric text vector vec lyric and the fused feature vector Enc corresponding to the candidate lyric text vector ens are spliced, after processing the spliced features through a multi-layer perceptron, the matching degree value between the candidate lyric text and the input lyric generation material is calculated through a softmax activation function. It can be specifically expressed as: score lyric =softmax(MLP([Enc ens ; vec lyric )). Wherein, score lyric represents the matching degree value, and MLP represents the multi-layer perceptron. As shown in the above Figure 3 , for the candidate lyric texts "Ji xxx, xxx yu" and "Yue xxx, xxx jie" generated by the lyric generation processing module, the matching degrees calculated by the lyric generation processing module are 0.72 and 0.85 respectively, then the lyric generation model can output the lyric text corresponding to the highest score of 0.85 as the lyric text meeting the preset matching degree condition.

[0090] Through the above embodiment, the terminal determines the matching degree between multiple sets of candidate lyric texts and the input lyric generation materials through the bidirectional gated recurrent unit encoder and the multi-layer perceptron, and then outputs the qualified lyric text based on the matching degree, so that the output lyrics are more in line with the situation and atmosphere described by the input image and text, and improves the accuracy of lyric generation.

[0091] In one embodiment, as Figure 5 shown, Figure 5This is a flowchart illustrating the lyrics generation method in another embodiment. In this embodiment, the terminal can first acquire the image input by the user and the text of a preset genre, taking a classical Chinese poem about a festival as an example. The terminal can extract image features using a lyrics generation model, for example, by using deep learning algorithms such as convolutional neural networks to extract features from the input image, obtaining the image's feature vector, also known as image features. The model architecture of the lyrics generation model can use any pre-trained multimodal model, including but not limited to UNITER, BEiT, VisualBERT, and any other neural network pre-trained language model capable of processing images and text sequences.

[0092] The terminal can also use a lyrics generation model to segment ancient poems related to festivals, construct a vocabulary, such as the word sequence mentioned above, and convert the word sequence containing the poem text into word vectors to obtain text features. Ancient poem text processing can be carried out using natural language processing techniques, such as word embedding, RNN (Recurrent Neural Network), Transformer and other deep neural network algorithms.

[0093] The terminal can also fuse the aforementioned image and text features through a lyrics generation model. For example, it can fuse the vectors of image features with the word vectors of classical poems to obtain a fused feature vector, also known as a fused feature. The terminal can then use a recurrent neural network or similar method within the lyrics generation model to generate classical Chinese lyrics that match the input image and classical poem within a specific festival range; these are also called candidate lyrics. Furthermore, the terminal can perform deduplication and sentence structure adjustments on the generated classical Chinese lyrics within the lyrics generation model, and rank the multiple candidate lyrics based on their matching scores. For example, it can use methods such as Sentence-BERT, Roberta, and Albert to rank the matching scores, thereby obtaining the matching score between each candidate lyrics and the corresponding image and classical poem. The lyrics generation model then outputs the classical Chinese lyrics with the highest matching score.

[0094] Specifically, in one application example, suppose a user wants to create a classical Chinese-style song lyric for the Mid-Autumn Festival. The user can first input a Mid-Autumn Festival image, such as the moon or lanterns, into the terminal. Then, they can input several lines of classical Chinese poetry related to the Mid-Autumn Festival, such as "The moon is xxx, at xxx time." The terminal can then use the aforementioned lyric generation model to fuse the image features with the classical poetry features and generate classical Chinese-style lyrics that fit the atmosphere of the Mid-Autumn Festival, serving as the target lyric text. The output classical Chinese-style lyrics might look like this: The moon is xxx, at xxx festival. Raise xxx, together xxx. Prosperous xxx. May xxx.

[0095] Through the above embodiments, the terminal generates suitable lyrics by combining images and text in a preset style, thereby ensuring that the lyrics match the atmosphere and context of the images and text, and improving the accuracy of the generated lyrics. Furthermore, users can easily generate classical-style lyrics that fit the festive atmosphere based on pictures and related ancient poems, satisfying personalized and diverse needs.

[0096] It should be understood that although the steps in the flowcharts of the above embodiments are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the above embodiments may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0097] In one embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 6 As shown, the computer device includes a processor, memory, communication interface, display screen, and input device connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, NFC (Near Field Communication), or other technologies. When executed by the processor, the computer program implements a lyric generation method. The display unit is used to form a visually visible image and can be a display screen, a projection device, or a virtual reality imaging device. The display screen can be an LCD screen or an e-ink screen. The input device can be a touch layer covering the display screen, buttons, a trackball, or a touchpad on the computer device casing, or an external keyboard, touchpad, or mouse.

[0098] Those skilled in the art will understand that Figure 6 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0099] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the lyrics generation method described above.

[0100] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the lyrics generation method described above.

[0101] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the lyrics generation method described above.

[0102] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.

[0103] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0104] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0105] The above embodiments are merely illustrative of several implementation methods of this application, and their descriptions are relatively specific and detailed. However, they should not be construed as limiting the scope of this application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A method for generating lyrics, characterized in that, The method includes: Acquire lyrics generation materials, which include images and text in a preset style; The image and text are input into a trained lyrics generation model. The feature processing module of the lyrics generation model extracts image features from the image and text features from the text. The text features and image features are then concatenated. The concatenated features are input into the multilayer perceptron of the lyrics generation model. The multilayer perceptron performs fusion encoding on the concatenated features to obtain a corresponding fusion encoding vector, which serves as the fusion feature. The lyrics generation processing module of the lyrics generation model generates multiple sets of candidate lyrics texts based on the fusion feature and obtains the matching degree between each set of candidate lyrics texts and the lyrics generation material. Based on the matching degree, the candidate lyrics texts that meet the preset matching degree conditions from the multiple sets of candidate lyrics texts are output. Based on the candidate lyric text output by the lyric generation model, the target lyric text that matches the lyric generation material is obtained.

2. The method according to claim 1, characterized in that, The feature processing module of the lyrics generation model extracts image features from the image and text features from the text, including: The feature processing module encodes the image using a visual geometry group convolutional neural network to obtain the image vector corresponding to the image, which serves as the image feature; The text vector corresponding to the text is extracted using a bidirectional gated recurrent unit neural network and used as the text feature.

3. The method according to claim 2, characterized in that, The step of extracting the text vector corresponding to the text through a bidirectional gated recurrent unit neural network includes: The text is segmented into words to obtain multiple corresponding word groups; A word sequence is generated based on the multiple word groups. The word sequence is then input into a bidirectional gated recurrent unit neural network. Each word in the word sequence is mapped to a word vector through the bidirectional gated recurrent unit neural network. The word sequence containing multiple word vectors is then encoded to obtain the text vector corresponding to the text.

4. The method according to claim 1, characterized in that, The lyrics generation and processing module includes a one-way gated loop unit decoder; The lyrics generation processing module of the lyrics generation model generates multiple sets of candidate lyrics text based on the fusion features, including: The fused features are input into the one-way gated loop unit decoder, and the fused features are decoded by the one-way gated loop unit decoder to obtain multiple sets of candidate lyrics text.

5. The method according to claim 1, characterized in that, The lyrics generation and processing module also includes: a bidirectional gated loop unit encoder and a multilayer sensor; The step of obtaining the matching degree between each group of candidate lyric texts and the lyric generation material, and outputting the candidate lyric texts that meet the preset matching degree conditions from the multiple groups of candidate lyric texts based on the matching degree, includes: The bidirectional gated loop unit encoder obtains multiple candidate lyric text vectors corresponding to the multiple candidate lyric texts. For each set of candidate lyrics text vectors, the candidate lyrics text vector is concatenated with the fusion feature, and the multilayer perceptron generates the matching degree between the candidate lyrics text vector and the lyrics generation material based on the concatenated candidate lyrics text vector and the fusion feature. The candidate lyrics text corresponding to the highest matching degree among multiple matching degrees is selected as the candidate lyrics text that meets the preset matching degree conditions and output.

6. The method according to claim 5, characterized in that, The step of obtaining multiple candidate lyrics text vectors corresponding to the multiple candidate lyrics texts through the bidirectional gated loop encoder includes: For each group of candidate lyrics text, the candidate lyrics text is segmented into words to obtain the candidate word group sequence corresponding to the candidate lyrics text; The candidate word sequence is mapped to word vectors to obtain the corresponding candidate word vector sequence; The candidate word vector sequence is encoded by the bidirectional gated loop encoder to obtain the corresponding candidate lyric text vector.

7. The method according to claim 1, characterized in that, The steps to obtain text in a preset style include: When a user enters a text keyword in the text input area, the system queries the text database based on the text keyword and displays multiple candidate texts corresponding to the text keyword. In response to the user's trigger information for the target text among the plurality of candidate texts, the target text is used as the text of the preset style.

8. A training method for a lyrics generation model, characterized in that, The method includes: Obtain lyrics generation material samples and corresponding lyrics samples, wherein the lyrics generation material samples include text samples and image samples of a preset style; The text samples and image samples are input into the lyrics generation model to be trained. The feature processing module of the lyrics generation model extracts the text features of the text samples and the image features of the image samples. The text features and image features are then concatenated. The concatenated features are input into the multilayer perceptron of the lyrics generation model. The multilayer perceptron performs fusion encoding on the concatenated features to obtain the corresponding fusion encoding vector, which serves as the fusion feature. The lyrics generation processing module of the lyrics generation model generates multiple sets of candidate lyrics text prediction results based on the fusion feature. Based on the multiple sets of candidate lyrics text prediction results and the fusion feature, the matching degree between each set of candidate lyrics text prediction results and the lyrics generation material samples is obtained. Based on the matching degree, the lyrics text prediction results that meet the preset matching degree conditions among the multiple sets of candidate lyrics text prediction results are output. Based on the matching degree between the predicted lyrics text and the lyrics generation material sample, the model parameters of the lyrics generation model to be trained are adjusted until a preset training termination condition is met, thus obtaining a trained lyrics generation model; the preset training termination condition includes the matching degree being greater than or equal to a first preset threshold.

9. The method according to claim 8, characterized in that, The method further includes: Obtain text samples and corresponding lyric samples for a preset text style; The text sample is input into the lyrics generation processing module to be trained, and the lyrics generation processing module outputs the corresponding lyrics text prediction result based on the text sample. Based on the matching degree between the predicted lyrics text and the text sample, the model parameters of the lyrics generation processing module to be trained are adjusted until a preset training termination condition is met, thus obtaining a trained lyrics generation processing module; the preset training termination condition includes the matching degree being greater than or equal to a second preset threshold.

10. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 9.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 9.

Images