Song content generation method and device and medium
Through the multimodal large language model, the lyrics and song posters are generated by integrating text and picture features, which solves the problem that lyrics generation in the prior art is difficult to meet the diverse needs of users, and achieves a fuller and more diverse lyric content.
Patent Information
- Application Number
- CN202411327197.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-23
- Publication Date
- 2024-12-31
AI Technical Summary
Existing lyrics-assisted creative tools generate lyrics through text input, which is difficult to meet the complex and diverse needs of users, and the generated lyrics lack a sense of environment and diversity.
The multimodal large language model is used to encode and fuse text instructions and target images, generate feature vector sequences, and generate target lyrics and song posters through decoding, combining text features and image features to enrich the lyrics content.
The generated lyrics are more in line with the user's complex and diverse needs, have a higher sense of environment, and enrich the diversity of song content through song posters.
Smart Images
Figure CN119227633A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of artificial intelligence, and in particular to a song content generation method, device and medium. Background Art
[0002] The current lyrics auxiliary creation tools support users to input the number of words, keywords, themes and styles of the lyrics they want to generate, so that the lyrics auxiliary creation tools can generate corresponding lyrics according to one or more elements input by the user. However, when generating corresponding lyrics through text input, the generated lyrics may not meet the complex and diverse lyrics needs of users. Therefore, how to make the generated lyrics better meet the lyrics needs of users is a problem to be solved. Summary of the invention
[0003] In view of this, the purpose of the present invention is to provide a song content generation method, device and medium, which can combine text features and image features to make the generated lyrics more complete and more environmental, so that the generated lyrics can better meet the complex and diverse lyrics needs of users, and can also generate song posters to enrich the diversity of generated song content. The specific scheme is as follows:
[0004] In a first aspect, the present application provides a method for generating song content, comprising:
[0005] Obtaining a text instruction and a target image, and encoding the target image to obtain an image feature vector;
[0006] Encoding the text instruction using a multimodal large language model to obtain a text feature vector, fusing the image feature vector and the text feature vector to obtain a feature vector sequence, and predicting and generating an image feature sequence and a text feature sequence based on the feature vector sequence;
[0007] The text feature sequence is decoded to generate target lyrics, and the image feature sequence is decoded to generate a target song poster.
[0008] Optionally, encoding the text instruction using a multimodal large language model to obtain a text feature vector includes:
[0009] Using the word segmenter in the multimodal large language model to perform word segmentation processing on the text instruction, so as to determine a corresponding text word segmentation sequence based on a number of instruction word segments obtained by the word segmentation processing;
[0010] Each instruction word segment in the text word segmentation sequence is mapped to a corresponding feature vector through the word vector in the multimodal large language model, so as to obtain the text feature vector based on the mapped feature vectors.
[0011] Optionally, after decoding the text feature sequence to generate target lyrics, the method further includes:
[0012] The target lyrics are post-processed to obtain final lyrics; wherein the post-processing includes repeated word removal and sentence structure adjustment.
[0013] Optionally, encoding the target image to obtain an image feature vector includes:
[0014] Encoding the target image using an image encoder to obtain an image feature vector;
[0015] Accordingly, the decoding of the text feature sequence to generate target lyrics, and the decoding of the image feature sequence to generate a target song poster, includes:
[0016] Decoding the text feature sequence using the multimodal large language model to generate target lyrics, and decoding the image feature sequence using an image decoder to generate a target song poster;
[0017] Wherein, the picture encoder, the multimodal large language model and the picture decoder are all located in a pre-trained preset generation model.
[0018] Optionally, encoding the target picture by using a picture encoder to obtain a picture feature vector includes:
[0019] Performing feature extraction on the target image using a deep neural network for image feature extraction in the image encoder to obtain image extraction features;
[0020] The image feature extraction is transformed using a multi-layer perceptron in the image encoder to obtain the image feature vector aligned with the vector space of the text feature vector.
[0021] Optionally, the step of decoding the text feature sequence using the multimodal large language model to generate target lyrics includes:
[0022] The text decoder in the multimodal large language model is used to decode each feature vector in the text feature sequence respectively, so as to generate corresponding target lyrics based on each text segmentation obtained by decoding.
[0023] Optionally, the method further includes:
[0024] The image encoder is constructed based on a deep neural network and a multi-layer perceptron; wherein the deep neural network is a pre-trained neural network for image feature extraction;
[0025] Constructing the picture decoder based on the diffusion model, and constructing an initial generation model based on the picture encoder, the picture decoder and the pre-trained multimodal large language model;
[0026] The initial generation model is trained to obtain the pre-trained preset generation model.
[0027] Optionally, the training of the initial generation model to obtain the pre-trained preset generation model includes:
[0028] Inputting training data including text instructions and pictures into the initial generation model, processing the training data using the picture encoder and the multimodal large language model, so as to predict and generate a training picture feature sequence and a training text feature sequence;
[0029] The multimodal large language model is trained based on the training text feature sequence, and the image decoder is trained based on the training image feature sequence to obtain the pre-trained preset generation model.
[0030] Optionally, the training the multimodal large language model based on the training text feature sequence includes:
[0031] The training text feature sequence is decoded using the multimodal large language model to generate predicted lyrics, and the multimodal large language model is trained based on a first training loss between the predicted lyrics and real lyrics corresponding to the training data.
[0032] Optionally, the training the picture decoder based on the training picture feature sequence includes:
[0033] Using the picture decoder to add potential noise to the training picture feature sequence at the current time step to obtain a noise latent variable, and performing noise prediction on the noise latent variable to obtain predicted noise;
[0034] A second training loss is determined based on a difference between the predicted noise and the potential noise, and the picture decoder is trained based on the second training loss.
[0035] Optionally, the training of the initial generation model to obtain the pre-trained preset generation model includes:
[0036] Determine first training input data based on the text instruction, and determine first training output data based on lyrics corresponding to the first training input data, so as to construct a first training set based on the first training input data and the first training output data;
[0037] Determine second training input data based on the text instructions and the picture, and determine second training output data based on the lyrics and song poster corresponding to the second training input data, so as to construct a second training set based on the second training input data and the second training output data;
[0038] The initial generation model is alternately trained using the first training set and the second training set to obtain the pre-trained preset generation model.
[0039] In a second aspect, the present application provides an electronic device, including:
[0040] Memory, used to store computer programs;
[0041] A processor is used to execute the computer program to implement the aforementioned song content generation method.
[0042] In a third aspect, the present application provides a computer-readable storage medium for storing a computer program, which implements the aforementioned song content generation method when executed by a processor.
[0043] In a fourth aspect, the present application provides a computer program product, including a computer program / instruction, which implements the aforementioned song content generation method when executed by a processor.
[0044] In the present application, a text instruction and a target image are obtained, and the target image is encoded to obtain an image feature vector; the text instruction is encoded using a multimodal large language model to obtain a text feature vector, the image feature vector and the text feature vector are fused to obtain a feature vector sequence, and a picture feature sequence and a text feature sequence are predicted based on the feature vector sequence; the text feature sequence is decoded to generate target lyrics, and the picture feature sequence is decoded to generate a target song poster. It can be seen that the present application uses text instructions and images as multimodal inputs to encode, fuse, predict and decode text instructions and images, thereby generating higher quality lyrics and song posters. In this way, the generated lyrics can be made fuller and more environmental by combining text features and image features, so that the generated lyrics are more in line with the complex and diverse lyrics needs of users, and song posters can also be generated to enrich the diversity of generated song content. BRIEF DESCRIPTION OF THE DRAWINGS
[0045] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.
[0046] Figure 1 This is a system architecture diagram for generating song content disclosed in this application;
[0047] Figure 2 A flow chart of a song content generation method disclosed in this application;
[0048] Figure 3 A schematic diagram of an interactive interface disclosed in this application;
[0049] Figure 4 A generation model architecture diagram disclosed in this application;
[0050] Figure 5 A training flow chart of a generation model disclosed in this application;
[0051] Figure 6 This is a structural diagram of an electronic device disclosed in this application. DETAILED DESCRIPTION
[0052] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0053] The current lyrics auxiliary creation tools support users to input the number of words, keywords, themes and styles of the lyrics they want to generate, so that the lyrics auxiliary creation tools generate corresponding lyrics according to one or more elements input by the user. However, the corresponding lyrics generated by text input may be difficult to meet the complex and diverse lyrics needs of users. To this end, the present application provides a song content generation method, which can make the generated lyrics more full and more environmental by combining text features and image features, so that the generated lyrics are more in line with the complex and diverse lyrics needs of users, and can also generate song posters to enrich the diversity of generated song content.
[0054] In the song content generation method of this application, the system framework used can be specifically referred to Figure 1 As shown, it may specifically include: a background server 01 and one or more user terminals 02 connected to the background server 01 .
[0055] In this application, the user terminal 02 is used to obtain the text instructions and target images input by the user, and send the text instructions and target images to the background server 01. The user terminal 02 can be a mobile terminal, such as a smart phone, a tablet computer, a laptop computer, etc., or a web terminal, such as using a web browser to generate song content. And the way to obtain the text instructions includes but is not limited to keyboard input, voice to text, picture to text, etc.; the way to obtain the target image includes but is not limited to local image upload, photo upload, hand-drawn simple picture upload, etc.
[0056] Furthermore, the backend server 01 is used to execute the steps of the song content generation method, including obtaining the text instructions and target images sent by the user terminal 02, encoding the target images to obtain image feature vectors, and using a multimodal large language model to encode the text instructions to obtain text feature vectors; wherein, the vector spaces of the image feature vectors and the text feature vectors need to be aligned; then the multimodal large language model is used to fuse the image feature vectors and the text feature vectors to obtain a feature vector sequence, and based on the feature vector sequence and in accordance with the method of predicting the following context from the previous context, the image feature sequence and the text feature sequence are predicted and generated; then the text feature sequence is decoded to generate target lyrics, and the image feature sequence is decoded to generate a target song poster; finally, the target lyrics and the target song poster are returned to the user terminal 02 for visual display.
[0057] See also Figure 2 As shown, the embodiment of the present invention discloses a method for generating song content, including:
[0058] Step S11, obtaining a text instruction and a target image, and encoding the target image to obtain an image feature vector.
[0059] In this embodiment, the text instruction and the target image input by the user are obtained, and the target image is encoded using the image encoder to obtain the image feature vector. Specifically, since the image encoder is located in a pre-trained preset generation model, the text instruction and the target image are the multimodal inputs of the preset generation module, and then the target image is encoded using the image encoder in the preset generation model to obtain the image feature vector.
[0060] Among them, for the image feature vector, the deep neural network used for image feature extraction in the image encoder can be used to extract features of the target image to obtain image extraction features; then the multilayer perceptron (MLP) in the image encoder is used to transform the image extraction features based on the dimension of the vector space of the text feature vector to obtain the image feature vector aligned with the vector space of the text feature vector, that is, the image feature vector and the text feature vector have a vector space with unified dimensions.
[0061] For example, the deep neural network used for image feature extraction in the image encoder can use VGGNet (Visual Geometry Group Network), specifically VGGNet16, where VGGNet16 has 16 network layers (13 convolutional layers and 3 fully connected layers) containing weight parameters. Perform feature extraction to obtain image extraction features , that is, ; Then use the multi-layer perceptron MLP in the image encoder to convert the image extracted features into an image feature vector aligned with the vector space of the text feature vector , that is, .
[0062] Step S12: Encode the text instruction using a multimodal large language model to obtain a text feature vector, fuse the image feature vector and the text feature vector to obtain a feature vector sequence, and predict and generate an image feature sequence and a text feature sequence based on the feature vector sequence.
[0063] In this embodiment, since the multimodal large language model is located in a pre-trained preset generation model, the multimodal large language model in the preset generation model can be used to encode the text instruction to obtain a text feature vector. It should be noted that in order to achieve the subsequent fusion of the image feature vector and the text feature vector, it is necessary to make the image feature vector and the text feature vector have a vector space of the same dimension.
[0064] Among them, for the text feature vector, the word segmenter in the multimodal large language model can be used to perform word segmentation processing on the text instruction, so as to determine the corresponding text word segmentation sequence based on the several instruction word segments obtained by the word segmentation processing; then, each instruction word segment in the text word segmentation sequence is mapped to a corresponding feature vector through the word vector in the multimodal large language model, so as to obtain the text feature vector based on each mapped feature vector. For example, the multimodal large language model can adopt LLaMA (Large Language Model Meta AI, a pre-trained large language model based on the Transformer neural network architecture).
[0065] Specifically, the word segmenter in the multimodal large language model is used to segment the text instruction text to obtain several instruction segmentations, for example, ,in, Indicates the nth instruction word in the text instruction text. Then, the corresponding text word sequence is determined based on several instruction words. . Then, the word vectors in the multimodal large language model are used to map each instruction word in the text word sequence into corresponding feature vectors, for example , thereby determining the corresponding text feature vector based on each mapped feature vector .
[0066] In this embodiment, after obtaining the image feature vector through the image encoder in the preset generation model, the image feature vector is transmitted to the multimodal large language model in the preset generation model, so as to use the multimodal large language model to fuse the image feature vector and the text feature vector in the vector space with the same dimension, so as to obtain a feature vector sequence; wherein the fusion of the feature vector may include the concatenation of the feature vectors; and then the multimodal large language model is used to predict and generate the image feature sequence and the text feature sequence based on the feature vector sequence and in accordance with the method of predicting the following context based on the previous context.
[0067] Specifically, we use a multimodal large language model to transform the image feature vector and text feature vector Concatenate to get the corresponding feature vector sequence , and then input the feature vector sequence into the neural network layer in the multimodal large language model, so as to predict the feature vector sequence by predicting the following word by word according to the previous word through the neural network layer, thereby generating a picture feature sequence and text feature sequences It should be noted that at this time, the image feature sequence and text feature sequence have combined the features of the text instructions and the target image. Therefore, when subsequently decoding and generating lyrics and song posters based on the image feature sequence and the text feature sequence, the generated lyrics and song posters can be combined with the features of the text instructions and the target image, which can better meet the diverse needs of users.
[0068] Step S13: decoding the text feature sequence to generate target lyrics, and decoding the image feature sequence to generate a target song poster.
[0069] In this embodiment, the text decoder in the multimodal large language model is used to decode each feature vector in the text feature sequence respectively, so as to generate corresponding target lyrics based on each text segmentation obtained by decoding. In addition, after the multimodal large language model predicts and generates the image feature sequence, the image feature sequence is transmitted to the image decoder in the preset generation model, so as to decode the image feature sequence using the image decoder in the preset generation model to generate the corresponding target song poster. Among them, the image decoder is a decoder constructed based on the diffusion model.
[0070] Specifically, the text decoder in the multimodal large language model is used to decode each feature vector in the text feature sequence. Decode them separately to get each text word, for example, ,in, Indicates the target lyrics The nth text segmentation in the text segmentation; then generate the corresponding target lyrics based on each text segmentation , that is, the target lyrics are composed of a series of text segmentations, which can be expressed as .
[0071] Furthermore, after the target lyrics are generated by decoding the multimodal large language model, the target lyrics can be post-processed to obtain the final lyrics, and then the final lyrics and the target song poster are output for visual display on the user side. The post-processing of the target lyrics includes but is not limited to the removal of repeated words and the adjustment of sentence structure.
[0072] Combination Figure 3 and Figure 4As shown, a picture input area, a text input area, a lyrics and song poster display area are displayed to the user through a preset interactive interface, and a target picture uploaded by the user is obtained through the picture input area, and a text instruction input by the user is obtained through the text input area. For example, the text instruction may be "Based on this photo, I want to write a song about the story of childhood and the sea, with seagulls and coconut trees in my memories." Then the text instruction and the target image are input into the preset generation model, so as to encode the target image using the image encoder in the preset generation model to obtain the image feature vector, and the text instruction is encoded using the multimodal large language model in the preset generation model to obtain the text feature vector; then the multimodal large language model is used to splice the image feature vector and the text feature vector to obtain the feature vector sequence, and the image feature sequence and the text feature sequence are predicted based on the feature vector sequence and according to the way of predicting the following text in the previous text; finally, the text feature sequence is decoded using the multimodal large language model to generate the target lyrics, and the image feature sequence is decoded using the image decoder in the preset generation model to generate the target song poster, and then the target lyrics and the target song poster are displayed in the lyrics and song poster display area on the preset interactive interface for the user to view, and the target lyrics and the target song poster displayed in the lyrics and song poster display area can also be copied and saved. It can be found that the lyrics and song posters generated by the preset generation model provided in this embodiment are in line with the description elements contained in the text instruction and the target image.
[0073] It should be noted that the method proposed in this embodiment can also be applied to other scenarios in which images and texts are generated based on images and texts, such as in advertising scenarios, to generate advertising images and advertising texts based on text instructions and input images.
[0074] It should also be noted that the song content generated by the preset generation model in the embodiment of the present invention may include not only lyrics and song posters, but also audio and video, etc.; accordingly, the multimodal input of the preset generation model may include not only text and pictures, but also voice, etc. For example, when the multimodal input includes text, pictures and voice, and the song content includes lyrics, song posters, audio and video, it is possible to add scenery description elements, artistic conception description elements, audio and video style description elements, rhythm description elements, etc. to the text of the multimodal input, thereby encoding, fusing, predicting and decoding the multimodal input by the preset generation model, etc., to generate more diverse and higher quality song content.
[0075] It can be seen that this embodiment uses text instructions and pictures as multimodal inputs of the model to encode, fuse, predict and decode the text instructions and pictures, thereby generating higher quality lyrics and song posters; in this way, by combining text features and picture features, the generated lyrics can be made fuller and more environmental, so that the generated lyrics can better meet the user's complex and diverse lyrics needs. At the same time, song posters can also be generated to enrich the diversity of the generated song content, and the generated lyrics and song posters are also more in line with the descriptive elements contained in the text instructions and pictures.
[0076] Based on the previous embodiment, it can be seen that the previous embodiment describes how to generate multimodal outputs of lyrics and song posters based on multimodal inputs of text instructions and pictures. Next, this embodiment will elaborate on how to train the generation model. Figure 5 As shown, the embodiment of the present invention discloses a process of generating model training, including:
[0077] Step S21, constructing an image encoder based on a deep neural network and a multi-layer perceptron; wherein the deep neural network is a pre-trained neural network for extracting image features.
[0078] In this embodiment, a picture encoder is constructed based on a pre-trained deep neural network for picture feature extraction and a multi-layer perceptron MLP. It should be noted that, when the initial generation model constructed based on the picture encoder, the picture decoder and the pre-trained multimodal large language model is subsequently trained, the model parameters of the picture encoder can be fixed and frozen first, and then the other model parameters in the initial generation model can be trained and adjusted, so as to speed up the model training while ensuring the encoding performance of the picture encoder.
[0079] Among them, for deep neural networks, any deep neural network that can be used for image feature extraction can be selected, such as CNN (Convolutional Neural Networks), RNN (Recurrent Neural Network), Transformer large model, etc., and CNN can use VGGNet neural network, etc. In addition, deep neural networks can also use other pre-trained multimodal encoders, such as ImageBind, UNITER (Universal Image-Text Representation Learning, a universal image-text representation learning), BEiT (Bidirectional Encoder representation from Image Transformers, bidirectional encoder representation from image transformers), VisualBERT (Visual Bidirectional Encoder Representationsfrom Transformers, a visual-language pre-training model), etc.
[0080] Step S22: constructing an image decoder based on the diffusion model, and constructing an initial generation model based on the image encoder, the image decoder and the pre-trained multimodal large language model.
[0081] In this embodiment, an image decoder is constructed based on a diffusion model, and an initial generation model is constructed based on an image encoder, an image decoder, and a pre-trained multimodal large language model; wherein the image encoder is connected to the multimodal large language model for data transmission, and the multimodal large language model is connected to the image decoder for data transmission.
[0082] Among them, for the Diffusion Model (DM), you can choose any model that can generate images, such as DALL-E2 (a text-generated image system based on image text description), ImaGen (a large model of text-generated image), ControlNet (control network), etc., or you can also choose other deep learning algorithms that can generate images, such as CVAE (Conditional Variational Autoencoder), DDPM (Denoising Diffusion Probabilistic Model), etc.
[0083] Among them, for the multimodal large language model, any pre-trained large language model can be selected, including but not limited to LLaMA, BLOOM, GLM (General Language Model), and any other pre-trained large language model that can process text feature sequences.
[0084] Step S23: training the initial generation model to obtain a pre-trained preset generation model.
[0085] In this embodiment, after the initial generation model is constructed, the initial generation model is trained to obtain a pre-trained preset generation model. Specifically, the initial generation model is iteratively trained to obtain a trained model, and when the current number of iterations reaches a preset number of iterations or the model accuracy of the trained model reaches a preset accuracy threshold, the model training is terminated, and the pre-trained preset generation model is determined based on the trained model.
[0086] According to one of the embodiments, training data including text instructions and pictures are input into an initial generation model, so that the training data is processed by using a picture encoder and a multimodal large language model in the initial generation model, thereby predicting and generating a training picture feature sequence and a training text feature sequence; then the multimodal large language model is trained based on the training text feature sequence, and the picture decoder is trained based on the training picture feature sequence, thereby obtaining the pre-trained preset generation model.
[0087] For the training picture feature sequence and the training text feature sequence, the picture encoder in the initial generation model is used to encode the pictures in the training data to obtain the training picture feature vector, and the pre-trained multimodal large language model in the initial generation model is used to encode the text instructions in the training data to obtain the training text feature vector; wherein the vector space of the training picture feature vector and the training text feature vector are aligned. Then, the pre-trained multimodal large language model in the initial generation model is used to splice the training picture feature vector and the training text feature vector to obtain the training feature vector sequence, and the training feature vector sequence is predicted according to the method of predicting the following from the previous context to generate the training picture feature sequence and the training text feature sequence.
[0088] In the process of training a multimodal large language model based on a training text feature sequence, the training text feature sequence is first decoded using the multimodal large language model to generate predicted lyrics, and then the first training loss between the predicted lyrics and the real lyrics corresponding to the training data is calculated, and the multimodal large language model is trained using the first training loss. The first training loss may be a cross entropy loss, that is, the cross entropy loss between the predicted lyrics and the real lyrics corresponding to the training data is calculated, and the pre-trained multimodal large language model in the initial generation model is trained based on the cross entropy loss.
[0089] In the process of training the picture decoder based on the training picture feature sequence, the picture decoder can first be used to add potential noise to the training picture feature sequence at the current time step to obtain a noise latent variable, and the noise latent variable can be predicted to obtain predicted noise; then, a second training loss is determined based on the difference between the predicted noise and the potential noise, and the picture decoder is trained based on the second training loss.
[0090] Specifically, the conditional encoder in the image decoder is used to encode the training image feature sequence, and potential noise is added to the image feature encoded by the conditional encoder at the current time step to obtain a noise latent variable, and then the UNet network in the image decoder is used to perform noise prediction on the noise latent variable to obtain predicted noise; then the noise difference between the predicted noise and the potential noise is calculated, and the image decoder is trained using the second training loss determined based on the noise difference. The calculation formula of the second training loss can be expressed as:
[0091] ;
[0092] in, represents the second training loss, represents the maximum likelihood estimate, represents the two-norm, represents the potential noise, represents the noise latent variable, represents the time step, represents the training image feature sequence, represents a conditional encoder, Represents the predicted noise obtained by the UNet network for noise latent variable prediction. In this way, the image decoder is trained using the second training loss to guide the image decoder to learn higher quality song poster generation.
[0093] It can be found that in the embodiment of the present invention, when the model parameters of the picture encoder are fixed and frozen, the multimodal large language model is trained using the first training loss, and the picture decoder is trained using the second training loss. Therefore, by combining the first training loss and the second training loss, the training of the generation model including the picture encoder, the multimodal large language model and the picture decoder is achieved, thereby improving the training effect and training speed of the generation model.
[0094] According to another embodiment, considering that there are more pure text training data containing text instructions and corresponding lyrics, and less graphic training data containing text instructions and pictures, lyrics and song posters corresponding to the text instructions and pictures. Therefore, in order to increase the speed of model training, a first training set can be constructed based on text instructions and corresponding lyrics, and a second training set can be constructed based on text instructions and pictures, lyrics and song posters corresponding to the text instructions and pictures, and then the initial generation model is alternately trained using the first training set and the second training set, so as to obtain a pre-trained preset generation model.
[0095] For the first training set, the first training input data is determined based on the text instruction, and the first training output data is determined based on the lyrics corresponding to the first training input data, so as to construct the first training set based on the first training input data and the first training output data. At this time, the first training set is a pure text training set. For example, the first training input data is "write a lyric about friendship", and the first training output data is "(Main Verse) [Line Break\n] On the road filled with sunshine, we walked side by side [Line Break\n] Those carefree days, laughter woven into ballads...".
[0096] For the second training set, the second training input data is determined based on the text instructions and the picture, and the second training output data is determined based on the lyrics and song poster corresponding to the second training input data, so as to construct the second training set based on the second training input data and the second training output data. At this time, the second training set is a picture-text training set. For example, the second training input data is "[Picture] Write a song about love and peace based on the picture", and the second training output data is "[Song poster][Line break\n] (Main Verse) [Line break\n] Under the blue sky, the dove of peace spreads its wings and flies [Line break\n] The Statue of Liberty stands, guarding the light of love and peace...".
[0097] It can be seen that when training the initial generation model constructed based on the image encoder, image decoder and pre-trained multimodal large language model, this embodiment first fixes and freezes the model parameters of the image encoder, and then trains and adjusts other model parameters in the initial generation model, thereby ensuring the encoding performance of the image encoder, improving the speed of model training and generating higher quality lyrics and song posters through the model; and, this embodiment uses a plain text training set and a graphic training set to alternately train the initial generation model, thereby improving the speed of model training. In addition, this embodiment constructs a picture decoder based on a diffusion model, and optimizes the training of the picture decoder, so that the picture decoder can learn to generate higher quality song posters.
[0098] Furthermore, the present application also discloses an electronic device. Figure 6 This is a structural diagram of an electronic device 10 according to an exemplary embodiment. The content in the diagram cannot be regarded as any limitation on the scope of use of the present application.
[0099] Figure 6 The present invention provides a schematic diagram of the structure of an electronic device 10 provided in an embodiment of the present application. The electronic device 10 may specifically include: at least one processor 11, at least one memory 12, a power supply 13, a communication interface 14, an input / output interface 15, and a communication bus 16. The memory 12 is used to store a computer program, which is loaded and executed by the processor 11 to implement the relevant steps in the song content generation method disclosed in any of the aforementioned embodiments. In addition, the electronic device 10 in this embodiment may specifically be an electronic computer.
[0100] In this embodiment, the power supply 13 is used to provide working voltage for each hardware device on the electronic device 10; the communication interface 14 can create a data transmission channel between the electronic device 10 and the external device, and the communication protocol it follows is any communication protocol that can be applied to the technical solution of the present application, and is not specifically limited here; the input and output interface 15 is used to obtain external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs and is not specifically limited here.
[0101] In addition, the memory 12, as a carrier for storing resources, can be a read-only memory, a random access memory, a disk or an optical disk, etc. The resources stored thereon can include an operating system 121, a computer program 122, etc., and the storage method can be temporary storage or permanent storage.
[0102] The operating system 121 is used to manage and control the hardware devices and computer programs 122 on the electronic device 10, which can be Windows Server, Netware, Unix, Linux, etc. In addition to including computer programs that can be used to complete the song content generation method performed by the electronic device 10 disclosed in any of the aforementioned embodiments, the computer program 122 can further include computer programs that can be used to complete other specific tasks.
[0103] Furthermore, the present application also discloses a computer-readable storage medium for storing a computer program; wherein the computer program, when executed by a processor, implements the song content generation method disclosed above. The specific steps of the method can refer to the corresponding contents disclosed in the above embodiments, and will not be repeated here.
[0104] Furthermore, the present application also discloses a computer program product, including a computer program / instruction, wherein the computer program / instruction, when executed by a processor, implements the song content generation method disclosed above. The specific steps of the method can refer to the corresponding contents disclosed in the above embodiments, and will not be repeated here.
[0105] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.
[0106] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.
[0107] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.
[0108] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.
[0109] The technical solution provided by the present application is introduced in detail above. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea. At the same time, for general technicians in this field, according to the idea of the present application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.
Claims
1. A method for generating song content, characterized in that: include: Obtaining a text instruction and a target image, and encoding the target image to obtain an image feature vector; Encoding the text instruction using a multimodal large language model to obtain a text feature vector, fusing the image feature vector and the text feature vector to obtain a feature vector sequence, and predicting and generating an image feature sequence and a text feature sequence based on the feature vector sequence; The text feature sequence is decoded to generate target lyrics, and the image feature sequence is decoded to generate a target song poster.
2. The song content generating method according to claim 1, characterized in that: The step of encoding the text instruction using a multimodal large language model to obtain a text feature vector includes: Using the word segmenter in the multimodal large language model to perform word segmentation processing on the text instruction, so as to determine a corresponding text word segmentation sequence based on a number of instruction word segments obtained by the word segmentation processing; Each instruction word segment in the text word segmentation sequence is mapped to a corresponding feature vector through the word vector in the multimodal large language model, so as to obtain the text feature vector based on the mapped feature vectors.
3. The song content generating method according to claim 1, characterized in that: After decoding the text feature sequence to generate the target lyrics, the method further includes: The target lyrics are post-processed to obtain final lyrics; wherein the post-processing includes repeated word removal and sentence structure adjustment.
4. The song content generating method according to any one of claims 1 to 3, characterized in that: The step of encoding the target image to obtain an image feature vector includes: Encoding the target image using an image encoder to obtain an image feature vector; Accordingly, the decoding of the text feature sequence to generate target lyrics, and the decoding of the image feature sequence to generate a target song poster, includes: Decoding the text feature sequence using the multimodal large language model to generate target lyrics, and decoding the image feature sequence using an image decoder to generate a target song poster; Wherein, the picture encoder, the multimodal large language model and the picture decoder are all located in a pre-trained preset generation model.
5. The song content generating method according to claim 4, characterized in that: The step of encoding the target picture by using a picture encoder to obtain a picture feature vector includes: Performing feature extraction on the target image using a deep neural network for image feature extraction in the image encoder to obtain image extraction features; The image feature extraction is transformed using a multi-layer perceptron in the image encoder to obtain the image feature vector aligned with the vector space of the text feature vector.
6. The song content generating method according to claim 4, characterized in that: The step of decoding the text feature sequence using the multimodal large language model to generate target lyrics includes: The text decoder in the multimodal large language model is used to decode each feature vector in the text feature sequence respectively, so as to generate corresponding target lyrics based on each text segmentation obtained by decoding.
7. The song content generating method according to claim 4, characterized in that: Also includes: The image encoder is constructed based on a deep neural network and a multi-layer perceptron; wherein the deep neural network is a pre-trained neural network for image feature extraction; Constructing the picture decoder based on the diffusion model, and constructing an initial generation model based on the picture encoder, the picture decoder and the pre-trained multimodal large language model; The initial generation model is trained to obtain the pre-trained preset generation model.
8. The song content generating method according to claim 7, characterized in that: The training of the initial generation model to obtain the pre-trained preset generation model includes: Inputting training data including text instructions and pictures into the initial generation model, processing the training data using the picture encoder and the multimodal large language model, so as to predict and generate a training picture feature sequence and a training text feature sequence; The multimodal large language model is trained based on the training text feature sequence, and the image decoder is trained based on the training image feature sequence to obtain the pre-trained preset generation model.
9. The song content generating method according to claim 8, characterized in that: The training of the multimodal large language model based on the training text feature sequence includes: The training text feature sequence is decoded using the multimodal large language model to generate predicted lyrics, and the multimodal large language model is trained based on a first training loss between the predicted lyrics and real lyrics corresponding to the training data.
10. The song content generating method according to claim 8, characterized in that: The training of the picture decoder based on the training picture feature sequence comprises: Using the picture decoder to add potential noise to the training picture feature sequence at the current time step to obtain a noise latent variable, and performing noise prediction on the noise latent variable to obtain predicted noise; A second training loss is determined based on a difference between the predicted noise and the potential noise, and the picture decoder is trained based on the second training loss.
11. The song content generating method according to claim 7, characterized in that: The training of the initial generation model to obtain the pre-trained preset generation model includes: Determine first training input data based on the text instruction, and determine first training output data based on lyrics corresponding to the first training input data, so as to construct a first training set based on the first training input data and the first training output data; Determine second training input data based on the text instructions and the picture, and determine second training output data based on the lyrics and song poster corresponding to the second training input data, so as to construct a second training set based on the second training input data and the second training output data; The initial generation model is alternately trained using the first training set and the second training set to obtain the pre-trained preset generation model.
12. An electronic device, characterized in that: include: Memory, used to store computer programs; A processor, configured to execute the computer program to implement the song content generation method as described in any one of claims 1 to 11.
13. A computer-readable storage medium, characterized in that: Used to store a computer program, which, when executed by a processor, implements the song content generation method as described in any one of claims 1 to 11.