Image generation method and device, equipment and storage medium

Abstract

The invention discloses an image generation method and device, equipment and a storage medium, and relates to the technical field of computers. The method comprises the following steps: acquiring basic information of a song; obtaining at least one target vocabulary corresponding to the basic information, wherein the target vocabulary is a vocabulary describing the style or content of the song; generating a target image corresponding to the at least one target vocabulary through a text graph model according to the at least one target vocabulary; according to the target image and the first cover image of the song, a fused image is obtained, and the fused image comprises a main body in the target image and a background in the first cover image; and generating a second cover image of the song according to the fused image and the first cover image through a graph-to-graph model. According to the method, the target image and the first cover image of the song are fused, so that the main body in the second cover image is regenerated, and the relevance between the second cover image and the basic information of the song is improved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to an image generation method, device, equipment and storage medium. Background Art

[0002] The text-generated image model is a generative model based on deep learning, which can generate original and realistic images according to text descriptions.

[0003] In related technologies, users generate song cover images through text-based graph models. First, based on the basic information of the song, keywords used to represent the meaning of the song are extracted. Then, the keywords and quality words are combined and input into the text-based graph model. The text-based graph model generates an image corresponding to the input text. Multiple images generated by the model are manually screened to obtain a high-quality cover image.

[0004] However, the cover image obtained by the above method has a poor match with the lyrics. Summary of the invention

[0005] The embodiments of the present application provide an image generation method, apparatus, device and storage medium. The technical solutions provided by the embodiments of the present application are as follows:

[0006] According to one aspect of an embodiment of the present application, a method for generating an image is provided, the method comprising:

[0007] Get basic information about the song;

[0008] Acquire at least one target word corresponding to the basic information, wherein the target word is a word describing the style or content of the song;

[0009] generating a target image corresponding to the at least one target word according to the at least one target word by using a text graph model;

[0010] Obtaining a fused image according to the target image and the first cover image of the song, wherein the fused image includes a subject in the target image and a background in the first cover image;

[0011] A second cover image of the song is generated according to the fused image and the first cover image through a graph-to-graph model.

[0012] According to one aspect of an embodiment of the present application, there is provided an image generating device, the device comprising:

[0013] Information acquisition module, used to obtain basic information of songs;

[0014] A vocabulary acquisition module, used to acquire at least one target vocabulary corresponding to the basic information, wherein the target vocabulary is a vocabulary describing the style or content of the song;

[0015] A text image module, configured to generate a target image corresponding to the at least one target word according to the at least one target word through a text image model;

[0016] An image fusion module, used for obtaining a fused image according to the target image and the first cover image of the song, wherein the fused image includes a subject in the target image and a background in the first cover image;

[0017] A graph generation module is used to generate a second cover image of the song according to the fused image and the first cover image through a graph generation model.

[0018] According to one aspect of an embodiment of the present application, a computer device is provided, the computer device comprising a processor and a memory, the memory storing a computer program, the computer program being loaded and executed by the processor to implement the above-mentioned image generation method.

[0019] According to one aspect of an embodiment of the present application, a computer-readable storage medium is provided, in which a computer program is stored. The computer program is loaded and executed by a processor to implement the above-mentioned image generation method.

[0020] According to one aspect of an embodiment of the present application, a computer program product is provided. The computer program product includes a computer program. The computer program is loaded and executed by a processor to implement the above-mentioned image generation method.

[0021] The technical solution provided in the embodiments of the present application can bring the following beneficial effects:

[0022] By fusing the target image generated by the text graph model with the first cover image of the song, the obtained fused image includes the subject in the target image and the background in the first cover image, thereby regenerating the subject in the second cover image. Compared with the first cover image, the subject in the second cover image can be highly associated with the basic information of the song, thereby improving the relevance of the second cover image with the basic information of the song. When the basic information of the song includes lyrics, the second cover image is made more consistent with the textual meaning represented by the lyrics, thereby improving the relevance and matching degree of the second cover image with the lyrics. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 It is a schematic diagram of an implementation environment of a solution provided by an embodiment of the present application;

[0024] Figure 2 is a flow chart of an image generation method provided by an embodiment of the present application;

[0025] Figure 3 is a schematic diagram of a complete generation process of a cover image provided by an embodiment of the present application;

[0026] Figure 4 is a block diagram of an image generating device provided by an embodiment of the present application;

[0027] Figure 5 It is a structural block diagram of a computer device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0028] In order to make the objectives, technical solutions and advantages of the present application clearer, the implementation methods of the present application will be further described in detail below with reference to the accompanying drawings.

[0029] Please refer to Figure 1 , which shows a schematic diagram of a solution implementation environment provided by an embodiment of the present application. The solution implementation environment can be implemented as an image generation system. The solution implementation environment may include: a terminal device 10 and a server 20.

[0030] The number of terminal devices 10 can be one or more. The terminal device 10 can be an electronic device such as a mobile phone, a tablet computer, a laptop computer, a desktop computer, a game console, an e-book reader, a multimedia player, a wearable device, an intelligent voice interaction device, a smart home appliance, a vehicle terminal, an aircraft, etc.

[0031] The terminal device 10 may be installed with a client of a target application, and the target application has the function of generating a second cover image of a song based on basic information of the song, and the second cover image is different from the first cover image of the song. The present application does not limit the type of the target application, including but not limited to question-and-answer applications, music applications, video applications, game applications, web applications, etc. Optionally, the target application may be an application that needs to be downloaded and installed, or may be a click-to-use application, and the present application does not limit this.

[0032] The server 20 is used to provide background services for the client of the target application installed and running in the terminal device 10. For example, the server 20 can be the background server of the above-mentioned target application. The server 20 can be an independent physical server, or a server cluster composed of multiple servers, or a cloud computing service center. Optionally, the server 20 provides background services for the target applications in multiple terminal devices 10 at the same time. The terminal device 10 and the server 20 can communicate with each other through the network.

[0033] In an embodiment of the present application, a user can input at least one basic information of a song in a target application, and the terminal device sends the at least one basic information input by the user to a server. After receiving the at least one basic information, the server obtains other basic information of the song, and based on the basic information of the song, obtains at least one target word corresponding to the basic information, the target word being a word describing the style or content of the song, and generates at least one target image corresponding to the target word based on the at least one target word through a text-generated graph model. Then, based on the target image and the first cover image of the song, a fused image including the subject in the target image and the background in the first cover image is obtained, and a second cover image of the song is generated based on the fused image and the first cover image through a text-generated graph model. When the correlation between the second cover image and the song satisfies the threshold condition, the server returns the second cover image to the terminal device and displays the second cover image in the display interface of the terminal device.

[0034] Please refer to Figure 2 , which shows a flow chart of an image generation method provided by an embodiment of the present application. The execution subject of each step of the method may be a server. The method may include at least one of the following steps 210 to 250:

[0035] Step 210, obtaining basic information of the song.

[0036] The user can input at least one basic information of the song in the target application. The server obtains other basic information of the song based on the at least one basic information input by the user in the target application, and obtains the basic information of the song based on the at least one basic information sent by the terminal device and other basic information obtained by the server.

[0037] Optionally, the user can enter one basic information of the song in the target application, and the server obtains other basic information of the song based on the one basic information entered by the user, and obtains the basic information of the song. Optionally, the user can also enter multiple basic information of the song in the target application, and the server obtains other basic information of the song based on the multiple basic information entered by the user, and obtains the basic information of the song. Optionally, the user can also enter all basic information of the song in the target application, and the server receives all basic information entered by the user and obtains the basic information of the song.

[0038] In some embodiments, the basic information includes song identification information and song content information, the song identification information is used to label the song, and the song content information is used to indicate the content of the song. The song content information includes at least one of lyrics and song comments, and the song identification information includes at least one of song audio, song name, and singer name.

[0039] The song identification information is the tag information of the song, which is a unique attribute of the song. The song can be searched according to the song identification information. The song identification information includes at least one of the song audio, the song name and the singer name. For example, the song can be searched according to the song audio, the song name or the singer name.

[0040] The song content information is used to indicate the content of the song, and the song content information includes at least one of lyrics and song comment information. The lyrics are used to indicate the textual meaning represented by the song itself, and the song comment is used to indicate other people's understanding of the content of the song.

[0041] Exemplarily, the user can input at least one of the song identification information in the target application, and the server obtains other song identification information of the song and the song content information of the song according to the at least one song identification information input by the user to obtain the basic information of the song.

[0042] Step 220, obtaining at least one target word corresponding to the basic information, where the target word is a word that describes the style or content of the song.

[0043] At least one target word is a word obtained based on the basic information, and is not a word directly extracted from the basic information. The process of obtaining the target word can refer to the following embodiment, which is not introduced here.

[0044] If the target vocabulary is a vocabulary describing the style or content of a song, then at least one target vocabulary includes a vocabulary describing the style of a song and a vocabulary describing the content of a song.

[0045] Step 230: Generate a target image corresponding to at least one target word according to at least one target word by using a text-based graph model.

[0046] The text graph model here can be any publicly available diffusion model, such as a diffusion structure-based model trained with a large amount of data. The large amount of data can reach a sample level of more than 100 million, and this application does not limit this.

[0047] At least one target word is input into the text generation model, and a target image corresponding to the at least one target word is output. Optionally, the image resolution of the target image may be a default image resolution preset by the text generation model, for example, if the default image resolution is 720×1280, the image resolution of the target image generated by the text generation model is 720×1280.

[0048] In some embodiments, a target image corresponding to at least one target vocabulary is generated according to at least one target vocabulary and a quality vocabulary through a text graph model, and the quality vocabulary is used to indicate the image quality of the image to be generated.

[0049] The quality vocabulary is a vocabulary describing the image quality of the image to be generated. Optionally, the quality vocabulary may be a vocabulary describing the image resolution of the image to be generated, for example, the quality vocabulary is "720×1280". Optionally, the quality vocabulary may also be a vocabulary describing the image clarity of the image to be generated, for example, the quality vocabulary is "high definition".

[0050] At least one target word and a quality word are input into the text graph model, and a target image corresponding to the at least one target word is output. The image quality of the target image is the same as the image quality indicated by the quality word.

[0051] By adding quality vocabulary when generating a target image, the image quality of the target image can be adjusted through the quality vocabulary, thereby generating a high-quality target image and improving the image quality of the generated target image.

[0052] Step 240, obtaining a fused image based on the target image and the first cover image of the song, wherein the fused image includes the subject in the target image and the background in the first cover image.

[0053] The first cover image is the initial cover image of the song. The present application is used to generate a new cover image of the song according to the basic information of the song and the first cover image, that is, to generate a second cover image.

[0054] The target image includes a main area and a background area, and the first cover image also includes a main area and a background area. The main area refers to the area of ​​the picture occupied by the main body in the image, and the background area refers to the area of ​​the picture occupied by the background in the image.

[0055] The subject refers to the screen object that occupies the main screen composition in the image, and is also understood as the most eye-catching screen object in the image, which can focus the visual center of the viewer. The subject can contain only one screen object or multiple screen objects, and this application does not limit this. Optionally, the screen object can be a person or object that actually exists. For example, if the screen object is a person who actually exists, the screen object can be any person image that actually exists, including but not limited to ordinary people, actors, singers, athletes, entrepreneurs and other characters. If the screen object is an object that actually exists, the screen object can be any object image that actually exists, including but not limited to objects such as trees, flowers, buildings, vehicles, tables and chairs. Optionally, the screen object can also be a person or object that exists virtually. For example, if the screen object is a person who exists virtually, the screen object can be a game character, an animated character, or any person image created independently. If the screen object is an object that exists virtually, the screen object can be any object image created independently.

[0056] Optionally, if the subject contains only one screen object, the screen object contained in the subject may be any real person or object, or any virtual person or object. Optionally, if the subject contains multiple screen objects, the screen objects contained in the subject may all be real people or objects, or all be virtual people or objects, or may include at least one real person or object and at least one virtual person or object.

[0057] The background refers to other objects in the image except the subject, that is, the background area is other areas in the image except the subject area.

[0058] Exemplarily, a subject image corresponding to the subject in the target image is obtained, and a background image corresponding to the background in the first cover image is obtained, wherein the subject image is an image corresponding to the subject area in the target image, and the background image is an image corresponding to the background area in the first cover image. The subject image corresponding to the subject in the target image and the background image corresponding to the background in the first cover image are fused to obtain a fused image. The fused image includes the subject in the target image and the background in the first cover image, that is, the fused image includes the subject area in the target image and the background area in the first cover image.

[0059] Step 250, generating a second cover image of the song according to the fused image and the first cover image through a graph-to-graph model.

[0060] The graph-generated graph model here can be any publicly available diffusion model, such as a diffusion structure-based model trained with a large amount of data. The large amount of data can reach a sample level of more than 100 million, and this application does not limit this.

[0061] The fused image and the first cover image are input into the graph generation model to output a second cover image of the song. Optionally, the image resolution of the second cover image can be a default image resolution preset by the graph generation model. For example, if the default image resolution is 720×1280, the image resolution of the second cover image generated by the graph generation model is 720×1280.

[0062] In some embodiments, a second cover image of the song is generated according to the fused image, the first cover image and the quality vocabulary by using a graph-generated graph model. The fused image, the first cover image and the quality vocabulary are input into the graph-generated graph model, and a second cover image is output, wherein the image quality of the second cover image is the same as the image quality indicated by the quality vocabulary.

[0063] It should be noted that the quality vocabulary added by the image-generated image model to generate the second cover image and the quality vocabulary added by the text-generated image model to generate the target image can be the same or different. If the quality vocabulary added by the image-generated image model to generate the second cover image and the quality vocabulary added by the text-generated image model to generate the target image are different, then the image qualities indicated by the two quality vocabulary can be the same or different.

[0064] The technical solution provided by the embodiment of the present application realizes the regeneration of the subject in the second cover image by fusing the target image generated by the text graph model with the first cover image of the song, so that the obtained fused image includes the subject in the target image and the background in the first cover image. Compared with the first cover image, the subject in the second cover image can be highly associated with the basic information of the song, thereby improving the relevance of the second cover image with the basic information of the song. In the case where the basic information of the song includes lyrics, the second cover image is made more consistent with the text meaning represented by the lyrics, thereby improving the relevance and matching degree of the second cover image with the lyrics.

[0065] In some embodiments, step 220 includes at least one of sub-steps 221 - 224 .

[0066] Sub-step 221, obtaining at least one style description word according to the song identification information.

[0067] The song identification information includes at least one of the song audio, the song name and the singer name. The style description vocabulary is a vocabulary used to describe the style of the song. At least one style description vocabulary is a vocabulary obtained based on the song identification information, and is not a vocabulary directly extracted from the song identification information.

[0068] In some embodiments, sub-step 221 includes at least one of sub-steps 2211 to 2215. In the case where the song identification information includes song audio, sub-step 221 includes sub-steps 2211 to 2212. In the case where the song identification information includes the song title and the singer name, sub-step 221 includes sub-steps 2213 to 2214. In the case where the song identification information includes song audio, song title and singer name, sub-step 221 includes sub-steps 2211 to 2215.

[0069] Sub-step 2211, obtaining the audio description text of the song audio based on the song audio through the multimodal understanding model.

[0070] The multimodal understanding model here can be any publicly available large language model, such as a natural language model based on a transformer structure obtained by training with a large amount of data, such as a large amount of data that can reach a sample level of more than 100 million, which is not limited in this application. The multimodal understanding model is used to extract the information content of the input information in the form of text, and the modality of the input information includes but is not limited to video modality, image modality and audio modality.

[0071] The song audio is input into the multimodal understanding model, and the audio description text of the song audio is output. The audio description text is a text used to describe the audio content of the song audio.

[0072] Sub-step 2212, obtaining style description vocabulary in the audio description text based on the audio description text through the large language model.

[0073] The large language model here can be any publicly available large language model, such as a natural language model based on a transformer structure trained with a large amount of data. The large amount of data can reach a sample level of more than 100 million, and this application does not limit this.

[0074] Exemplarily, the style description vocabulary in the audio description text is obtained by the large language model according to the audio description text and the first prompt information. The first prompt information is used to prompt the large language model to extract vocabulary describing the style of the song in the audio description text. Exemplarily, the first prompt information may be "Please extract vocabulary based on the audio description text, and the extracted vocabulary focuses on describing the style of the song."

[0075] Sub-step 2213, using the large language model to obtain the story text corresponding to the song title and singer name based on the song title and singer name, the story text corresponding to the song title and singer name is used to completely describe the song style represented by the song title and singer name.

[0076] Exemplarily, the story text corresponding to the song name and the singer name is obtained through the large language model according to the song name, the singer name and the second prompt information. The second prompt information is used to prompt the generation of the story text corresponding to the song name and the singer name. The story text corresponding to the song name and the singer name is a complete story text obtained by understanding the song name and the singer name again, and the story text corresponding to the song name and the singer name is used to fully describe the song style represented by the song name and the singer name.

[0077] Normally, there is no character limit for the story text corresponding to the song title and singer name generated by the large language model, but in some embodiments, the second prompt information can limit the number of characters in the story text generated by the large language model to obtain the story text corresponding to the song title and singer name with a limited number of characters.

[0078] It should be noted that if a large language model is used multiple times based on the song names and singer names, multiple story texts corresponding to the song names and singer names can be obtained, and there are differences between the story texts corresponding to the multiple song names and singer names. However, the story emotions to be expressed in the story texts corresponding to each song name and singer name are the same or similar, and this part of the difference is within the expected controllable range of this application.

[0079] Sub-step 2214, using the large language model, based on the story text corresponding to the song title and the singer name, obtains the style description vocabulary in the story text corresponding to the song title and the singer name.

[0080] Exemplarily, the style description vocabulary in the story text corresponding to the song name and the singer name is obtained through the large language model according to the story text corresponding to the song name and the singer name, and the third prompt information. The third prompt information is used to prompt the large language model to extract the vocabulary describing the style of the song in the story text corresponding to the song name and the singer name. Exemplarily, the third prompt information can be "Please extract vocabulary based on the story text corresponding to the song name and the singer name, and the extracted vocabulary focuses on describing the style of the song."

[0081] Sub-step 2215, obtaining at least one style description vocabulary according to the style description vocabulary in the audio description text and the style description vocabulary in the story text corresponding to the song title and the singer name.

[0082] The style description words in the audio description text and the style description words in the story text corresponding to the song name and the singer name are fused, and the same words are removed to obtain at least one style description word.

[0083] By converting song audio into audio description text to extract style description vocabulary, and converting song names and singer names into story text to extract style description vocabulary, the difficulty of extracting style description vocabulary is reduced, and the vocabulary describing style can be captured more accurately, so that the style description vocabulary can be used to generate the target image, which can improve the accuracy of the target image.

[0084] In some embodiments, if the song identification information only includes song audio, at least one style description word is obtained based on the style description words in the audio description text.

[0085] In some embodiments, if the song identification information includes the song title and singer name but does not include the song audio, at least one style description word is obtained based on the style description words in the story text corresponding to the song title and singer name.

[0086] In some embodiments, if the song identification information includes the song audio and the song title, but does not include the singer's name, the story text corresponding to the song title is obtained based on the song title through the large language model, and the story text corresponding to the song title is used to fully describe the song style represented by the song title. The style description vocabulary in the story text corresponding to the song title is obtained through the large language model based on the story text corresponding to the song title. At least one style description vocabulary is obtained based on the style description vocabulary in the audio description text and the style description vocabulary in the story text corresponding to the song title.

[0087] In some embodiments, if the song identification information includes the song audio and the singer name, but does not include the song name, the story text corresponding to the singer name is obtained based on the singer name through the large language model, and the story text corresponding to the singer name is used to fully describe the song style represented by the singer name. The style description vocabulary in the story text corresponding to the singer name is obtained through the large language model based on the story text corresponding to the singer name. At least one style description vocabulary is obtained based on the style description vocabulary in the audio description text and the style description vocabulary in the story text corresponding to the singer name.

[0088] Sub-step 222, using a large language model to obtain a story text corresponding to the song content information based on the song content information, and the story text corresponding to the song content information is used to completely describe the song content in the song content information.

[0089] Exemplarily, the story text corresponding to the song content information is obtained through the large language model according to the song content information and the fourth prompt information. The fourth prompt information is used to prompt the generation of the story text corresponding to the song content information. The story text corresponding to the song content information is a complete story text obtained by understanding the song content information again, and the story text corresponding to the song content information is used to completely describe the song content in the song content information.

[0090] If the song content information only includes lyrics, the story text corresponding to the song content information is the story text corresponding to the lyrics, and the story text corresponding to the lyrics is used to fully describe the song content contained in the lyrics. If the song content information only includes song reviews, the story text corresponding to the song content information is the story text corresponding to the song reviews, and the story text corresponding to the song reviews is used to fully describe the song content contained in the song reviews. If the song content information includes lyrics and song reviews, the story text corresponding to the song content information is the story text corresponding to the lyrics and song reviews, and the story text corresponding to the lyrics and song reviews is used to fully describe the song content contained in the lyrics and song reviews.

[0091] Normally, there is no limit on the number of characters in the story text corresponding to the song content information generated by the large language model, but in some embodiments, the fourth prompt information can limit the number of characters in the story text generated by the large language model to obtain a story text corresponding to the song content information with a limited number of characters.

[0092] It should be noted that if a large language model is used multiple times based on the song content information, multiple story texts corresponding to the song content information can be obtained, and there are differences between the story texts corresponding to the multiple song content information. However, the story content to be expressed in the story text corresponding to each song content information is the same or similar, and this part of the difference is within the expected controllable range of this application.

[0093] Sub-step 223, obtaining at least one content description word according to the story text corresponding to the song content information through the large language model.

[0094] Exemplarily, at least one content description word is obtained through the large language model according to the story text corresponding to the song content information and the fifth prompt information. The fifth prompt information is used to prompt the large language model to extract words describing the song content in the story text corresponding to the song content information. Exemplarily, the fifth prompt information may be "Please extract words based on the story text corresponding to the song content information, and the extracted words focus on describing the content of the song."

[0095] It should be noted that the content description vocabulary here can also be understood as the entity words of the story text corresponding to the song content information, that is, through the large language model, at least one entity word in the story text corresponding to the song content information is extracted, and the entity word can be a noun, adjective, pronoun or other vocabulary.

[0096] Sub-step 224, obtaining at least one target word according to at least one style description word and at least one content description word.

[0097] At least one style description word and at least one content description word are combined to obtain at least one target word.

[0098] By extracting style description words based on song identification information, and extracting content description words based on the story text corresponding to the song content information, the obtained target words can describe the style and content of the song more comprehensively and accurately, thereby generating a target image based on the target words, which can improve the accuracy of the target image and the relevance between the target image and the basic information, thereby helping to improve the relevance between the second cover image and the basic information.

[0099] In some embodiments, step 240 includes at least one of sub-steps 241 - 244 .

[0100] Sub-step 241 , segmenting the subject and the background in the target image to obtain a subject image corresponding to the subject in the target image.

[0101] The subject and background in the target image are segmented by the image segmentation model to obtain the subject area and the background area in the target image. According to the subject area in the target image, a subject image corresponding to the subject in the target image is obtained. The subject image is an image corresponding to the subject area in the target image.

[0102] The image segmentation model used can be any publicly available image segmentation model, and this application does not limit this.

[0103] Sub-step 242, segmenting the subject and the background in the first cover image to obtain a background image corresponding to the background in the first cover image.

[0104] The subject and background in the first cover image are segmented by an image segmentation model to obtain a subject region and a background region in the first cover image. According to the background region in the first cover image, a background image corresponding to the background in the first cover image is obtained. The background image is an image corresponding to the background region in the first cover image.

[0105] Sub-step 243, filling the main body area in the background image with the first color to obtain a filled background image.

[0106] The subject is removed from the background image, so the subject area in the background image is vacant, and the subject area in the background image is filled with the first color to obtain a filled background image. The filled background image includes the background area in the first cover image and the subject area showing the first color.

[0107] The present application does not limit the setting of the first color, which can be any color. Exemplarily, the first color is white.

[0108] Sub-step 244, obtaining a fused image according to the main image and the filled background image.

[0109] The main image and the filled background image are fused, and the main area corresponding to the main image is filled into the area of ​​the first color in the filled background image to obtain a fused image.

[0110] By fusing the subject in the target image and the background in the first cover image, the subject area in the obtained fused image is highly correlated with the basic information, so that a second cover image matching the basic information can be generated based on the fused image, thereby improving the correlation and matching degree between the second cover image and the basic information.

[0111] In some embodiments, step 250 includes at least one of sub-steps 251 - 254 .

[0112] Sub-step 251, extracting features of the first cover image through a neural network model to obtain depth features and edge features of the first cover image, wherein the depth features are used to indicate image content in the first cover image, and the edge features are used to indicate locally discontinuous image features in the first cover image.

[0113] The neural network model is a model used to extract features. The neural network model used here can be any publicly available neural network model, and this application does not limit this.

[0114] The depth features and edge features are used to represent the image features of the first cover image.

[0115] Sub-step 252, generating a secondary redrawn image of the first cover image according to the depth features and edge features by using a stable diffusion model.

[0116] The stable diffusion model may be any publicly available diffusion model, which is not limited in the present application. For example, the stable diffusion model may be a SD (Stable Diffusion) model.

[0117] The depth features and edge features are input into the stable diffusion model, and a secondary redrawn image of the first cover image is output, where the secondary redrawn image is an image regenerated based on the depth features and the edge features, and there is a difference between the secondary redrawn image and the first cover image. Optionally, the image resolution of the secondary redrawn image can be a default image resolution preset by the stable diffusion model, for example, if the default image resolution is 720×1280, then the image resolution of the secondary redrawn image generated by the stable diffusion model is 720×1280.

[0118] In some embodiments, a second redrawn image of the first cover image is obtained by a stable diffusion model based on the depth features, edge features and quality vocabulary, and the quality vocabulary is used to indicate the image quality of the image to be generated. The depth features, edge features and quality vocabulary are input into the stable diffusion model, and a second redrawn image is output, and the image quality of the second redrawn image is the same as the image quality indicated by the quality vocabulary.

[0119] Sub-step 253, generating a third-order redrawing image of the first cover image according to the second-order redrawing image by using a stable diffusion model.

[0120] The tertiary redrawn image is an image regenerated from the secondary redrawn image, and there are differences between the tertiary redrawn image, the secondary redrawn image, and the first cover image. Optionally, the image resolution of the tertiary redrawn image may be a default image resolution preset by the stable diffusion model, for example, if the default image resolution is 720×1280, then the image resolution of the tertiary redrawn image generated by the stable diffusion model is 720×1280.

[0121] In some embodiments, a tertiary redrawn image of the cover image is generated based on the secondary redrawn image and the quality vocabulary by a stable diffusion model. The image quality of the tertiary redrawn image is the same as the image quality indicated by the quality vocabulary.

[0122] In some embodiments, multiple redrawn images of the first cover image may be generated based on the three redrawn images using a stable diffusion model, and the multiple redrawn images may be used to generate the second cover image.

[0123] Sub-step 254, generating a second cover image according to the fused image and the three-time redrawn image by using the image-to-image model.

[0124] The fused image and the three-times redrawn image are input into the image generation model, and the second cover image is output.

[0125] In some embodiments, a second cover image is generated by a graph-based model based on the fused image, the three-times redrawn image, and the quality vocabulary.

[0126] It should be noted that the quality vocabulary added when the image generation model generates an image and the quality vocabulary added when the above-mentioned stable diffusion model and the Wensheng image model generate an image can be the same or different, and this application does not limit this. If the quality vocabulary added when the image generation model generates an image and the quality vocabulary added when the above-mentioned stable diffusion model and the Wensheng image model generate an image are different, the image quality indicated by each quality vocabulary can be the same or different.

[0127] By redrawing the first cover image, the obtained redrawn image can more accurately represent the image features of the cover image, so as to generate a second cover image based on the fused image and the redrawn image, which can improve the image quality of the second cover image. Moreover, by adding quality vocabulary in the process of generating the image, the image quality of the second cover image can be adjusted by the quality vocabulary, thereby further improving the image quality of the second cover image.

[0128] In some embodiments, step 250 is followed by step 260 , and step 260 includes at least one of sub-steps 261 - 265 .

[0129] Sub-step 261, obtaining an image description text of the second cover image according to the second cover image through a multimodal understanding model.

[0130] The image description text is text for describing the image content of the second cover image.

[0131] Sub-step 262, obtaining the similarity between the image description text and the lyrics through a text matching model based on the image description text and the lyrics.

[0132] The text matching model is used to calculate the semantic similarity between two texts. The text matching model here can be any publicly available text matching model, and this application does not limit this.

[0133] Sub-step 263, when the similarity is greater than or equal to the first threshold, determining the second cover image as the cover image to be finally output.

[0134] When the similarity is greater than or equal to the first threshold, it can be considered that the second cover image is related to the lyrics, so the second cover image is determined as the final output image. The specific value of the first threshold is set by the technician according to the text matching requirements, and this application does not limit it.

[0135] By determining the relevance of the second cover image to the lyrics, the image finally outputted is a cover image highly relevant to the lyrics, thereby ensuring the relevance and matching degree of the image finally outputted to the lyrics.

[0136] Sub-step 264, when the similarity is less than the first threshold, adjusting the weight of at least one target word according to the difference between the image description text and the lyrics, to obtain the weight corresponding to the at least one target word.

[0137] When the similarity is less than the first threshold, it can be considered that the second cover image is irrelevant to the lyrics, so the second cover image needs to be regenerated.

[0138] The weight of at least one target word is initially set to 1, that is, the weight of each target word is the same. For example, if at least one target word includes "at the seaside" and "a girl", the initial weights of "at the seaside" and "a girl" are 1:1. According to the difference between the image description text and the lyrics, the weight of at least one target word is adjusted to obtain the weight corresponding to at least one target word. For example, if there are more descriptions about the seaside in the lyrics, and less descriptions about the seaside in the image description text, the weight of "at the seaside" can be increased, and the weights of "at the seaside" and "a girl" can be adjusted to 1.5:1. If there are more descriptions about girls in the lyrics, and less descriptions about girls in the image description text, the weight of "a girl" can be increased, and the weights of "at the seaside" and "a girl" can be adjusted to 1:1.5.

[0139] Sub-step 265, based on the weights corresponding to at least one target word, restart from the step of generating a target image corresponding to at least one target word through the text graph model according to at least one target word, to obtain a regenerated second cover image.

[0140] According to the weights corresponding to at least one target word, the process starts again from step 230, and generates a regenerated target image according to at least one target word and the weights corresponding to at least one target word through the text-generated graph model. A regenerated fused image is obtained according to the regenerated target image and the first cover image of the song, and the regenerated fused image includes the subject in the regenerated target image and the background in the first cover image. A regenerated second cover image is generated according to the regenerated fused image and the first cover image through the text-generated graph model.

[0141] By adjusting the weight of the target vocabulary, the second cover image is regenerated, the relevance of the regenerated second cover image to the lyrics is improved, and the cover image related to the lyrics is obtained as much as possible before the upper limit of the regeneration times is reached, thereby ensuring the relevance and matching degree of the final output image to the lyrics.

[0142] In some embodiments, when the number of times the second cover image is regenerated reaches a second threshold and the similarity corresponding to each regenerated second cover image is less than the first threshold, the regenerated second cover image corresponding to the maximum value of the similarities corresponding to the regenerated second cover images is determined as the final output cover image.

[0143] The specific value of the second threshold is set by the technician according to the text matching requirements, and this application does not limit it. For example, if the second threshold is 5, then when the number of times the second cover image is regenerated reaches 5 times, and the similarities corresponding to the 5 regenerated second cover images are all less than the first threshold, the regenerated second cover image corresponding to the maximum value of the similarities corresponding to the 5 regenerated second cover images is determined as the final output cover image.

[0144] By setting an upper limit on the number of times the regenerated image is generated, the similarity corresponding to the regenerated second cover image is avoided to be always less than the first threshold, and the generation step is repeatedly executed. This saves computing resources while ensuring the quality of image generation and ensures the efficiency of image generation.

[0145] Figure 3The schematic diagram of the complete generation process of the cover image is shown. The basic information of the song includes the song audio, song name, singer name, lyrics and song comments. For the song audio, the audio description text of the song audio is extracted by the multimodal understanding model, and the style description vocabulary in the audio description text is extracted by the large language model. For the song name and singer name, the story text corresponding to the song name and singer name is generated by the large language model, and the style description vocabulary in the story text corresponding to the song name and singer name is extracted by the large language model. The style description vocabulary in the audio description text and the style description vocabulary in the story text corresponding to the song name and singer name are fused to obtain at least one style description vocabulary. For the lyrics and song comments, the story text corresponding to the song content information is generated by the large language model, and at least one entity word in the story text corresponding to the song content information is extracted by the large language model. At least one style description vocabulary and at least one entity word are input into the text graph model to generate a target image, and the target image is segmented to obtain a main image corresponding to the target image. For the first cover image of the song, the main area in the first cover image is removed, and the main area is filled with color to obtain the background image of the first cover image, and the main image corresponding to the target image and the background image of the first cover image are fused to obtain a fused image. Moreover, for the first cover image of the song, it is also necessary to extract the depth features and edge features of the first cover image through a neural network model, generate a secondary redrawing image of the first cover image through a stable diffusion model, and generate a tertiary redrawing image of the first cover image again through a stable diffusion model. Finally, the second cover image is generated according to the fused image and the tertiary redrawing image through a graph-to-graph model. After the second cover image is generated, the lyrics relevance of the second cover image is determined. If the second cover image is related to the lyrics, the second cover image is determined as the final output cover image, and the image generation process is terminated. If the second cover image is not related to the lyrics, the weight of the vocabulary is adjusted according to the difference between the second cover image and the lyrics, and the second cover image is regenerated. Until the number of regenerations reaches the upper limit, the regenerated second cover image corresponding to the maximum value of the similarities corresponding to the regenerated second cover images is determined as the final output cover image.

[0146] The following is an embodiment of the device of the present application, which can be used to execute the embodiment of the method of the present application. For details not disclosed in the embodiment of the device of the present application, please refer to the embodiment of the method of the present application.

[0147] Please refer to Figure 4 , which shows a block diagram of an image generation device provided by an embodiment of the present application. The device has the function of implementing the above-mentioned image generation method, and the function can be implemented by hardware, or by hardware executing corresponding software. The device can be the server introduced above, or it can be set in the server. Figure 4 As shown, the device 400 may include: an information acquisition module 410 , a vocabulary acquisition module 420 , a text-to-image module 430 , an image fusion module 440 and a picture-to-image module 450 .

[0148] The information acquisition module 410 is used to acquire basic information of the song.

[0149] The vocabulary acquisition module 420 is used to acquire at least one target vocabulary corresponding to the basic information, where the target vocabulary is a vocabulary that describes the style or content of the song.

[0150] The text graph module 430 is configured to generate a target image corresponding to the at least one target word according to the at least one target word by using a text graph model.

[0151] The image fusion module 440 is used to obtain a fused image based on the target image and the first cover image of the song, wherein the fused image includes the subject in the target image and the background in the first cover image.

[0152] The image generation module 450 is used to generate a second cover image of the song according to the fused image and the first cover image through a image generation model.

[0153] In some embodiments, the basic information includes song identification information and song content information, the song identification information is used to label the song, the song content information is used to indicate the content of the song, and the song content information includes at least one of lyrics and song comments; the vocabulary acquisition module 420 is used to:

[0154] Obtaining at least one style description word according to the song identification information;

[0155] Obtaining a story text corresponding to the song content information through a large language model according to the song content information, wherein the story text corresponding to the song content information is used to completely describe the song content in the song content information;

[0156] Obtaining at least one content description word according to the story text corresponding to the song content information through the large language model;

[0157] The at least one target vocabulary is obtained according to the at least one style description vocabulary and the at least one content description vocabulary.

[0158] In some embodiments, the song identification information includes at least one of the song audio, the song title and the singer name; the vocabulary acquisition module 420 is used to:

[0159] Obtaining an audio description text of the song audio according to the song audio through a multimodal understanding model;

[0160] Obtaining style description vocabulary in the audio description text according to the audio description text by using the large language model;

[0161] Obtaining a story text corresponding to the song title and the singer name through the large language model according to the song title and the singer name, wherein the story text corresponding to the song title and the singer name is used to completely describe the song style represented by the song title and the singer name;

[0162] Obtaining, by means of the large language model, style description words in the story text corresponding to the song title and the singer name;

[0163] The at least one style description vocabulary is obtained according to the style description vocabulary in the audio description text and the style description vocabulary in the story text corresponding to the song name and the singer name.

[0164] In some embodiments, the image fusion module 440 is used to:

[0165] Segmenting the subject and the background in the target image to obtain a subject image corresponding to the subject in the target image;

[0166] Segmenting the subject and the background in the first cover image to obtain a background image corresponding to the background in the first cover image;

[0167] Filling a main body area in the background image with a first color to obtain a filled background image;

[0168] The fused image is obtained according to the main image and the filled background image.

[0169] In some embodiments, the image generation module 450 is used to:

[0170] Extracting features of the first cover image through a neural network model to obtain depth features and edge features of the first cover image, wherein the depth features are used to indicate image content in the first cover image, and the edge features are used to indicate local discontinuous image features in the first cover image;

[0171] generating a secondary redrawing image of the first cover image according to the depth feature and the edge feature by using a stable diffusion model;

[0172] generating a third-order redrawing image of the first cover image according to the second-order redrawing image by using the stable diffusion model;

[0173] The second cover image is generated by the image-generated model according to the fused image and the three-times redrawn image.

[0174] In some embodiments, the image generation module 450 is used to:

[0175] Obtaining a secondary redrawn image of the first cover image by using the stable diffusion model according to the depth feature, the edge feature and a quality vocabulary, wherein the quality vocabulary is used to indicate the image quality of the image to be generated;

[0176] generating a tertiary redrawing image of the cover image according to the secondary redrawing image and the quality vocabulary by using the stable diffusion model;

[0177] The second cover image is generated by the image-generated model according to the fused image, the three-times redrawn image and the quality vocabulary.

[0178] In some embodiments, the apparatus 400 further includes a lyrics matching module, wherein the lyrics matching module is configured to:

[0179] Obtaining an image description text of the second cover image according to the second cover image by using a multimodal understanding model;

[0180] Obtaining the similarity between the image description text and the lyrics through a text matching model according to the image description text and the lyrics;

[0181] When the similarity is greater than or equal to the first threshold, the second cover image is determined as the cover image to be finally output.

[0182] In some embodiments, the lyrics matching module is used to:

[0183] When the similarity is less than the first threshold, adjusting the weight of the at least one target word according to the difference between the image description text and the lyrics, to obtain the weight corresponding to the at least one target word;

[0184] According to the weights respectively corresponding to the at least one target word, the step of generating a target image corresponding to the at least one target word through the text graph model is restarted to obtain a regenerated second cover image.

[0185] In some embodiments, the lyrics matching module is used to:

[0186] When the number of times the second cover image is regenerated reaches a second threshold and the similarity corresponding to each regenerated second cover image is less than the first threshold, the regenerated second cover image corresponding to the maximum value of the similarities corresponding to the regenerated second cover images is determined as the final output cover image.

[0187] It should be noted that the device provided in the above embodiment, when implementing its functions, is only illustrated by the division of the above functional modules. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the content structure of the device is divided into different functional modules to complete all or part of the functions described above. In addition, the device and method embodiments provided in the above embodiment belong to the same concept, and the specific implementation process is detailed in the method embodiment, which will not be repeated here.

[0188] Please refer to Figure 5 , which shows a block diagram of a computer device 500 provided in one embodiment of the present application. The computer device 500 may be any electronic device with data calculation, processing and storage functions. The computer device 500 may be used to implement the image generation method provided in the above embodiment.

[0189] Typically, the computer device 500 includes a processor 501 and a memory 502 .

[0190] The processor 501 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 501 may be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field Programmable Gate Array), and PLA (Programmable Logic Array). The processor 501 may also include a main processor and a coprocessor. The main processor is a processor for processing data in an awake state, also known as a CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in a standby state. In some embodiments, the processor 501 may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 501 may also include an AI processor, which is used to process computing operations related to machine learning.

[0191] The memory 502 may include one or more computer-readable storage media, which may be non-transitory. The memory 502 may also include a high-speed random access memory, and a non-volatile memory, such as one or more disk storage devices, flash memory storage devices. In some embodiments, the non-transitory computer-readable storage medium in the memory 502 is used to store a computer program, which is configured to be executed by one or more processors to implement the above-mentioned image generation method.

[0192] Those skilled in the art will understand that Figure 5 The structure shown in the figure does not constitute a limitation on the computer device 500, and the computer device 500 may include more or less components than those shown in the figure, or combine some components, or adopt a different arrangement of components.

[0193] In an illustrative embodiment, a computer-readable storage medium is also provided, wherein a computer program is stored in the storage medium, and when the computer program is executed by a processor of a computer device, the image generation method is implemented. Optionally, the computer-readable storage medium may be a ROM (Read-Only Memory), a RAM (Random Access Memory), a CD-ROM (Compact Disc Read-Only Memory), a magnetic tape, a floppy disk, an optical data storage device, etc.

[0194] In an exemplary embodiment, a computer program product is also provided, the computer program product comprising a computer program, the computer program being stored in a computer-readable storage medium. A processor of a computer device reads the computer program from the computer-readable storage medium, and the processor executes the computer program, so that the computer device executes the above-mentioned image generation method.

[0195] It should be understood that the "multiple" mentioned in this article refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. In addition, the step numbers described in this article only illustrate a possible execution sequence between the steps. In some other embodiments, the above steps may not be executed in the order of the numbers, such as two steps with different numbers are executed at the same time, or two steps with different numbers are executed in the opposite order to the diagram. The embodiments of the present application are not limited to this.

[0196] The above description is only an exemplary embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. An image generation method, characterized in that: The method comprises: Get basic information about the song; Acquire at least one target word corresponding to the basic information, wherein the target word is a word describing the style or content of the song; generating a target image corresponding to the at least one target word according to the at least one target word by using a text graph model; Obtaining a fused image according to the target image and the first cover image of the song, wherein the fused image includes a subject in the target image and a background in the first cover image; A second cover image of the song is generated according to the fused image and the first cover image through a graph-to-graph model.

2. The method according to claim 1, characterized in that The basic information includes song identification information and song content information, wherein the song identification information is used to label the song, and the song content information is used to indicate the content of the song, and the song content information includes at least one of lyrics and song comments; The acquiring at least one target word corresponding to the basic information includes: Obtaining at least one style description word according to the song identification information; Obtaining a story text corresponding to the song content information through a large language model according to the song content information, wherein the story text corresponding to the song content information is used to completely describe the song content in the song content information; Obtaining at least one content description word according to the story text corresponding to the song content information through the large language model; The at least one target vocabulary is obtained according to the at least one style description vocabulary and the at least one content description vocabulary.

3. The method according to claim 2, characterized in that The song identification information includes at least one of song audio, song name and singer name; The step of obtaining at least one style description word according to the song identification information includes: Obtaining an audio description text of the song audio according to the song audio through a multimodal understanding model; Obtaining style description vocabulary in the audio description text according to the audio description text by using the large language model; Obtaining a story text corresponding to the song title and the singer name through the large language model according to the song title and the singer name, wherein the story text corresponding to the song title and the singer name is used to completely describe the song style represented by the song title and the singer name; Obtaining, by means of the large language model, style description words in the story text corresponding to the song title and the singer name; The at least one style description vocabulary is obtained according to the style description vocabulary in the audio description text and the style description vocabulary in the story text corresponding to the song name and the singer name.

4. The method according to claim 1, characterized in that: The step of obtaining a fused image according to the target image and the first cover image of the song includes: Segmenting the subject and the background in the target image to obtain a subject image corresponding to the subject in the target image; Segmenting the subject and the background in the first cover image to obtain a background image corresponding to the background in the first cover image; Filling a main body area in the background image with a first color to obtain a filled background image; The fused image is obtained according to the main image and the filled background image.

5. The method according to claim 1, characterized in that The step of generating a second cover image of the song according to the fused image and the first cover image by using a graph-to-graph model includes: Extracting features of the first cover image through a neural network model to obtain depth features and edge features of the first cover image, wherein the depth features are used to indicate image content in the first cover image, and the edge features are used to indicate local discontinuous image features in the first cover image; generating a secondary redrawing image of the first cover image according to the depth feature and the edge feature by using a stable diffusion model; generating a third-order redrawing image of the first cover image according to the second-order redrawing image by using the stable diffusion model; The second cover image is generated by the image-generated model according to the fused image and the three-times redrawn image.

6. The method according to claim 5, characterized in that The step of generating a secondary redrawing image of the first cover image according to the depth feature and the edge feature by using a stable diffusion model includes: Obtaining a secondary redrawing image of the first cover image by using the stable diffusion model according to the depth feature, the edge feature and a quality vocabulary, wherein the quality vocabulary is used to indicate the image quality of the image to be generated; The step of generating a third redrawn image of the first cover image according to the second redrawn image by using the stable diffusion model includes: generating a tertiary redrawing image of the cover image according to the secondary redrawing image and the quality vocabulary by using the stable diffusion model; The step of generating the second cover image according to the fused image and the three-times redrawn image by using the image-generated image model includes: The second cover image is generated by the image-generated model according to the fused image, the three-times redrawn image and the quality vocabulary.

7. The method according to any one of claims 1 to 6, characterized in that: After generating the second cover image of the song according to the fused image and the first cover image by using the graph-to-graph model, the method further includes: Obtaining an image description text of the second cover image according to the second cover image by using a multimodal understanding model; Obtaining the similarity between the image description text and the lyrics through a text matching model according to the image description text and the lyrics; When the similarity is greater than or equal to the first threshold, the second cover image is determined as the cover image to be finally output.

8. The method according to claim 7, characterized in that The method further comprises: When the similarity is less than the first threshold, adjusting the weight of the at least one target word according to the difference between the image description text and the lyrics, to obtain the weight corresponding to the at least one target word; According to the weights respectively corresponding to the at least one target word, the step of generating a target image corresponding to the at least one target word through the text graph model is restarted to obtain a regenerated second cover image.

9. The method according to claim 8, characterized in that The method further comprises: When the number of times the second cover image is regenerated reaches a second threshold and the similarity corresponding to each regenerated second cover image is less than the first threshold, the regenerated second cover image corresponding to the maximum value of the similarities corresponding to the regenerated second cover images is determined as the final output cover image.

10. An image generating device, characterized in that: The device comprises: Information acquisition module, used to obtain basic information of songs; A vocabulary acquisition module, used to acquire at least one target vocabulary corresponding to the basic information, wherein the target vocabulary is a vocabulary describing the style or content of the song; A text image module, configured to generate a target image corresponding to the at least one target word according to the at least one target word through a text image model; An image fusion module, used for obtaining a fused image according to the target image and the first cover image of the song, wherein the fused image includes a subject in the target image and a background in the first cover image; A graph generation module is used to generate a second cover image of the song according to the fused image and the first cover image through a graph generation model.

11. A computer device, characterized in that: The computer device comprises a processor and a memory, wherein a computer program is stored in the memory, and the computer program is loaded and executed by the processor to implement the image generating method according to any one of claims 1 to 9.

12. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and the computer program is loaded and executed by a processor to implement the image generating method according to any one of claims 1 to 9.

13. A computer program product, characterized in that The computer program product comprises a computer program, which is loaded and executed by a processor to implement the image generation method according to any one of claims 1 to 9.

Images