Image generation method and device

Abstract

The invention discloses an image generation method and device. Belongs to the technical field of artificial intelligence. The method comprises the steps of obtaining a first input text, wherein the first input text comprises first description information; under the condition that it is determined that the first model is used for image generation based on the first input text, the first model is loaded, the first model is obtained by retraining a text generation graph model through a user-defined sample set, and the user-defined sample set comprises user-defined sample images and sample description information corresponding to the sample images; and inputting the first input text into the first model to obtain a first image which is output by the first model and is matched with the first description information.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of artificial intelligence technology, and specifically to an image generation method and device thereof. Background Art

[0002] With the rapid development of artificial intelligence technology, AIGC (Artificial Intelligence Generated Content) has become a hot topic. Among them, text-to-image technology, as an important branch of AIGC, can transform text descriptions into vivid pictures.

[0003] In the prior art, a general text-generated graph model is usually used for image generation. However, the general text-generated graph model cannot match the personalized description in the user's text, resulting in that the generated image cannot accurately match the description information input by the user, and thus the accuracy of the generated image is low. Summary of the invention

[0004] The purpose of the embodiments of the present application is to provide an image generation method and device thereof, which can improve the accuracy of images generated by a model.

[0005] In a first aspect, an embodiment of the present application provides an image generation method, the method comprising: obtaining a first input text, the first input text including first description information; loading the first model when determining to use a first model for image generation based on the first input text, the first model being obtained by retraining a text image model through a user's customized sample set, the customized sample set including a user-customized sample image and sample description information corresponding to the sample image; inputting the first input text into the first model to obtain a first image output by the first model that matches the first description information.

[0006] In a second aspect, an embodiment of the present application provides an image generating device, which includes: a first acquisition unit, used to acquire a first input text, wherein the first input text includes first description information; a loading unit, used to load the first model when determining to use a first model for image generation based on the first input text, wherein the first model is obtained by retraining a text image model with a user's customized sample set, wherein the customized sample set includes a user-defined sample image and sample description information corresponding to the sample image; a first generation unit, used to input the first input text into the first model to obtain a first image output by the first model that matches the first description information.

[0007] In a third aspect, an embodiment of the present application provides an electronic device, which includes a processor and a memory, wherein the memory stores programs or instructions that can be run on the processor, and when the program or instructions are executed by the processor, the steps of the method described in the first aspect are implemented.

[0008] In a fourth aspect, an embodiment of the present application provides a readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the method described in the first aspect above are implemented.

[0009] In a fifth aspect, an embodiment of the present application provides a chip, comprising a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run a program or instruction to implement the method described in the first aspect.

[0010] In a sixth aspect, an embodiment of the present application provides a computer program product, which is stored in a storage medium and is executed by at least one processor to implement the method described in the first aspect.

[0011] In an embodiment of the present application, a first input text is first obtained, and when it is determined based on the first input text that the first model is used for image generation, the first model is loaded, so that the first input text is input into the first model, and a first image output by the first model that matches the first description information in the first input text is obtained. Since the first model is obtained by retraining the text map model through the user's customized sample set, and the customized sample set includes the user-defined sample images and the sample description information corresponding to the sample images, the first model can learn the personalized description in the sample description information and the personalized features in the user-defined sample images, so that the generated first image can match the personalized description in the first description information, thereby improving the matching degree between the first image and the first description information, thereby improving the accuracy of the generated first image. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 is a flow chart of an image generation method provided in an embodiment of the present application;

[0013] Figure 2 It is one of the schematic diagrams of the application scenario of the image generation method provided in the embodiment of the present application;

[0014] Figure 3 This is a second schematic diagram of an application scenario of the image generation method provided in an embodiment of the present application;

[0015] Figure 4 This is a third schematic diagram of an application scenario of the image generation method provided in an embodiment of the present application;

[0016] Figure 5 is a schematic diagram of the structure of an image generating device provided in an embodiment of the present application;

[0017] Figure 6 is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application;

[0018] Figure 7 It is a schematic diagram of the hardware structure of an electronic device suitable for implementing the embodiments of the present application. DETAILED DESCRIPTION

[0019] The following will be combined with the drawings in the embodiments of the present application to clearly describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the present application belong to the scope of protection of this application.

[0020] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first", "second", etc. are generally of one type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0021] The image generation method and device provided in the embodiments of the present application are described in detail below through specific embodiments and their application scenarios in conjunction with the accompanying drawings.

[0022] Please refer to Figure 1 , which shows one of the flow charts of the image generation method provided in the embodiment of the present application. The image generation method provided in the embodiment of the present application can be applied to electronic devices. In practice, the above electronic devices can be electronic devices such as smart phones, tablet computers, laptops, and wearable devices.

[0023] The process of the image generation method provided in the embodiment of the present application includes the following steps:

[0024] Step 101: Obtain a first input text, where the first input text includes first description information.

[0025] In this embodiment, the first input text may be used to request the generation of a first image matching the first description information. The first description information may be information used to describe the first image to be generated. The first description information may include, but is not limited to, information used to describe the style, scene, subject, behavior, etc. of the first image.

[0026] As an example, the first input text is "generate a photo of my puppy running on the beach", the first description text is "my puppy running on the beach", the scene is "the beach", the subject is "my puppy", and the behavior is "running".

[0027] In practice, the user may directly input the first input text in the input box of the conversation page. At this time, the electronic device may extract the first input text in the input box. Alternatively, the user may input a voice in the conversation page. The electronic device may first extract the voice, and then convert the voice into text, thereby obtaining the first input text.

[0028] Step 102, when it is determined based on the first input text to use the first model to generate the image, the first model is loaded, and the first model is obtained by retraining the text image model through the user's customized sample set, and the customized sample set includes the user-defined sample image and the sample description information corresponding to the user-defined sample image.

[0029] In this embodiment, after obtaining the first input text, the first input text can be analyzed first to determine whether to use the first model. When the first description information in the first input text contains a personalized description, it can be determined to use the first model for image generation. As an example, when the user's first input text is "generate a photo of my puppy running on the beach", because the first input text involves the user's specific personalized content, that is, "my puppy", the general text image model cannot understand the user's concept of puppy. Therefore, it can be determined to use the first model for image generation.

[0030] In practice, it is possible to determine whether the first input text contains a personalized description, such as "I", "my", "my puppy", "my avatar", "daughter", "son", etc., by string matching. If it does, it can be determined to use the first model. In addition, the first input text can also be input into a model generated based on artificial intelligence technology, and the model can be used to determine whether to use the first model for image generation.

[0031] In this embodiment, when it is determined based on the first input text to use the first model for image generation, the first model can be loaded. The first model can be obtained by retraining the Text-to-image model through the user's customized sample set. The first model and the Text-to-image model can have the same model structure and different parameter configurations. The electronic device can pre-store a parameter configuration file of the first model and a parameter configuration file of the Text-to-image model. When it is determined based on the first input text to use the first model for image generation, the parameter configuration file of the first model can be loaded, and then the model structure file can be loaded, thereby realizing the loading of the first model.

[0032] Among them, the text graph model is an artificial intelligence technology based on deep learning that can convert descriptive text into corresponding images. The core function of this model is to interpret the natural language descriptions entered by users and generate visual content based on these descriptions. The implementation of the text graph model usually relies on deep learning frameworks such as generative adversarial networks or variational autoencoders. It is understandable that the general text graph model cannot handle the personalized text graph needs of users. It cannot obtain the characteristic information of a specific individual from the text, so it cannot generate images containing specific individuals. For example, it cannot match personalized descriptions in user texts, such as "me", "my dog", "my yard" and other personalized descriptions.

[0033] In this embodiment, the first model can be obtained by retraining the text graph model through the user's customized sample set. The above customized sample set may include user-defined sample images and sample description information corresponding to the sample images. The sample images can be selected and uploaded by the user from a personal album. The sample images can be images with user personalized features, for example, including but not limited to the user's own portrait, photos of the user's pets, photos of the user's family, photos of the user's work scenes or life scenes, etc.

[0034] Each sample image may have corresponding sample description information. The sample description information corresponding to different sample images may be the same or different. For example, a custom sample set includes multiple sample images, and three of the sample images contain the user's pet dog. Then the sample description information corresponding to the three sample images may include "my pet dog". The sample image can be used as a label for the sample description information to guide the retraining of the text graph model.

[0035] Since the first model is obtained by retraining the text image model through the user's customized sample set, and the customized sample set includes the user-customized sample images and the sample description information corresponding to the sample images, the first model can learn the personalized description in the sample description information and the personalized features in the user-customized sample images, so that the generated first image can match the personalized description in the first description information, thereby improving the matching degree between the first image and the first description information, thereby improving the accuracy of the generated first image.

[0036] Furthermore, the custom sample set can be further divided to obtain multiple sub-sample sets. Each sub-sample set may include at least one sample image. The sample images in the same sub-sample set may have the same sample description information. The Wensheng graph model can be retrained based on different sub-sample sets to obtain parameter configuration files of the first model corresponding to different sample description information. In the case of determining to use the first model for image generation based on the first input text, the target parameter configuration file is selected based on the matching of the first description information in the first input text with the different sample description information, so as to load the target parameter configuration file to achieve more targeted loading of the first model. In this way, it can be ensured that the generated image is highly consistent with the specific needs of the user. This saving mechanism not only facilitates further fine-tuning in the future, but also ensures the continuity and stability of the generated results.

[0037] Step 103: input the first input text into the first model to obtain an image output by the first model that matches the first description information.

[0038] In this embodiment, after the first model is loaded, the first input text may be input into the first model. The first model may perform semantic understanding on the first input text and generate an image that conforms to the semantics.

[0039] As an example, the first input text is "Generate a photo of my puppy running on the beach", and the custom sample set includes sample images containing "puppy". After the first input text is input into the first model trained based on the custom sample set, the first model can accurately understand the user's description of "my puppy", including the puppy's appearance, body shape, coat color, breed and other characteristics, and ensure that the generated first image is consistent with the puppy's features in the photo uploaded by the user. Ultimately, the first model will generate a photo showing a puppy running on the beach, which not only accurately reflects the content described in the first description information, but is also highly consistent with the image of the puppy in the puppy photo provided by the user.

[0040] The method provided by the above-mentioned embodiment of the present application first obtains a first input text, and in the case of determining to use a first model for image generation based on the first input text, loads the first model, thereby inputting the first input text into the first model, and obtaining a first image output by the first model that matches the first description information in the first input text. Since the first model is obtained by retraining the text map model through the user's customized sample set, and the customized sample set includes the user-customized sample images and the sample description information corresponding to the sample images, the first model can learn the personalized description in the sample description information and the personalized features in the user-customized sample images, so that the generated first image can match the personalized description in the first description information, thereby improving the matching degree between the first image and the first description information, thereby improving the accuracy of the generated first image.

[0041] In some optional embodiments, the user-defined sample image and the sample description information corresponding to the user-defined sample image may be obtained through the following steps:

[0042] Step S11: receiving a first input of a user on an image set.

[0043] Here, the first input may be used to select at least one user-defined sample image in the image set that contains the same subject. Figure 2 , an image selection page of an image set may be displayed first. The image selection page may include a plurality of pre-stored images. The user may click at least one of the images containing the same subject, and then click a "Finish" button to select a user-defined sample image, thereby completing the first input.

[0044] It should be noted that the input in the embodiment of the present application can be a touch input, or a voice command, or a specific gesture input by the user, or other feasible input, which can be determined according to actual use requirements, and the embodiment of the present application is not limited. The specific gesture in the embodiment of the present application can be any one of a single-click gesture, a sliding gesture, a drag gesture, a pressure recognition gesture, a long press gesture, an area change gesture, a double-press gesture, and a double-click gesture. The click input in the embodiment of the present application can be a single-click input, a double-click input, or any number of click inputs, etc., and can also be a long press input or a short press input.

[0045] Step S12, in response to the first input, determining a user-defined sample image in the image set, and inputting the sample image into the second model to obtain at least one candidate description information output by the second model.

[0046] Here, the second model can be a large language model (LLM), which is a deep learning model trained based on massive text data. It can not only generate natural language text, but also deeply understand the meaning of the text and handle various natural language tasks, such as text summarization, question and answer, translation, etc. In practical applications, the above-mentioned large language model can be trained based on an existing language model, or it can be trained using a self-developed language model, which is not specifically limited here. The large language model here can specifically use a multimodal large language model (MLLM). A multimodal large language model is a large language model that combines multimodal information such as text, images, videos, and audio for training. The multimodal large language model can recognize and extract the content in the above-mentioned sample images and generate corresponding text descriptions.

[0047] As an example, see Figure 3 After the user selects six images containing his daughter and son in the image collection, the system can input the image into a large language model, extract the characters in the image and recognize the scene in the image through the large language model, so as to generate at least one candidate description information for the image, for example, "the little boy and the little girl are running happily", "the babies are running happily", "the little boy and the little girl".

[0048] Step S13, displaying the at least one candidate description information. As an example, the display interface of the at least one candidate description information can be seen in Figure 3 In addition to the above-mentioned at least one candidate description information, the display interface may also include a user-defined sample image selected by the user. The user can browse other selected user-defined sample images by sliding left and right.

[0049] Step S14: receiving a second input from a user regarding the at least one piece of candidate description information. Here, the second input may be used to select sample description information corresponding to the sample image from the at least one piece of candidate description information.

[0050] Step S15, in response to the second input, determining sample description information corresponding to the sample image from the at least one piece of candidate description information.

[0051] As an example, see Figure 3 , the user clicks the candidate description information "a little boy and a little girl are running happily" shown in label 301, and then clicks the "confirm" button shown in label 302 to complete the second input. At this time, the candidate description information "a little boy and a little girl are running happily" shown in label 301 can be determined as the sample description information corresponding to the above sample image.

[0052] By receiving the user's first input in the image set, personalized sample images can be acquired conveniently and quickly, so that the generated images can better reflect the user's specific needs and unique characteristics; by generating candidate description information with the help of the second model, sample description information can be added to the sample image conveniently and quickly, thereby improving the generation efficiency of customized sample sets.

[0053] In some optional embodiments, the above-mentioned step S15 of determining the sample description information corresponding to the above-mentioned sample image from at least one candidate description information may further include: selecting the initial description information corresponding to the user-defined sample image from at least one candidate description information; receiving a third input from the user for the initial description information; and updating the initial description information in response to the third input to obtain the sample description information corresponding to the user-defined sample image.

[0054] The third input can be used to edit the initial description. Figure 3 After the user clicks the candidate description information "a little boy and a little girl are running happily" shown in label 301, the candidate description information "a little boy and a little girl are running happily" shown in label 301 can be used as the initial description information. The user clicks the "Edit" button shown in label 303 to display Figure 4 In the editing interface shown in FIG. 401, the user can edit the initial description information in the control shown in reference number 401, for example, modify "the little boy and the little girl are running happily" to "the son and the daughter are running happily". After the editing is completed, the "confirm" button shown in reference number 402 can be clicked. At this time, the editing result "the son and the daughter are running happily" can be used as the sample description information corresponding to the sample image.

[0055] By providing the description information editing function, it is convenient for users to further adjust the automatically generated candidate description information, so that the sample description information corresponding to the sample image is more accurate.

[0056] In some optional embodiments, the sample images in the custom sample set and their corresponding sample description information may also be preprocessed, which may specifically include:

[0057] Step S21, preprocessing the sample image.

[0058] The above-mentioned preprocessing may include but is not limited to interpolation, denoising, etc. Specifically, an interpolation algorithm can be used to adjust the user-defined sample image to a target size to update the user-defined sample image. Here, in order to adapt to the input requirements of the first model, an interpolation algorithm such as bicubic interpolation can be used to adjust the sample image to a uniform size, for example, 512×512 pixels. Bicubic interpolation is a smooth interpolation method that can better preserve the details and edges of the image during the scaling process. In addition, the noise that may exist in the interpolated sample image can be denoised using denoising techniques such as Gaussian filtering to improve the clarity and quality of the sample image.

[0059] Step S22: pre-process the sample description information.

[0060] Specifically, the sample description information can be first segmented to obtain a segmentation result. Then, invalid words in the segmentation result are deleted, and invalid words include at least one of the following: punctuation marks, numbers, special characters, stop words, and redundant words. After that, for each word in the segmentation result after the invalid words are deleted, semantic disambiguation is performed using methods such as knowledge base matching, corpus matching, and open source models to obtain the target word corresponding to the word. Finally, the target words are concatenated to obtain the updated sample description information.

[0061] By preprocessing the sample images and sample description information, the quality of the sample images and the sample description information can be effectively improved, and the matching of the sample images and the sample description information can be improved, providing an accurate data basis for the subsequent training of the first model.

[0062] In some optional embodiments, the first model is obtained by retraining the text graph model through the following steps:

[0063] Step S31, input the sample description information corresponding to the sample image into the Wensheng graph model to obtain the predicted image output by the Wensheng graph model. Specifically, the sample images and their corresponding sample description information in the custom sample set can be first converted into a coding form using a multimodal pre-trained neural network (Contrastive Language-Image Pre-Training, CLIP). Then, the coding corresponding to the sample image is processed by convolution and the like, and finally the coding of the predicted image is obtained.

[0064] Step S32, based on the sample image, the predicted image and the first loss function, determine the first loss value of the Wensheng graph model. Here, the first loss function can be used to minimize the distance between sample images corresponding to the same sample description information. As an example, the first loss function can refer to the following formula 1:

[0065] L contrastive=max(0,margin-similarity(image1, image2) (1)

[0066] Among them, L contrastive is the first loss function; max() is the maximum value function, which is used to ensure that the loss value is always non-negative. margin is a predefined hyperparameter; similarity() function is the similarity calculation function, which can specifically adopt the cosine similarity calculation function; image1 is the encoding of the sample image; image2 is the encoding of the predicted image.

[0067] Step S33, input the sample image and the predicted image into the discriminator to obtain a discrimination result.

[0068] Here, the Wensheng graph model can be used as a generator and connected to the discriminator, so that the predicted image output by the Wensheng graph model can be input into the discriminator to obtain the discrimination result output by the discriminator. The discriminator here is used to determine the probability that the predicted image input into it is a real image.

[0069] Step S34, based on the discrimination result and the second loss function, determine the second loss value of the Wensheng graph model. Here, the second loss function can be used to minimize the difference between the predicted image and the real image. As an example, the second loss function can refer to the following formula 2:

[0070]

[0071] Among them, L adversarial is the second loss function; z is the noise vector extracted from the latent space; G(z) is the predicted image generated by the Wensheng graph model, which is a mapping of the noise variable z; D(G(z)) is the probability that the above predicted image is a true sample; represents the expected calculation of all possible noise vectors.

[0072] Step S35, based on the first loss value and the second loss value, determine the total loss value of the Wensheng graph model. Here, the first loss value and the second loss value may be weighted and summed to obtain the total loss value of the Wensheng graph model. As an example, see the following formula 3:

[0073] L composite =α·L contrastive +β·L adversarial (3)

[0074] Among them, L composite is the total loss value; α and β are weight coefficients, which can be pre-set as needed.

[0075] Step S36, based on the total loss value, update the parameters of the Wensheng graph model to obtain the first model. Here, the back propagation algorithm can be used to obtain the gradient of the loss value relative to the Wensheng graph model parameters, and then the gradient descent algorithm can be used to update the model parameters based on the gradient. In this way, the training of the Wensheng graph model can be completed once. The above steps S31 to S36 can be iteratively executed multiple times until the end iteration condition is met. At this time, the first model can be obtained.

[0076] By combining the first loss function and the second loss function to determine the total loss value of the Wensheng graph model, it is possible to minimize the distance between sample images corresponding to the same sample description information while minimizing the difference between the predicted image and the real image, so that the trained first model can generate more realistic and accurate images.

[0077] In some optional embodiments, after executing step 101 and before executing step 102, the following steps may also be executed: generating a second input text based on the first input text; inputting the second input text into a second model, and determining whether to use the second model for image generation through the second model.

[0078] Among them, the second input text can be used to request a large language to determine whether to use the first model for image generation. In practice, the first input text can be substituted into the preset first request template to obtain the second input text. For example, the first input text is "Generate a photo of my puppy running on the beach", and the first request template is "If you are a second model, please determine whether the following request text uses the first model. Request text: ". Then the second input text can be "If you are a second model, please determine whether the following request text uses the first model. Request text: Generate a photo of my puppy running on the beach".

[0079] The second model can be a model with strong semantic understanding capabilities, such as a large language model. Therefore, by performing semantic analysis on the first input text through the second model, it can be convenient, quick and accurate to determine whether to use the first model for image generation, thereby improving the accuracy of model selection.

[0080] In some optional embodiments, after executing step 103, the following steps may also be executed: obtaining a third input text, the third input text including the second description information; inputting the third input text and the first image into the graph-generated image model to obtain information generated by the graph-generated image model that matches the second description information and the first image, the information including at least one of the following: a three-dimensional model, a second image, and a video.

[0081] Among them, the third input text can be used to request the graph generation model to generate a second image or video based on the second description information and the first image. The second description information may be information used to describe the second image or video to be generated. The second description information may include but is not limited to information used to describe the style, scene, subject, behavior, type, etc. of the second image or video. The graph generation model is an image generation technology based on a generative adversarial network, which is mainly used to generate new images by preprocessing existing images. The core principle is to upload a picture, process it, and then combine the prompt words and adjust the parameters to generate new image-related information. This method can be applied to a variety of scenarios such as line drawing coloring, sketch refinement, style transfer, and image enlargement.

[0082] As an example, the subject in the second image is the user's puppy, and the third input text is "Please generate a three-dimensional model of my puppy based on the following image". After inputting the above second image and third input text into the graph-generated image model, a three-dimensional model of the user's puppy generated by the graph-generated image model can be obtained, and the three-dimensional model matches the features of the user's puppy in the second image.

[0083] As another example, the subject in the second image is the user's dog, and the content is the dog running on the beach. The third input text is "Please generate an image of my dog ​​running on the grass based on the following image", then after the second image and the third input text are input into the graph-generated graph model, a second image of the user's dog running on the grass generated by the graph-generated graph model can be obtained, and the dog in the second image matches the features of the dog in the second image.

[0084] As another example, the subject in the second image is the user's dog, and the content is the dog running on the beach. The third input text is "Please generate a video of my dog ​​jumping on the grass based on the following image", then after the second image and the third input text are input into the graph-generated image model, a video of the user's dog jumping on the grass generated by the graph-generated image model can be obtained, and the dog in the video matches the features of the dog in the second image.

[0085] By using a graph-based model to generate a three-dimensional model, a second image or a video that matches the second description information and the first image, the form of information generation can be enriched, thereby being applied to game design, virtual reality and other fields.

[0086] It should be noted that the image generation method provided in the embodiment of the present application can be executed by an image generation device. In the embodiment of the present application, the image generation device executing the image generation method is taken as an example to illustrate the image generation device provided in the embodiment of the present application.

[0087] like Figure 5As shown, the image generation device 500 of this embodiment includes: a first acquisition unit 501, used to acquire a first input text, wherein the first input text includes first description information; a loading unit 502, used to load the first model when it is determined to use a first model for image generation based on the first input text, wherein the first model is obtained by retraining a text image model through a user-defined sample set, wherein the user-defined sample set includes a user-defined sample image and sample description information corresponding to the sample image; a first generation unit 503, used to input the first input text into the first model, and obtain a first image output by the first model that matches the first description information.

[0088] In some optional implementations of the present embodiment, the sample image and the sample description information corresponding to the sample image are obtained through the following steps: receiving a first input of a user to an image set; in response to the first input, selecting a user-defined sample image from the image set, and inputting the sample image into a second model to obtain at least one candidate description information output by the second model; displaying the at least one candidate description information; receiving a second input of the user to the at least one candidate description information; in response to the second input, determining the sample description information corresponding to the sample image from the at least one candidate description information. By receiving the first input of the user in the image set, it is possible to conveniently and quickly obtain personalized sample images, so that the generated images can better reflect the user's specific needs and unique features; by generating candidate description information with the help of the second model, it is possible to conveniently and quickly add sample description information to the sample image, thereby improving the efficiency of generating a custom sample set.

[0089] In some optional implementations of this embodiment, determining the sample description information corresponding to the sample image from the at least one candidate description information includes: selecting initial description information corresponding to the sample image from the at least one candidate description information; receiving a third input from a user on the initial description information; and updating the initial description information in response to the third input to obtain the sample description information corresponding to the sample image. By providing a description information editing function, it is convenient for users to further adjust the automatically generated candidate description information, so that the sample description information corresponding to the sample image is more accurate.

[0090] In some optional implementations of the present embodiment, the first model is obtained by retraining the Vincent graph model through the following steps: inputting the sample description information corresponding to the sample image into the Vincent graph model to obtain the predicted image output by the Vincent graph model; determining the first loss value based on the sample image, the predicted image and the first loss function; inputting the sample image and the predicted image into the discriminator to obtain the discrimination result; determining the second loss value based on the discrimination result and the second loss function; determining the total loss value based on the first loss value and the second loss value; updating the parameters of the Vincent graph model based on the total loss value to train the first model. By combining the first loss function and the second loss function to determine the total loss value of the Vincent graph model, the difference between the predicted image and the real image can be minimized while minimizing the distance between the sample images corresponding to the same sample description information, so that the trained first model can generate a more realistic and accurate image.

[0091] In some optional implementations of this embodiment, the device further includes: a second generation unit, for generating a second input text based on the first input text; and a determination unit, for inputting the second input text into a second model, and determining whether to use the first model for image generation through the second model. Since the second model has a strong semantic understanding capability, the semantic analysis of the first input text through the second model can conveniently, quickly and accurately determine whether to use the first model for image generation, thereby improving the accuracy of model selection.

[0092] In some optional implementations of this embodiment, the device further includes: a second acquisition unit, used to acquire a third input text, wherein the third input text includes second description information; and a third generation unit, used to input the third input text and the first image into a graph-generated graph model, and obtain information generated by the graph-generated graph model that matches the second description information and the first image, wherein the information includes at least one of the following: a three-dimensional model, a second image, and a video. By using the graph-generated graph model to generate a three-dimensional model, a second image, or a video that matches the second description information and the first image, the form of information generation can be enriched, thereby being applied to game design, virtual reality, and other fields.

[0093] The device provided by the above-mentioned embodiment of the present application first obtains a first input text, and in the case of determining to use a first model for image generation based on the first input text, loads the first model, thereby inputting the first input text into the first model, and obtaining a first image output by the first model that matches the first description information in the first input text. Since the first model is obtained by retraining the text image model through the user's customized sample set, and the customized sample set includes the user-defined sample images and the sample description information corresponding to the sample images, the first model can learn the personalized description in the sample description information and the personalized features in the user-defined sample images, so that the generated first image can match the personalized description in the first description information, thereby improving the matching degree between the first image and the first description information, thereby improving the accuracy of the generated first image.

[0094] The image generating device in the embodiment of the present application can be an electronic device, or a component in the electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal, or other devices other than a terminal. Exemplarily, the electronic device can be a mobile phone, a tablet computer, a laptop computer, a PDA, a vehicle-mounted electronic device, a mobile Internet device (Mobile Internet Device, MID), an augmented reality (Augmented Reality, AR) / virtual reality (Virtual Reality, VR) device, a robot, a wearable device, an ultra-mobile personal computer (Ultra-Mobile Personal Computer, UMPC), a netbook or a personal digital assistant (Personal Digital Assistant, PDA), etc., and can also be a server, a network attached storage (Network Attached Storage, NAS), a personal computer (Personal Computer, PC), a television (Television, TV), a teller machine or a self-service machine, etc., and the embodiment of the present application is not specifically limited.

[0095] The image generation device in the embodiment of the present application may be a device having an operating system. The operating system may be an Android operating system, an iOS operating system, or other possible operating systems, which are not specifically limited in the embodiment of the present application.

[0096] The image generation device provided in the embodiment of the present application can achieve Figure 1 To avoid repetition, the various processes implemented by the method embodiment are not described here.

[0097] Alternatively, if Figure 6As shown, an embodiment of the present application further provides an electronic device 600, including a processor 601 and a memory 602, wherein the memory 602 stores programs or instructions that can be executed on the processor 601, and when the program or instructions are executed by the processor 601, the various steps of the above-mentioned image generation method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, they are not described here.

[0098] It should be noted that the electronic devices in the embodiments of the present application include the above-mentioned mobile electronic devices and non-mobile electronic devices.

[0099] Figure 7 A schematic diagram of the hardware structure of an electronic device to implement an embodiment of the present application.

[0100] The electronic device 700 includes but is not limited to: a radio frequency unit 701, a network module 702, an audio output unit 703, an input unit 704, a sensor 705, a display unit 706, a user input unit 707, an interface unit 708, a memory 709, and a processor 710.

[0101] Those skilled in the art will appreciate that the electronic device 700 may also include a power source (such as a battery) for supplying power to each component, and the power source may be logically connected to the processor 710 through a power management system, thereby implementing functions such as managing charging, discharging, and power consumption management through the power management system. Figure 7 The electronic device structure shown in the figure does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently, which will not be described in detail here.

[0102] Among them, the processor 710 is used to obtain a first input text, which includes first description information; when it is determined to use a first model to generate an image based on the first input text, the first model is loaded, and the first model is obtained by retraining a text image model through a user's customized sample set, and the customized sample set includes a user-defined sample image and sample description information corresponding to the sample image; the first input text is input into the first model to obtain a first image output by the first model that matches the first description information.

[0103] Since the first model is obtained by retraining the text image model through the user's customized sample set, and the customized sample set includes the user-customized sample images and the sample description information corresponding to the sample images, the first model can learn the personalized description in the sample description information and the personalized features in the user-customized sample images, so that the generated first image can match the personalized description in the first description information, thereby improving the matching degree between the first image and the first description information, thereby improving the accuracy of the generated first image.

[0104] In some optional implementations of this embodiment, the processor 710 is further used to receive a first input of a user to an image set through the user input unit 707; in response to the first input, select a user-defined sample image from the image set, and input the sample image to the second model to obtain at least one candidate description information output by the second model; display the at least one candidate description information; receive a second input of the user to the at least one candidate description information through the user input unit 707; in response to the second input, determine the sample description information corresponding to the sample image from the at least one candidate description information. By receiving the first input of the user in the image set, it is possible to conveniently and quickly obtain personalized sample images, so that the generated images can better reflect the user's specific needs and unique features; by generating candidate description information with the help of the second model, it is possible to conveniently and quickly add sample description information to the sample image, thereby improving the generation efficiency of the custom sample set.

[0105] In some optional implementations of this embodiment, the processor 710 is further configured to select initial description information corresponding to the sample image from the at least one candidate description information; receive a third input of the user to the initial description information through the user input unit 707; and update the initial description information in response to the third input to obtain sample description information corresponding to the sample image. By providing a description information editing function, it is convenient for the user to further adjust the automatically generated candidate description information, so that the sample description information corresponding to the sample image is more accurate.

[0106] In some optional implementations of this embodiment, the processor 710 is further configured to input the sample description information corresponding to the sample image into the Vincent graph model to obtain a predicted image output by the Vincent graph model; determine a first loss value based on the sample image, the predicted image and the first loss function; input the sample image and the predicted image into a discriminator to obtain a discrimination result; determine a second loss value based on the discrimination result and the second loss function; determine a total loss value based on the first loss value and the second loss value; and update the parameters of the Vincent graph model based on the total loss value to train the first model. By combining the first loss function and the second loss function to determine the total loss value of the Vincent graph model, the difference between the predicted image and the real image can be minimized while minimizing the distance between the sample images corresponding to the same sample description information, so that the trained first model can generate a more realistic and accurate image.

[0107] In some optional implementations of this embodiment, the processor 710 is further configured to generate a second input text based on the first input text; input the second input text into the second model, and determine whether to use the first model for image generation through the second model. Since the second model has a strong semantic understanding ability, the semantic analysis of the first input text through the second model can conveniently, quickly and accurately determine whether to use the first model for image generation, thereby improving the accuracy of model selection.

[0108] In some optional implementations of this embodiment, the processor 710 is further configured to obtain a third input text, wherein the third input text includes the second description information; input the third input text and the first image into the graph-generated graph model, and obtain information generated by the graph-generated graph model that matches the second description information and the first image, wherein the information includes at least one of the following: a three-dimensional model, a second image, and a video. By using the graph-generated graph model to generate a three-dimensional model, a second image, or a video that matches the second description information and the first image, the form of information generation can be enriched, thereby being applied to game design, virtual reality, and other fields.

[0109] It should be understood that in the embodiment of the present application, the input unit 704 may include a graphics processor (Graphics Processing Unit, GPU) 7041 and a microphone 7042, and the graphics processor 7041 processes the image data of the static picture or video obtained by the image capture device (such as a camera) in the video capture mode or the image capture mode. The display unit 706 may include a display panel 7061, and the display panel 7061 may be configured in the form of a liquid crystal display, an organic light emitting diode, etc. The user input unit 707 includes a touch panel 7071 and at least one of other input devices 7072. The touch panel 7071 is also called a touch screen. The touch panel 7071 may include two parts: a touch detection device and a touch controller. Other input devices 7072 may include, but are not limited to, a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, and a joystick, which will not be repeated here.

[0110] The memory 709 can be used to store software programs and various data. The memory 709 may mainly include a first storage area for storing programs or instructions and a second storage area for storing data, wherein the first storage area may store an operating system, an application program or instructions required for at least one function (such as a sound playback function, an image playback function, etc.), etc. In addition, the memory 709 may include a volatile memory or a non-volatile memory, or the memory 709 may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), a static random access memory (SRAM), a dynamic random access memory (DRAM), a synchronous dynamic random access memory (SDRAM), a double data rate synchronous dynamic random access memory (DDRSDRAM), an enhanced synchronous dynamic random access memory (ESDRAM), a synchronous link dynamic random access memory (SLDRAM) and a direct memory bus random access memory (DRRAM). The memory 709 in the embodiment of the present application includes but is not limited to these and any other suitable types of memory.

[0111] The processor 710 may include one or more processing units; optionally, the processor 710 integrates an application processor and a modem processor, wherein the application processor mainly processes operations related to an operating system, a user interface, and application programs, and the modem processor mainly processes wireless communication signals, such as a baseband processor. It is understandable that the modem processor may not be integrated into the processor 710.

[0112] An embodiment of the present application also provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, the various processes of the above-mentioned image generation method embodiment are implemented and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0113] The processor is the processor in the electronic device described in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory ROM, a random access memory RAM, a magnetic disk or an optical disk.

[0114] An embodiment of the present application further provides a chip, which includes a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned image generation method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0115] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

[0116] An embodiment of the present application provides a computer program product, which is stored in a storage medium. The program product is executed by at least one processor to implement the various processes of the above-mentioned image generation method embodiment and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0117] It should be noted that, in this article, the terms "comprise", "include" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise one..." do not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0118] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a computer software product, which is stored in a storage medium (such as ROM / RAM, a disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0119] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

Claims

1. An image generation method, characterized in that: The method comprises: Acquire a first input text, where the first input text includes first description information; In the case where it is determined based on the first input text to use a first model for image generation, loading the first model, the first model being obtained by retraining a text image model through a user-defined sample set, the user-defined sample set including a user-defined sample image and sample description information corresponding to the sample image; The first input text is input into the first model to obtain a first image output by the first model that matches the first description information.

2. The method according to claim 1, characterized in that The sample image and the sample description information corresponding to the sample image are obtained by the following steps: receiving a first input of a user on a collection of images; In response to the first input, a sample image is selected from the image set, and the sample image is input into a second model to obtain at least one candidate description information output by the second model; Displaying the at least one candidate description information; receiving a second input of the at least one piece of candidate description information by a user; In response to the second input, sample description information corresponding to the sample image is determined from the at least one piece of candidate description information.

3. The method according to claim 2, characterized in that The step of determining the sample description information corresponding to the sample image from the at least one piece of candidate description information includes: Selecting initial description information corresponding to the sample image from the at least one piece of candidate description information; receiving a third input of the initial description information from a user; In response to the third input, the initial description information is updated to obtain sample description information corresponding to the sample image.

4. The method according to claim 1, characterized in that: After obtaining the first input text, the method further includes: Based on the first input text, generate a second input text; The second input text is input into the second model, and the second model is used to determine whether to use the first model for image generation.

5. The method according to claim 1, characterized in that After obtaining the first image output by the first model and matching the first description information, the method further includes: Acquire a third input text, wherein the third input text includes the second description information; The third input text and the first image are input into a graph-generated image model to obtain information generated by the graph-generated image model and matching the second description information and the first image, wherein the information includes at least one of the following: a three-dimensional model, a second image, and a video.

6. An image generating device, characterized in that: The device comprises: A first acquisition unit, configured to acquire a first input text, wherein the first input text includes first description information; a loading unit, configured to load the first model when it is determined based on the first input text that a first model is used for image generation, the first model being obtained by retraining a text image model through a user-defined sample set, the user-defined sample set including a user-defined sample image and sample description information corresponding to the sample image; The first generating unit is used to input the first input text into the first model to obtain a first image output by the first model that matches the first description information.

7. The device according to claim 6, characterized in that The sample image and the sample description information corresponding to the sample image are obtained by the following steps: receiving a first input of a user on a collection of images; In response to the first input, selecting a user-defined sample image from the image set, and inputting the sample image into a second model to obtain at least one candidate description information output by the second model; Displaying the at least one candidate description information; receiving a second input of the at least one piece of candidate description information by a user; In response to the second input, sample description information corresponding to the sample image is determined from the at least one piece of candidate description information.

8. The device according to claim 7, characterized in that The step of determining the sample description information corresponding to the sample image from the at least one piece of candidate description information includes: Selecting initial description information corresponding to the sample image from the at least one piece of candidate description information; receiving a third input of the initial description information from a user; In response to the third input, the initial description information is updated to obtain sample description information corresponding to the sample image.

9. The device according to claim 6, characterized in that The device also includes: A second generating unit, configured to generate a second input text based on the first input text; A determination unit is used to input the second input text into a second model, and determine whether to use the first model to generate an image through the second model.

10. The device according to claim 6, characterized in that The device also includes: A second acquisition unit, configured to acquire a third input text, wherein the third input text includes second description information; A third generating unit is used to input the third input text and the first image into the graph-generated image model to obtain information generated by the graph-generated image model that matches the second description information and the first image, wherein the information includes at least one of the following: a three-dimensional model, a second image, and a video.

Images