Image generation method and device, electronic equipment and computer readable storage medium

Abstract

The embodiment of the invention discloses an image generation method and device, electronic equipment and a computer readable storage medium. According to the embodiment of the invention, after the original image and the image description text corresponding to the original image are acquired and word segmentation is performed on the image description text to obtain the text word set, feature extraction is performed on the original image, at least one negative word is generated based on the extracted image features, and feature extraction is performed on the text word set and the negative words respectively; obtaining positive text features corresponding to the text word set and negative text features corresponding to the negative words, then adjusting the positive text features by using the negative text features to obtain target text features, and generating a target image corresponding to the original image based on the target text features and the image features; according to the scheme, the image generation efficiency can be improved.

Description

Technical Field

[0001] The present invention relates to the field of image processing, and in particular to an image generation method, device, electronic device and computer-readable storage medium. Background Art

[0002] In recent years, with the rapid development of Internet technology, customized image editing has become very popular and in high demand. Customized image editing requires retaining some of the content in the uploaded original image, such as the appearance content or other content that needs to be retained, and then editing and replacing other content to generate a customized image. Existing image generation methods often need to use the text information corresponding to the target object to fine-tune the base model, and use the fine-tuned model to generate images for the target object.

[0003] In the process of research and practice of the prior art, the inventors of the present application found that the fine-tuning strategy requires the input of multiple images, and that, in the fine-tuning strategy, the base model needs to be fine-tuned for each object, which makes the model less compatible and, therefore, results in lower efficiency in image generation. Summary of the invention

[0004] Embodiments of the present invention provide an image generation method, an apparatus, an electronic device, and a computer-readable storage medium, which can improve the efficiency of image generation.

[0005] A method for generating an image, comprising:

[0006] Acquire an original image and an image description text corresponding to the original image, and segment the image description text to obtain a text word set;

[0007] Extracting features from the original image, and generating at least one negative word based on the extracted image features, wherein the negative word represents content deleted from the original image during the image generation process;

[0008] Performing feature extraction on the text word set and the negative words respectively to obtain positive text features corresponding to the text word set and negative text features corresponding to the negative words;

[0009] Using the negative text features to adjust the positive text features to obtain target text features;

[0010] A target image corresponding to the original image is generated based on the target text feature and the image feature.

[0011] Accordingly, an embodiment of the present invention provides an image generating device, including:

[0012] An acquisition unit, used to acquire an original image and an image description text corresponding to the original image, and segment the image description text to obtain a text word set;

[0013] A first extraction unit is used to extract features from the original image, and generate at least one negative word based on the extracted image features, wherein the negative word represents content deleted from the original image during the image generation process;

[0014] A second extraction unit is used to perform feature extraction on the text word set and the negative words respectively to obtain positive text features corresponding to the text word set and negative text features corresponding to the negative words;

[0015] An adjustment unit, used to adjust the positive text feature using the negative text feature to obtain a target text feature;

[0016] A generating unit is used to generate a target image corresponding to the original image based on the target text feature and the image feature.

[0017] In some embodiments, the adjustment unit may be specifically used to obtain a feature fusion parameter corresponding to a preset text feature space; and according to the feature fusion parameter, the negative text feature and the positive text feature are fused to obtain a target text feature.

[0018] In some embodiments, the adjustment unit can be specifically used to calculate the feature difference between the negative text feature and the positive text feature; fuse the feature fusion parameter with the feature difference to obtain the adjusted positive text feature; fuse the adjusted positive text feature with the negative text feature to obtain the target text feature.

[0019] In some embodiments, the generating unit can be specifically used to obtain the domain category of the original image, and adjust the image features according to the domain category to obtain target image features; add preset image noise to the target image features to obtain noisy image features, and denoise the noisy image features based on the target text features to obtain target content features; feature decode the target content features to obtain the target image corresponding to the original image.

[0020] In some embodiments, the generating unit can be specifically used to use the image feature as the target image feature when the domain category is a target domain category; when the domain category is a domain category other than the target domain category, set the feature value of the image feature to zero to obtain the target image feature.

[0021] In some embodiments, the acquisition unit can be specifically used to receive an image generation request, which carries an original image and image generation information corresponding to the original image; when the image generation information contains a current image description text, the current image description text is used as the image description text corresponding to the original image; when the image generation information does not contain the current image description text, the image description text corresponding to the original image is generated based on the image generation information and the original image.

[0022] In some embodiments, the acquisition unit may be specifically configured to identify image description information in the original image based on the image generation request; and generate the image description text according to the image generation information and the image description information.

[0023] In some embodiments, the acquisition unit can be specifically used to identify image content in the original image according to the image generation request to obtain content description text; determine the image title corresponding to the original image based on the content description text; obtain image attribute information of the original image, and use the content description text, image title and image attribute information as the image description information.

[0024] In some embodiments, the first extraction unit can be specifically used to remove the background image in the original image to obtain a main image; use an image encoding network of an image generation model to encode the main image to obtain image features; and generate at least one negative word based on the extracted image features, including: using a negative word generation network of the image generation model to generate at least one negative word based on the image features.

[0025] In some embodiments, the image generating device may further include a training unit, which may be specifically used to obtain an original image sample and a description text sample corresponding to the original image sample; using a preset image generation model to generate a candidate image according to the original image sample, the description text sample and the preset image noise sample; determining the noise loss based on the preset image noise sample and the candidate image, and converging the preset image generation model according to the noise loss to obtain the image generation model.

[0026] In some embodiments, the training unit can be specifically used to obtain an initial original image sample, and remove the background image from the initial original image sample to obtain an original image sample; identify image description information in the original image template to obtain sample image description information; and generate a description text sample corresponding to the original image sample based on the sample image description information.

[0027] In some embodiments, the training unit can be specifically used to use a preset image generation model to extract features from the original image sample and the description text sample to obtain a sample feature set, wherein the sample feature set includes sample image features and sample text features; adding the preset image noise sample to the sample image feature to obtain the noisy sample image feature; based on the sample text feature, denoising the noisy sample image feature to obtain candidate content features and predicted noise features corresponding to the candidate content features, and feature decoding the candidate content features to obtain a candidate image; determining the noise loss based on the preset image noise sample and the candidate image includes: determining the noise loss based on the predicted noise features and the noise features of the preset image noise sample.

[0028] In some embodiments, the training unit can be specifically used to use a preset image generation model to feature the original image sample, and generate at least one sample negative word based on the extracted initial sample image features; extract text features from the descriptive text sample and the sample negative word respectively, and fuse the extracted text features to obtain sample text features; adjust the initial sample image features according to the sample domain category of the original image sample to obtain sample image features, and use the sample text features and the sample image features as a sample feature set.

[0029] In some embodiments, the training unit can be specifically used to adjust the initial sample image features according to the sample object identifier marked in the original image sample to obtain sample image features when the sample domain category is a target domain category; when the sample domain category is a domain category other than the target domain category, set the feature value of the initial sample image feature to zero to obtain sample image features.

[0030] In some embodiments, the training unit can be specifically used to predict the object identification of the object in the original image sample based on the initial sample image feature to obtain a predicted probability, wherein the predicted probability includes the probability that the object identification is the sample object identification; when the predicted probability is less than a preset probability threshold, the average sample image feature corresponding to the sample object identification is obtained, and the average sample image feature is used as the sample image feature; when the predicted probability is greater than or equal to the preset probability threshold, the initial sample image feature is used as the sample image feature.

[0031] In addition, an embodiment of the present invention further provides an electronic device, including a processor and a memory, wherein the memory stores an application program, and the processor is used to run the application program in the memory to implement the image generation method provided by the embodiment of the present invention.

[0032] In addition, an embodiment of the present invention further provides a computer-readable storage medium, which stores a plurality of instructions, and the instructions are suitable for a processor to load to execute the steps in any one of the image generation methods provided by the embodiments of the present invention.

[0033] In addition, an embodiment of the present application also provides a computer program product, including a computer program or instructions, which, when executed by a processor, implements the steps in the image generation method provided in the embodiment of the present application.

[0034] The embodiment of the present invention obtains the original image and the image description text corresponding to the original image, and performs word segmentation on the image description text to obtain a text word set, then performs feature extraction on the original image, and generates at least one negative word based on the extracted image features, and performs feature extraction on the text word set and the negative words respectively to obtain positive text features corresponding to the text word set and negative text features corresponding to the negative words, and then uses the negative text features to adjust the positive text features to obtain target text features, and generates a target image corresponding to the original image based on the target text features and the image features; since the scheme can generate a corresponding target image for an original image in different fields (source domain and target domain), there is no need to fine-tune each object, and the positive text features of the image description text can also be adjusted by the negative text features of the negative words to avoid overfitting, so that the content that needs to be retained in images of different fields can be accurately retained, and the accuracy and compatibility of image generation can be increased, thereby improving the efficiency of image generation. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0036] Figure 1 is a scene schematic diagram of the image generation method provided by an embodiment of the present invention;

[0037] Figure 2 is a flowchart of an image generating method provided by an embodiment of the present invention;

[0038] Figure 3 is a schematic diagram of comparing an original image and a target image while retaining appearance information provided by an embodiment of the present invention;

[0039] Figure 4 is a schematic diagram of comparing a target image and an original image corresponding to a first description text provided by an embodiment of the present invention;

[0040] Figure 5 is a schematic diagram of comparison between a target image and an original image corresponding to the second description text provided by an embodiment of the present invention;

[0041] Figure 6 is a schematic diagram of the structure of a stable diffusion network structure provided by an embodiment of the present invention;

[0042] Figure 7 is a schematic diagram of a training process of a preset image generation model provided by an embodiment of the present invention;

[0043] Figure 8 is a schematic diagram of comparing an original image and a target image in a scene where an animal image is an original image provided by an embodiment of the present invention;

[0044] Fig. 9 is another flowchart of the image generating method provided by an embodiment of the present invention;

[0045] Fig.10 is a schematic structural diagram of an image generating device provided by an embodiment of the present invention;

[0046] Fig.11 is another structural schematic diagram of the image generating device provided by an embodiment of the present invention;

[0047] Fig.12 It is a schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0048] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0049] The embodiment of the present invention provides an image generation method, device, electronic device and computer readable storage medium. The image generation device can be integrated in an electronic device, which can be a server or a terminal.

[0050] Among them, the server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, network acceleration services (Content Delivery Network, CDN), and basic cloud computing services such as big data and artificial intelligence platforms. The terminal can be a smart phone, tablet computer, laptop computer, desktop computer, smart speaker, smart watch, etc., but is not limited to this. The terminal and the server can be directly or indirectly connected via wired or wireless communication, and this application does not limit this.

[0051] For example, see Figure 1 Taking the image generation device integrated in an electronic device as an example, the electronic device obtains the original image and the image description text corresponding to the original image, and performs word segmentation on the image description text to obtain a text word set, then extracts features from the original image, and generates at least one negative word based on the extracted image features, and performs feature extraction on the text word set and the negative words respectively to obtain positive text features corresponding to the text word set and negative text features corresponding to the negative words, then adjusts the positive text features using the negative text features to obtain target text features, and generates a target image corresponding to the original image based on the target text features and image features, thereby improving the efficiency of image generation.

[0052] The image generation method provided in the embodiment of the present application relates to computer vision and natural language processing in artificial intelligence. The embodiment of the present application can obtain an original image and an image description text corresponding to the original image, edit the original image based on the image description text through an image generation model, obtain a target image of the original image, and thus improve the efficiency of image generation.

[0053] Among them, artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that machines have the functions of perception, reasoning and decision-making.

[0054] Artificial intelligence technology is a comprehensive discipline that covers a wide range of fields, including both hardware-level and software-level technologies. The basic technologies of artificial intelligence generally include sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics and other technologies. Artificial intelligence software technology mainly includes computer vision technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0055] Among them, Computer Vision (CV) is a science that studies how to make machines "see". To put it more specifically, it refers to the use of cameras and computers to replace human eyes to identify, collect and measure targets, and further process graphics so that the computer processes them into images that are more suitable for human observation or transmission to instruments for detection. As a scientific discipline, computer vision studies related theories and technologies, and attempts to establish an artificial intelligence system that can obtain information from images or multidimensional data. Computer vision technology usually includes image processing, image recognition, image semantic understanding, image retrieval, OCR, video processing, video semantic understanding, video content / behavior recognition, three-dimensional object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous positioning and map construction, and other technologies, as well as common biometric recognition technologies such as facial recognition and fingerprint recognition.

[0056] Among them, natural language processing (NLP) is an important direction in the fields of computer science and artificial intelligence. It studies various theories and methods that can achieve effective communication between people and computers using natural language. Natural language processing is a science that integrates linguistics, computer science, and mathematics. Therefore, research in this field will involve natural language, that is, the language people use in daily life, so it is closely related to the study of linguistics. Natural language processing technology usually includes text processing, semantic understanding, machine translation, robot question answering, knowledge graph and other technologies.

[0057] It can be understood that in the specific implementation of the present application, related data of objects such as original images and image texts are required. When the following embodiments of the present application are applied to specific products or technologies, permission or consent is required, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards of relevant countries and regions.

[0058] It should be noted that the description order of the following embodiments is not intended to limit the preferred order of the embodiments.

[0059] This embodiment will be described from the perspective of an image generating device, which can be specifically integrated into an electronic device, which can be a server or a terminal. The terminal can include a tablet computer, a laptop computer, a personal computer (PC), a wearable device, a virtual reality device, or other smart devices that can generate images.

[0060] A method for generating an image, comprising:

[0061] The original image and the image description text corresponding to the original image are obtained, and the image description text is segmented to obtain a text word set, and features are extracted for the original image, and at least one negative word is generated based on the extracted image features, and the negative word represents the content deleted from the original image during the image generation process; the text word set and the negative words are respectively extracted to obtain positive text features corresponding to the text word set and negative text features corresponding to the negative words, and the positive text features are adjusted using the negative text features to obtain target text features, and a target image corresponding to the original image is generated based on the target text features and the image features.

[0062] like Figure 2 As shown, the specific process of the image generation method is as follows:

[0063] 101. Obtain an original image and an image description text corresponding to the original image, and segment the image description text to obtain a text word set.

[0064] The original image can be a reference image in the image generation process, and is mainly used to provide some reference content or reserved content for the final generated target image. The so-called target image can be understood as retaining part of the content in the original image, and then deleting or adjusting other content in the original image, so that the generated target image has a different style or content. Taking the appearance information of the retained content as an example, the comparison between the original image and the corresponding target image can be as follows: Figure 3 shown.

[0065] The image description text may be understood as text that describes the image content of the generated target image. For example, if the original image is a dog and the target image to be generated is a dog on a scooter, then "a dog on a scooter" may be the corresponding image description text.

[0066] The original image and the image description text corresponding to the original image are obtained, and the image description text is segmented to obtain a text word set, which can be specifically as follows:

[0067] (1) Obtain the original image and the image description text corresponding to the original image.

[0068] For example, an image generation request can be accepted, which carries an original image and image generation information corresponding to the original image. When the image generation information contains the current image description text, the current image description text is used as the image description text corresponding to the original image. When the image generation information does not contain the current image description text, the image description text corresponding to the original image is generated based on the image generation information and the original image.

[0069] The image generation information may characterize or indicate the direction, type, style or other indicative information of editing the original image. For example, it may include the current description text that directly describes the image generation, or it may also include other text information or other information that indirectly describes the image generation, such as the type of image generation activity or activity time in which the object participates when uploading the original image, or other information that can characterize or indicate the image generation direction / type / style, etc. The so-called current description text may include the description text entered by the object when uploading the original image, the description text identified in the template image uploaded at the same time as the original image, the preset description text selected from the preset description text set when uploading the original image, or the description text obtained in other ways, etc. When the image generation information contains the current image description text, the current image description text may be directly used as the image description text corresponding to the original image. When the image generation information does not contain the current image description text, the image description text corresponding to the original image may be generated based on the image generation information. There may be multiple ways to generate the image description text corresponding to the original image, for example, based on the image generation request, the image description information may be identified in the original image, and the image description text may be generated according to the image generation information and the image description information.

[0070] The image description information may include information describing the content in the original image, such as content description text of the original image, image title, and image attribute information, etc. Based on the image generation request, there may be multiple ways to identify the image description information in the original image, such as identifying the image content in the original image according to the image generation request to obtain the content description text, determining the image title corresponding to the original image based on the content description text, obtaining the image attribute information of the original image, and using the content description text, image title, and image attribute information as the image description information.

[0071] The content description text may be text information describing the content in the original image. There may be multiple ways to identify the image content in the original image. For example, based on an image generation request, an image content recognition model may be used to identify the image content in the original image to obtain the content description text of the original image. There may be multiple network structures of the image content recognition model. For example, it may include PaLI (a visual language model) or other recognition models that can identify image content, and so on.

[0072] After identifying the content description text, the image title corresponding to the original image can be determined based on the content description text. There are many ways to determine the image title. For example, keywords can be identified in the content description text, and based on the keywords, an image identifier of the original image can be generated. Alternatively, an image title recognition model can be used to identify the image title in the content description text. Alternatively, an image title recognition model can be used to identify the image title in the original image and the content description text, and so on. The image title is an important representation method other than the image itself. The text modality and the visual modality together can form a multimodal, and the representation ability is much stronger than a single visual modality. In addition, it should be noted that the image title can be a coarse-grained image title or a fine-grained image title.

[0073] The image attribute information may include information representing the attributes of the original image, for example, the image type, the terminal identifier for uploading the original image, the upload time, the image size, or other attribute information.

[0074] After the image attribute information of the original image is obtained, the content description text, the image title and the image attribute information can be used as the image description information.

[0075] After the image description information is identified, the image description text can be generated according to the image generation information and the image description information. There are many ways to generate the image description text. For example, based on the image generation information, a target description text template can be selected from the preset description text template, the description text can be extracted from the image description information, and the description text can be added to the target description text template to obtain the image description text. Alternatively, the description text can be extracted from the image description information, and the image description text can be generated based on the description text and the image generation information, and so on.

[0076] Among them, for the first method of generating image description text, take the original image as the facial image of object A, the image generation information indicates that the image style of the target image is the oil painting of painter A, and the description text is the facial image of object A as an example. The target description text template can be "generate an image with the oil painting style of painter A". By adding the description text to the target description text template, the image description text can be obtained as "a facial image of object A similar to the oil painting style of painter A". The original image and the target image generated based on the image description text can be as follows Figure 4 As shown in the second method of generating image description text, take the original image as the facial image of object B, the image generation information indicates that the generated image is an image of a game character corresponding to Children's Day, and the description text is the text of the face of standard object B as an example. Based on the image generation information and the description text, the generated image description text can be "a cartoon image of the game character similar to the face of object B". The original image and the target image generated based on the image description text can be as follows Figure 5 shown.

[0077] (2) Segment the image description text to obtain a text word set.

[0078] For example, a Tokenizer may be used to segment the image description text to obtain multiple text words, which are then used as a text word set. Alternatively, other types of segmentation tools may be used to segment the image description text to obtain multiple text words, which are then used as a text word set, and so on.

[0079] 102. Perform feature extraction on the original image, and generate at least one negative word based on the extracted image features.

[0080] Among them, negative words represent the content deleted from the original image during the image generation process. Negative words can also be called negative prompt words, which are negative words (negative prompt words) corresponding to the text feature space level. They are not negative words in the image description text. They can be understood as the content of the original image that is not desired to appear in the generated target image. The text words (positive text words) after the image description text segmentation are adjusted at the feature level through negative words, so as to further reduce the overfitting result, and then make the image generation model compatible with various target domains (domains, which can include source domains and target domains).

[0081] There are many ways to extract features from the original image, which can be as follows:

[0082] For example, the background image can be removed from the original image to obtain the main image, and the image encoding network of the image generation model is used to encode the main image to obtain image features.

[0083] The subject image may include an image of the subject image retained after removing the background. The background is removed to prevent interference caused by the background.

[0084] It should be noted that by extracting features from the subject image, the content of the subject object in the original image that needs to be retained can be more accurately retained, for example, it can include appearance information or other content that needs to be retained.

[0085] After extracting the image features of the original image, at least one negative word can be generated based on the extracted image features. There are many ways to generate negative words, for example, a negative word generation network of an image generation model can be used to generate at least one negative word based on the image features.

[0086] Optionally, the trained image generation model may be pre-set by an operation and maintenance personnel, or may be obtained by self-training of the image generation device, that is, after the step of "using the image encoding network of the image generation model to encode the subject image to obtain image features", the image generation method may further include:

[0087] Obtain the original image sample and the description text sample corresponding to the original image sample, use the preset image generation model, generate the candidate image according to the original image sample, the description text sample and the preset image noise sample, determine the noise loss based on the preset image noise sample and the candidate image, and converge the preset image generation model according to the noise loss to obtain the image generation model, which can be specifically as follows:

[0088] (1) Obtain the original image sample and the description text sample corresponding to the original image sample.

[0089] For example, an initial original image sample may be obtained, and the background image may be removed from the initial original image to obtain an original image sample, image description information may be identified in the original image sample to obtain sample image description information, and a description text sample corresponding to the original image sample may be generated based on the sample image description information.

[0090] The method of obtaining the initial original image sample may be similar to the method of obtaining the original image, as described above, and will not be described in detail here.

[0091] It should be noted that when generating description text samples, during the training process, there may not be current description text or even image generation information. In this case, the description text samples corresponding to the original image samples can be directly generated based on the sample image description information. The description text samples can be generated randomly or as description text samples corresponding to image description information generated by a text generation model, etc.

[0092] (2) Using a preset image generation model, a candidate image is generated based on the original image sample, the description text sample and the preset image noise sample.

[0093] The preset image noise sample may be a pre-set noise sample used to add noise to an image or an image feature.

[0094] Among them, the preset image generation model can be a pre-set model for generating images. The preset image generation model can include an image encoding network, a text encoding network, a negative word generation network, a diffusion network and a feature decoding network. The image encoding network can be a network that encodes the features of an image. The network structure of the image encoding network can be multiple, for example, it can include CLIP (a multimodal visual network) or other networks that can encode images, etc. The text encoding network can be a network that encodes the features of text. The network structure of the text encoding network can be multiple, for example, it can include T5-XXL (a text processing network) or other networks that can encode text. The negative word generation network can be a network that generates negative words based on image features, and the diffusion network can be a network that denoises the image features after adding noise based on text features. The network structure of the diffusion network can be multiple, for example, it can include stable diffusion (an image generation network) or other diffusion networks that can denoise, etc. The feature decoding network can be a network that converts the denoised features into images. The feature decoding network and the image encoding network can correspond to the same feature space.

[0095] There are multiple ways to generate candidate images using a preset image generation model according to the original image sample, the description text sample and the preset image noise sample, which can be as follows:

[0096] For example, a preset image generation model may be used to extract features from original image samples and description text samples to obtain a sample feature set, the sample feature set including sample image features and sample text features, a preset image noise sample is added to the sample image features to obtain noisy sample image features, based on the sample text features, the noisy sample image features are denoised to obtain candidate content features and predicted noise features corresponding to the candidate content features, and the candidate content features are feature decoded to obtain a candidate image, specifically as follows:

[0097] S1: Use a preset image generation model to extract features from original image samples and description text samples to obtain a sample feature set.

[0098] For example, a preset image generation model can be used to feature the original image sample, and based on the extracted initial sample image features, at least one sample negative word can be generated, text features can be extracted from the descriptive text sample and the sample negative word respectively, and the extracted text features can be fused to obtain sample text features, and the initial sample image features can be adjusted according to the sample domain category of the original image sample to obtain sample image features, and the sample text features and sample image features can be used as a sample feature set.

[0099] There are many ways to characterize the original image samples using a preset image generation model. For example, the image encoding network of the preset image generation model can be used to characterize the original image samples to obtain initial sample image features corresponding to the original image samples.

[0100] After extracting features from the original image sample, at least one sample negative word can be generated based on the extracted initial sample image features. There are many ways to generate sample negative words, for example, a negative word generation network of a preset image generation model can be used to generate at least one sample negative word based on the initial sample image features.

[0101] Among them, there can be many ways to extract text features from the description text sample and the sample negative words respectively. For example, the description text sample is segmented to obtain at least one sample text word, and the text encoding network in the preset image generation model is used to perform feature encoding on the sample text words and the sample negative words respectively to obtain the positive sample text features corresponding to the sample text words and the negative sample text features corresponding to the sample negative words.

[0102] After extracting text features from the description text sample and the sample negative word, the extracted text features can be fused to obtain sample text features. There are many ways to fuse the extracted text features. For example, feature fusion parameters corresponding to a preset text feature space can be obtained, and according to the feature fusion parameters, positive sample text features and negative sample text features can be fused to obtain sample text features.

[0103] Among them, the feature fusion parameter can be a fusion weight between the negative sample text feature and the positive sample text feature, which is used to fine-tune the positive text feature. According to the feature fusion parameter, there are many ways to fuse the negative sample text feature with the positive sample text feature. For example, the feature difference between the negative sample text feature and the positive sample text feature is calculated to obtain the sample feature difference, and the feature fusion parameter is fused with the sample feature difference to obtain the adjusted positive sample text feature and the negative sample text feature. The sample text feature can be obtained as shown in formula (1). Specifically, it can be as follows:

[0104]

[0105] in, and f m (z p , z n ) is the sample text feature, z p is the positive sample feature, z n is the negative sample text feature, and γ is the feature fusion parameter.

[0106] The sample domain category may be the domain category of the original image sample. According to the sample domain category of the original image sample, there may be multiple ways to adjust the initial sample image features. For example, when the sample domain category is the target domain category, the initial sample image features are adjusted according to the sample object identifiers marked in the original image sample to obtain the sample image features. When the sample domain category is a domain category other than the target domain category, the feature values ​​of the initial sample image features are set to zero to obtain the sample image features.

[0107] The target domain may be a target domain, and the domain categories other than the target domain category may be other domain categories in the pre-training sample. When the sample domain category is the target domain category, there may be multiple ways to adjust the initial sample image features according to the sample object identifiers marked in the original image samples. For example, according to the initial sample image features, the object identifiers of the objects in the original image samples are predicted to obtain the predicted probability. When the predicted probability is less than a preset probability threshold, the average sample image features corresponding to the sample object identifiers are obtained, and the average sample image features are used as sample image features. When the predicted probability is greater than or equal to the preset probability threshold, the initial sample image features are used as sample image features.

[0108] The predicted probability includes the probability that the object identifier is the sample object identifier. There are many ways to predict the object identifier of the object in the original image sample based on the initial sample image features. For example, the object identifier corresponding to the original image sample identified by the image recognition model in the initial sample image features can be used to obtain a candidate object identifier set and a predicted probability set corresponding to the candidate object identifier set, and the predicted probability that the predicted object identifier is the sample object identifier is screened out from the predicted probability set.

[0109] When the predicted probability is less than the preset probability threshold, it means that the accuracy of the extracted initial sample image features in representing the objects in the original image samples is low. At this time, the average sample image features corresponding to the sample object identifiers can be obtained and used as sample image features. The average image features can be pre-set image features, or they can be image features corresponding to the mean of the initial sample image features of the original image samples corresponding to the sample object identifiers in the current training set. When the predicted probability is greater than or equal to the preset probability threshold, it means that the accuracy of the extracted initial sample image features meets the requirements, and the initial sample image features can be used as sample image features, as shown in formula (2). Specifically, it can be as follows:

[0110]

[0111] Among them, z is the sample image feature, x is the original image sample, is the average sample image feature, o is the initial sample image feature, and t is the description text sample.

[0112] Among them, the adjustment of the initial sample image features is mainly aimed at the initial sample image features with low accuracy. After replacing the initial sample image features with the average sample image features of the sample object identifier, the accuracy of the sample image features can be improved, so that the diffusion model of the image generation model can extract key information from the adjusted sample image features and generate better images.

[0113] Among them, when the sample domain category is a domain category other than the target domain category, it means that the original image sample at this time can be pre-trained data outside the target domain. Since the representation ability of image features is stronger than that of text features, the image generation model will eventually fail to control the text. Therefore, for the original image samples of domain categories other than the non-target domain category, the feature value of the corresponding initial sample image feature can be set to zero, that is, the corresponding initial sample image feature is set to empty, thereby obtaining the sample image feature. Therefore, the sample image features corresponding to different domain categories can be different, as shown in formula (3):

[0114]

[0115] Among them, z is the sample image feature, x is the original image sample, t is the description text sample, and o is the sample image feature of the target domain category (domain-speccifc). is an empty set, that is, the sample image features of other domain categories.

[0116] It should be noted that since the amount of training data in the target domain is often much smaller than the amount of pre-training data, if the training data in the target domain is used to train the preset image generation model, it will lead to overfitting, causing the preset image generation model to forget the information learned in the pre-training process. If the training data in the target domain and the pre-training data are directly combined for training, the conditions of the image features will not take effect due to the difference in the quantity of the two types of data. Therefore, this scheme adopts a normalized joint training method. For the training data in the target domain, the initial sample image features with low accuracy are replaced with the average sample image features of the corresponding object, thereby improving the image generation quality. For the training data in the non-target domain, since the representation ability of image features is stronger than that of text features, the initial sample image features of this part of the training data can be set to empty, thereby improving the image generation model's control over the text.

[0117] After the sample text features and the sample image features are determined, the sample text features and the sample image features can be used as a sample feature set.

[0118] S2. Add a preset image noise sample to the sample image feature to obtain the sample image feature after the noise is added.

[0119] For example, a preset image noise sample can be obtained, and the preset image noise sample can be added to the sample image feature to obtain the sample image feature after adding noise. Alternatively, a preset image noise sample can be obtained, and the preset image noise sample can be feature encoded to obtain the sample noise feature, and the sample noise feature can be fused with the sample image feature to obtain the sample image feature after adding noise, and so on.

[0120] S3. Based on the sample text features, the sample image features after adding noise are denoised to obtain candidate content features and predicted noise features corresponding to the candidate content features, and the candidate content features are decoded to obtain a candidate image.

[0121] There are many ways to reduce the noise of the sample image features after adding noise based on the sample text features, which can be specifically as follows:

[0122] For example, a diffusion network of a preset image generation model can be used to fuse sample text features and noisy sample image features to obtain fused content features, and predict predicted noise features corresponding to the noisy sample image features, and based on the predicted noise features, the fused content features are denoised to obtain candidate content features, and the predicted noise features are used as predicted noise features corresponding to the candidate content features.

[0123] Taking the diffusion network as stable diffusion (a diffusion network) as an example, we can add cross-attention mechanism to the UNet backbone network (a network used for estimating and removing noise in stable diffusion) to achieve the fusion of sample text features and sample image features after noise addition and predict noise features. The network structure of stable diffusion can be as follows: Figure 6 The method of fusing the sample text features with the sample image features after noise addition can be shown in formula (4), which can be specifically as follows:

[0124]

[0125] Among them, Attention(Q,K,V) is the fused content feature, Q is the query vector, K is the key vector, V is the value vector, and d is the dimension of the value vector. is the initial query vector generated based on the features of the sample image after adding noise, is an intermediate representation of Unet, is the initial value vector generated based on the sample image features after adding noise, τ θ (y) is the domain-specific encoder τ θ An intermediate representation of the sample text feature map is mapped, and y is the sample text feature. At this time, the sample text feature can be integrated into the corresponding intermediate layer of Unet as a control information.

[0126] After denoising the features of the sample image after adding noise, the denoised candidate content features can be feature decoded to obtain a candidate image. There are many ways to feature decode the candidate content features. For example, a feature decoding network of an image generation model can be used to feature decode the candidate content features to obtain a candidate image.

[0127] (3) Based on the preset image noise samples and the candidate images, the noise loss is determined, and the preset image generation model is converged according to the noise loss to obtain the image generation model.

[0128] The noise loss may include the loss caused by the difference between the image noise predicted by the diffusion network and the preset image noise sample.

[0129] There are multiple ways to determine the noise loss based on the preset image noise sample and the candidate image, which can be as follows:

[0130] For example, the noise loss may be determined based on the predicted noise feature corresponding to the candidate content feature and the noise feature of the preset image noise sample. For example, the noise feature of the preset image noise sample may be obtained, and the feature distance between the noise feature of the preset image noise sample and the predicted noise feature may be calculated, thereby obtaining the noise loss. Specifically, the noise loss may be as shown in formula (5), which may be as follows:

[0131]

[0132] Among them, Loss is the noise loss, x t is the original image sample, T is the image description text, c is the sample text feature, ∈ θ (x t ,t,c) is the predicted noise feature, ∈ is the noise feature of the preset noise sample.

[0133] After determining the noise loss, the preset image generation model can be converged based on the noise loss to obtain the image generation model. There are many ways to converge the preset image generation model. For example, a gradient descent algorithm can be used to update the network parameters of the preset image generation model based on the noise loss to obtain an updated preset image generation model. The updated preset image generation model is used as the preset image generation model, and the steps of obtaining the original image sample and the description text sample corresponding to the original image sample are returned until the preset image generation model converges to obtain the trained image generation model. Alternatively, other network parameter update algorithms can be used to update the network parameters of the preset image generation model based on the noise loss to obtain an updated preset image generation model. The updated preset image generation model is used as the preset image generation model, and the steps of obtaining the original image sample and the description text sample corresponding to the original image sample are returned until the preset image generation model converges to obtain the trained image generation model, and so on.

[0134] Among them, the process of training the preset image generation model can be as follows Figure 7As shown in FIG, the training may include three parts. The first part is a pre-trained generative model that supports text conditions. The second part is a feature extraction process. For the original image samples in the target domain, on the one hand, the image encoding network (Object encoder) is used to encode the content to be retained in the original image samples, thereby obtaining sample image features (e.g., sample facial features). On the other hand, the description text samples identified in the original image samples by the text description system (captioning) are extracted from the description text samples by the text encoding network (Text encoder) to obtain positive sample text features. The third part is the learning of negative words (negative prompt words). At least one sample negative word corresponding to the generated text feature space is then feature encoded by the text encoding network (Text encoder) to obtain negative sample text features. Then, the positive sample text features are adjusted by the negative sample text features to obtain sample text features. Finally, the diffusion network (Diffusion The cross-attention plug-in in the model) fuses the sample text features with the sample image features based on the preset image noise samples to obtain the fused content features, and predicts the noise features. Then, the fused content features are denoised based on the predicted noise features to obtain the final generated candidate image. Then, the noise loss is determined based on the predicted noise features and the preset image noise samples, and the preset image generation model is converged based on the noise loss to obtain the trained image generation model.

[0135] After training the preset image generation model, the image encoding network of the trained image generation model can be used to encode the subject image to obtain image features, and the negative word generation network of the image generation model can be used to generate at least one negative word based on the image features.

[0136] It should be noted that the execution entities for training the preset image generation model and using the preset image generation model to generate images may be the same or different. For example, taking server A as an example for training the preset image generation model, server A may use the trained image generation model to generate a target image based on the original image and the image description text, and server A may also send the trained image generation model to at least one server B or terminal C, so that server B or terminal C may use the trained image generation model to generate a target image based on the original image and the image description text, and so on.

[0137] 103. Perform feature extraction on the text word set and the negative words respectively to obtain positive text features corresponding to the text word set and negative text features corresponding to the negative words.

[0138] Among them, the positive text features can be content features that are expected to appear in the generated image during the image generation process, and the negative text features can be features of negative content that are not expected to appear in the generated image during the image generation process. The so-called negative content can be content in the original image that is not expected to appear during the image generation process.

[0139] There are many ways to extract features from text word sets and negative words, which can be as follows:

[0140] For example, the text encoding network of the image generation model can be used to extract features of text words in a text word set to obtain positive text features, and the text encoding network of the image generation model can be used to extract features of negative words to obtain negative text features.

[0141] 104. Use the negative text features to adjust the positive text features to obtain the target text features.

[0142] For example, a feature fusion parameter corresponding to a preset text feature space may be obtained, and the negative text feature and the positive text feature may be fused according to the feature fusion parameter to obtain the target text feature.

[0143] There are many ways to fuse the negative text features with the positive text features. For example, the feature difference between the negative text features and the positive text features can be calculated, and the feature fusion parameter and the feature difference can be fused to obtain the adjusted positive text features. The adjusted positive text features and the negative text features can be fused to obtain the target text features, as shown in formula (6). Specifically, it can be as follows:

[0144]

[0145] in, is the target text feature, z p1 is the positive text feature, z n1 is the negative text feature, and γ is the feature fusion parameter.

[0146] 105. Generate a target image corresponding to the original image based on the target text features and image features.

[0147] For example, the domain category of the original image can be obtained, and the image features can be adjusted according to the domain category to obtain the target image features, preset image noise can be added to the target image features to obtain the noisy image features, and based on the target text features, the noisy image features can be denoised to obtain the target content features, and the target content features can be feature decoded to obtain the target content features corresponding to the original image.

[0148] Among them, there are many ways to adjust the image features according to the domain category. For example, when the domain category is the target domain category, the image features are used as target image features. When the domain category is a domain category other than the target domain category, the feature values ​​of the image features are set to zero to obtain the target image features.

[0149] After adjusting the image features, a preset image noise can be added to the target image features to obtain the noisy image features, and the noisy image features can be denoised based on the target text features to obtain the target content features. The method of obtaining the target content features is similar to the method of obtaining the candidate content features, as described above, and will not be repeated here.

[0150] After denoising the image features after adding noise, the target content features can be feature decoded to obtain the target image corresponding to the original image. There are many ways to feature decode the target content features. For example, the decoding network of the image generation model can be used to feature decode the target content features to obtain the target image corresponding to the original image.

[0151] Among them, for the image customized editing scenario, the object can input an image as the original image, and then generate image description text based on the input original image, and use the image generation model to generate a target image corresponding to the original image based on the image description text and the original image. Alternatively, the object can also input an image as the original image, and can also input editing text for the original image, and use the editing text as the image description text corresponding to the original image, and use the image generation model to generate a target image corresponding to the original image based on the image description text and the original image, and so on.

[0152] Among them, in this scheme, the content that needs to be retained in the original image can be accurately learned through an original image, and there is no need to fine-tune the model or train a corresponding model for each object separately, thereby realizing one-shot image generation, that is, only one forward pass is needed to generate the target image, and there is no need to fine-tune the model for each object, so that the target image can be generated more efficiently. Taking the original images as various animal images as an example, the comparison between the generated target image and the original image can be as follows Figure 8 As shown, it can be found that this solution accurately retains the appearance information of the target object and other content that needs to be retained.

[0153] Among them, one-shot means that during the process of model learning (training), one or a very small number of samples of a certain category are used, while during testing or application, the model can use other samples of the category.

[0154] Optionally, in some embodiments, after the target image corresponding to the original image is generated, the target object may be displayed, or the target image may be sent to a terminal so that the terminal displays the target image.

[0155] After displaying the target image, feedback information for the target image can also be received, and then text information can be extracted from the feedback information, the target image can be used as the original image, and the text information can be used as the image description text corresponding to the target image, and the step of extracting features from the original image can be returned to obtain an updated target image.

[0156] From the above, it can be seen that the embodiment of the present application obtains the original image and the image description text corresponding to the original image, and performs word segmentation on the image description text to obtain a text word set, then performs feature extraction on the original image, and generates at least one negative word based on the extracted image features, and performs feature extraction on the text word set and the negative words respectively to obtain positive text features corresponding to the text word set and negative text features corresponding to the negative words, and then uses the negative text features to adjust the positive text features to obtain target text features, and generates a target image corresponding to the original image based on the target text features and the image features; since the scheme can generate a corresponding target image for an original image in different fields (source domain and target domain), there is no need to fine-tune each object, and the positive text features of the image description text can also be adjusted by the negative text features of the negative words to avoid overfitting, so that the content that needs to be retained in images of different fields can be accurately retained, and the accuracy and compatibility of image generation can be increased, thereby improving the efficiency of image generation.

[0157] According to the method described in the above embodiment, the following is further described in detail with examples.

[0158] In this embodiment, the image generation device is specifically integrated in an electronic device, the electronic device is a server, and the diffusion network of the image generation model is a stable diffusion.

[0159] (a) Training of image generation model.

[0160] (1) The server obtains the original image sample and the description text sample corresponding to the original image sample.

[0161] For example, the server can obtain an initial original image sample, and remove the background image from the initial original image to obtain an original image sample, identify image description information in the original image sample to obtain sample image description information, and generate a description text sample corresponding to the original image sample based on the sample image description information. For details, please refer to the above description, which will not be repeated here.

[0162] (2) The server uses a preset image generation model to generate candidate images based on the original image samples, description text samples, and preset image noise samples.

[0163] For example, the server may use an image encoding network of a preset image generation model to perform feature encoding on the original image sample to obtain an initial sample image feature corresponding to the original image sample. A negative word generation network of a preset image generation model is used to generate at least one sample negative word based on the initial sample image feature.

[0164] The server can obtain the feature fusion parameters corresponding to the preset text feature space, calculate the feature difference between the negative sample text feature and the positive sample text feature, obtain the sample feature difference, fuse the feature fusion parameters with the sample feature difference, obtain the adjusted positive sample text feature, fuse it with the negative sample text feature, and obtain the sample text feature, which can be specifically shown in formula (1).

[0165] When the sample domain category is the target domain category, the server predicts the object identifier of the object in the original image sample based on the initial sample image feature to obtain the predicted probability. When the predicted probability is less than the preset probability threshold, the server obtains the average sample image feature corresponding to the sample object identifier and uses the average sample image feature as the sample image feature. When the predicted probability is greater than or equal to the preset probability threshold, the server uses the initial sample image feature as the sample image feature, which can be specifically shown in formula (2). When the sample domain category is a domain category other than the target domain category, the feature value of the initial sample image feature is set to zero to obtain the sample image feature, which can be specifically shown in formula (3). The sample text feature and the sample image feature are used as a sample feature set.

[0166] The server obtains a preset image noise sample, and may add the preset image noise sample to the sample image feature to obtain the sample image feature after adding noise. Alternatively, the server may obtain a preset image noise sample, perform feature encoding on the preset image noise sample to obtain the sample noise feature, fuse the sample noise feature with the sample image feature to obtain the sample image feature after adding noise, and so on.

[0167] The server can use the stable diffusion of the preset image generation model to fuse the sample text features and the sample image features after noise addition to obtain the fused content features, and predict the predicted noise features corresponding to the sample image features after noise addition, and based on the predicted noise features, denoise the fused content features to obtain candidate content features, and use the predicted noise features as the predicted noise features corresponding to the candidate content features. The feature fusion process can be shown as formula (4).

[0168] The server may use a feature decoding network of an image generation model to perform feature decoding on candidate content features to obtain a candidate image.

[0169] (3) The server determines the noise loss based on the preset image noise sample and the candidate image, and converges the preset image generation model according to the noise loss to obtain the image generation model.

[0170] For example, the server may obtain the noise feature of the preset image noise sample, and calculate the feature distance between the noise feature of the preset image noise sample and the predicted noise feature, thereby obtaining the noise loss, which may be specifically shown in formula (5).

[0171] The server can use a gradient descent algorithm to update the network parameters of the preset image generation model based on noise loss to obtain an updated preset image generation model, use the updated preset image generation model as the preset image generation model, and return to execute the step of obtaining the original image sample and the description text sample corresponding to the original image sample until the preset image generation model converges to obtain the trained image generation model. Alternatively, other network parameter updating algorithms can be used to update the network parameters of the preset image generation model based on noise loss to obtain an updated preset image generation model, use the updated preset image generation model as the preset image generation model, and return to execute the step of obtaining the original image sample and the description text sample corresponding to the original image sample until the preset image generation model converges to obtain the trained image generation model, and so on.

[0172] (ii) The server uses the trained image generation model to generate a target image based on the original image and image description text.

[0173] Among them, the trained image generation model may include an image encoding network, a text encoding network, a negative word generation network, a diffusion network (stable diffusion), and the like.

[0174] like Fig. 9 As shown, an image generation method, the specific process is as follows:

[0175] 201. The server obtains an original image and an image description text corresponding to the original image.

[0176] For example, the server may accept an image generation request that carries an original image and image generation information corresponding to the original image. When the image generation information contains the current image description text, the current image description text is used as the image description text corresponding to the original image. When the image generation information does not contain the current image description text, based on the image generation request, PaLI is used to identify the image content in the original image to obtain the content description text of the original image; the keywords identified in the content description text are used, and based on the keywords, an image identifier of the original image is generated, or an image title recognition model may be used to identify the image title in the content description text, or an image title recognition model may be used to identify the image title in the original image and the content description text; the image attribute information of the original image is obtained, and the content description text, image title and image attribute information are used as image description information.

[0177] The server can filter out a target description text template from a preset description text template based on the image generation information, extract the description text from the image description information, and add the description text to the target description text template to obtain the image description text. Alternatively, the server can extract the description text from the image description information and generate the image description text based on the description text and the image generation information, and so on.

[0178] 202. The server performs word segmentation on the image description text to obtain a text word set.

[0179] For example, the server may use a Tokenizer to segment the image description text to obtain multiple text words, and use the multiple text words as a text word set. Alternatively, the server may use other types of segmentation tools to segment the image description text to obtain multiple text words, and use the multiple text words as a text word set, and so on.

[0180] 203. The server extracts features from the original image to obtain image features.

[0181] For example, the server can remove the background image from the original image to obtain the main image, and use the image encoding network of the image generation model to encode the main image to obtain image features.

[0182] 204. The server generates at least one negative word based on the image features.

[0183] For example, the server may use a negative word generation network of an image generation model to generate at least one negative word based on image features.

[0184] 205. The server extracts features from the text word set and the negative words respectively to obtain positive text features corresponding to the text word set and negative text features corresponding to the negative words.

[0185] For example, the server may use the text encoding network of the image generation model to perform feature extraction on text words in the text word set to obtain positive text features, and use the text encoding network of the image generation model to perform feature extraction on negative words to obtain negative text features.

[0186] 206. The server adjusts the positive text feature using the negative text feature to obtain the target text feature.

[0187] For example, the server can obtain the feature fusion parameters corresponding to the preset text feature space, calculate the feature difference between the negative text feature and the positive text feature, fuse the feature fusion parameters with the feature difference to obtain the adjusted positive text feature, and fuse the adjusted positive text feature with the negative text feature to obtain the target text feature, as shown in formula (6).

[0188] 207. The server generates a target image corresponding to the original image based on the target text features and the image features.

[0189] For example, the server can obtain the domain category of the original image, and when the domain category is the target domain category, use the image features as the target image features; when the domain category is a domain category other than the target domain category, set the feature value of the image features to zero to obtain the target image features.

[0190] The server adds preset image noise to the target image features to obtain the noisy image features, and uses the stable diffusion of the image generation model to denoise the noisy image features based on the target text features to obtain the target content features. The details are described above and will not be repeated here.

[0191] The server can use the decoding network of the image generation model to decode the target content features, thereby obtaining a target image corresponding to the original image.

[0192] Optionally, in some embodiments, the server may further display the target object, or may further send the target image to the terminal so that the terminal displays the target image.

[0193] In some embodiments, after the server or terminal displays the target image, the server may also receive feedback information for the target image, and then extract text information from the feedback information, use the target image as the original image, and use the text information as the image description text corresponding to the target image, and return to execute the step of extracting features from the original image to obtain an updated target image.

[0194] From the above, it can be seen that in the embodiment of the present application, after obtaining the original image and the image description text corresponding to the original image, and segmenting the image description text to obtain a text word set, the server performs feature extraction on the original image, and generates at least one negative word based on the extracted image features, and performs feature extraction on the text word set and the negative words respectively to obtain positive text features corresponding to the text word set and negative text features corresponding to the negative words, and then, the positive text features are adjusted using the negative text features to obtain target text features, and the target image corresponding to the original image is generated based on the target text features and the image features; since the scheme can generate a corresponding target image for an original image in different fields (source domain and target domain), there is no need to fine-tune each object, and the positive text features of the image description text can also be adjusted by the negative text features of the negative words to avoid overfitting, so that the content that needs to be retained in images of different fields can be accurately retained, and the accuracy and compatibility of image generation can be increased, thereby improving the efficiency of image generation.

[0195] In order to better implement the above method, an embodiment of the present invention further provides an image generating device, which can be integrated into an electronic device, such as a server or a terminal. The terminal may include a tablet computer, a laptop computer and / or a personal computer.

[0196] For example, Fig.10 As shown, the image generating device may include an acquiring unit 301, a first extracting unit 302, a second extracting unit 303, an adjusting unit 304 and a generating unit 305, as follows:

[0197] (1) Acquisition unit 301;

[0198] The acquisition unit 301 is used to acquire the original image and the image description text corresponding to the original image, and segment the image description text to obtain a text word set.

[0199] For example, the acquisition unit 301 can be specifically used to accept an image generation request, which carries an original image and image generation information corresponding to the original image. When the image generation information contains the current image description text, the current image description text is used as the image description text corresponding to the original image. When the image generation information does not contain the current image description text, the image description text corresponding to the original image is generated based on the image generation information and the original image. The image description text is segmented using a Tokenizer to obtain multiple text words, and the multiple text words are used as a text word set.

[0200] (2) a first extraction unit 302;

[0201] The first extraction unit 302 is used to extract features from the original image and generate at least one negative word based on the extracted image features, where the negative word represents the content deleted from the original image during the image generation process.

[0202] For example, the first extraction unit 302 can be specifically used to remove the background image in the original image to obtain the main image, use the image encoding network of the image generation model to encode the main image to obtain image features, and use the negative word generation network of the image generation model to generate at least one negative word based on the image features.

[0203] (3) a second extraction unit 303;

[0204] The second extraction unit 303 is used to perform feature extraction on the text word set and the negative words respectively to obtain positive text features corresponding to the text word set and negative text features corresponding to the negative words.

[0205] For example, the second extraction unit 303 can be specifically used to use the text encoding network of the image generation model to perform feature extraction on text words in the text word set to obtain positive text features, and use the text encoding network of the image generation model to perform feature extraction on negative words to obtain negative text features.

[0206] (4) adjustment unit 304;

[0207] The adjustment unit 304 is used to adjust the positive text feature using the negative text feature to obtain the target text feature.

[0208] For example, the adjustment unit 304 can be specifically used to obtain feature fusion parameters corresponding to a preset text feature space, calculate the feature difference between the negative text feature and the positive text feature, fuse the feature fusion parameters with the feature difference to obtain an adjusted positive text feature, and fuse the adjusted positive text feature with the negative text feature to obtain a target text feature.

[0209] (5) generating unit 305;

[0210] The generating unit 305 is used to generate a target image corresponding to the original image based on the target text features and the image features.

[0211] For example, the generation unit 305 can be specifically used to obtain the domain category of the original image, and adjust the image features according to the domain category to obtain the target image features, add preset image noise to the target image features to obtain the noisy image features, and denoise the noisy image features based on the target text features to obtain the target content features, perform feature decoding on the target content features, and obtain the target content features corresponding to the original image.

[0212] Optionally, the image generating device may further include a training unit 306, such as Fig.11 As shown, the specific details can be as follows:

[0213] The training unit 306 is used to train the preset image generation model to obtain a trained image generation model.

[0214] For example, the training unit 306 can be specifically used to obtain an original image sample and a description text sample corresponding to the original image sample, use a preset image generation model, generate a candidate image based on the original image sample, the description text sample and the preset image noise sample, determine the noise loss based on the preset image noise sample and the candidate image, and converge the preset image generation model according to the noise loss to obtain an image generation model.

[0215] In specific implementation, the above units can be implemented as independent entities, or can be arbitrarily combined to be implemented as the same or several entities. The specific implementation of the above units can refer to the previous method embodiments, which will not be repeated here.

[0216] As can be seen from the above, in this embodiment, after the acquisition unit 301 acquires the original image and the image description text corresponding to the original image, and performs word segmentation on the image description text to obtain a text word set, the first extraction unit 302 performs feature extraction on the original image, and generates at least one negative word based on the extracted image features, and the second extraction unit 303 performs feature extraction on the text word set and the negative word respectively to obtain positive text features corresponding to the text word set and negative text features corresponding to the negative word, and then, the adjustment unit 304 adjusts the positive text features using the negative text features to obtain target text features, and the generation unit 305 generates a target image corresponding to the original image based on the target text features and the image features; since the scheme can generate a corresponding target image for an original image in different fields (source domain and target domain), there is no need to fine-tune each object, and the positive text features of the image description text can also be adjusted by the negative text features of the negative words to avoid overfitting, so that the content that needs to be retained in images of different fields can be accurately retained, and the accuracy and compatibility of image generation can be increased, so that the efficiency of image generation can be improved.

[0217] The embodiment of the present invention further provides an electronic device, such as Fig.12 As shown, it shows a schematic diagram of the structure of an electronic device involved in an embodiment of the present invention, specifically:

[0218] The electronic device may include components such as a processor 401 with one or more processing cores, a memory 402 with one or more computer-readable storage media, a power supply 403, and an input unit 404. Those skilled in the art will appreciate that Fig.12The electronic device structure shown in the figure does not constitute a limitation on the electronic device, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0219] The processor 401 is the control center of the electronic device, and uses various interfaces and lines to connect various parts of the entire electronic device. By running or executing software programs and / or modules stored in the memory 402, and calling data stored in the memory 402, the processor 401 performs various functions of the electronic device and processes data. Optionally, the processor 401 may include one or more processing cores; preferably, the processor 401 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface, and application programs, and the modem processor mainly processes wireless communications. It is understandable that the above-mentioned modem processor may not be integrated into the processor 401.

[0220] The memory 402 can be used to store software programs and modules. The processor 401 executes various functional applications and data processing by running the software programs and modules stored in the memory 402. The memory 402 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area may store data created according to the use of the electronic device, etc. In addition, the memory 402 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices. Accordingly, the memory 402 may also include a memory controller to provide the processor 401 with access to the memory 402.

[0221] The electronic device also includes a power supply 403 for supplying power to each component. Preferably, the power supply 403 can be logically connected to the processor 401 through a power management system, so as to manage charging, discharging, power consumption and other functions through the power management system. The power supply 403 can also include one or more DC or AC power supplies, recharging systems, power failure detection circuits, power converters or inverters, power status indicators and other arbitrary components.

[0222] The electronic device may further include an input unit 404, which may be used to receive input digital or character information and generate keyboard, mouse, joystick, optical or trackball signal input related to user settings and function control.

[0223] Although not shown, the electronic device may further include a display unit, etc., which will not be described in detail herein. Specifically in this embodiment, the processor 401 in the electronic device will load the executable files corresponding to the processes of one or more application programs into the memory 402 according to the following instructions, and the processor 401 will run the application programs stored in the memory 402, thereby realizing various functions, as follows:

[0224] The original image and the image description text corresponding to the original image are obtained, and the image description text is segmented to obtain a text word set, and features are extracted for the original image, and at least one negative word is generated based on the extracted image features, and the negative word represents the content deleted from the original image during the image generation process; the text word set and the negative words are respectively extracted to obtain positive text features corresponding to the text word set and negative text features corresponding to the negative words, and the positive text features are adjusted using the negative text features to obtain target text features, and a target image corresponding to the original image is generated based on the target text features and the image features.

[0225] For example, an electronic device receives an image generation request, which carries an original image and image generation information corresponding to the original image. When the image generation information contains the current image description text, the current image description text is used as the image description text corresponding to the original image. When the image generation information does not contain the current image description text, the image description text corresponding to the original image is generated based on the image generation information and the original image. A Tokenizer is used to segment the image description text to obtain multiple text words, and the multiple text words are used as a text word set. The background image is removed from the original image to obtain a main image, and the image encoding network of the image generation model is used to encode the main image to obtain image features. The negative word generation network of the image generation model is used to generate at least one negative word based on the image features. The text encoding network of the image generation model is used to extract features from the text words in the text word set to obtain positive text features, and the text encoding network of the image generation model is used to extract features from negative words to obtain negative text features. The feature fusion parameters corresponding to the preset text feature space are obtained, the feature difference between the negative text feature and the positive text feature is calculated, the feature fusion parameters are fused with the feature difference to obtain the adjusted positive text feature, and the adjusted positive text feature is fused with the negative text feature to obtain the target text feature. The domain category of the original image is obtained, and the image feature is adjusted according to the domain category to obtain the target image feature, and the preset image noise is added to the target image feature to obtain the noisy image feature, and the noisy image feature is denoised based on the target text feature to obtain the target content feature, and the target content feature is feature decoded to obtain the target content feature corresponding to the original image.

[0226] The specific implementation of the above operations can be found in the previous embodiments and will not be described in detail here.

[0227] From the above, it can be seen that the embodiment of the present application obtains the original image and the image description text corresponding to the original image, and performs word segmentation on the image description text to obtain a text word set, then performs feature extraction on the original image, and generates at least one negative word based on the extracted image features, and performs feature extraction on the text word set and the negative words respectively to obtain positive text features corresponding to the text word set and negative text features corresponding to the negative words, and then uses the negative text features to adjust the positive text features to obtain target text features, and generates a target image corresponding to the original image based on the target text features and the image features; since the scheme can generate a corresponding target image for an original image in different fields (source domain and target domain), there is no need to fine-tune each object, and the positive text features of the image description text can also be adjusted by the negative text features of the negative words to avoid overfitting, so that the content that needs to be retained in images of different fields can be accurately retained, and the accuracy and compatibility of image generation can be increased, thereby improving the efficiency of image generation.

[0228] A person of ordinary skill in the art will appreciate that all or part of the steps in the various methods of the above embodiments may be completed by instructions, or by controlling related hardware through instructions. The instructions may be stored in a computer-readable storage medium and loaded and executed by a processor.

[0229] To this end, an embodiment of the present invention provides a computer-readable storage medium, in which a plurality of instructions are stored, and the instructions can be loaded by a processor to execute the steps in any one of the image generation methods provided in the embodiments of the present invention. For example, the instructions can execute the following steps:

[0230] The original image and the image description text corresponding to the original image are obtained, and the image description text is segmented to obtain a text word set, and features are extracted for the original image, and at least one negative word is generated based on the extracted image features, and the negative word represents the content deleted from the original image during the image generation process; the text word set and the negative words are respectively extracted to obtain positive text features corresponding to the text word set and negative text features corresponding to the negative words, and the positive text features are adjusted using the negative text features to obtain target text features, and a target image corresponding to the original image is generated based on the target text features and the image features.

[0231] For example, an electronic device receives an image generation request, which carries an original image and image generation information corresponding to the original image. When the image generation information contains the current image description text, the current image description text is used as the image description text corresponding to the original image. When the image generation information does not contain the current image description text, the image description text corresponding to the original image is generated based on the image generation information and the original image. A Tokenizer is used to segment the image description text to obtain multiple text words, and the multiple text words are used as a text word set. The background image is removed from the original image to obtain a main image, and the image encoding network of the image generation model is used to encode the main image to obtain image features. The negative word generation network of the image generation model is used to generate at least one negative word based on the image features. The text encoding network of the image generation model is used to extract features from the text words in the text word set to obtain positive text features, and the text encoding network of the image generation model is used to extract features from negative words to obtain negative text features. The feature fusion parameters corresponding to the preset text feature space are obtained, the feature difference between the negative text feature and the positive text feature is calculated, the feature fusion parameters are fused with the feature difference to obtain the adjusted positive text feature, and the adjusted positive text feature is fused with the negative text feature to obtain the target text feature. The domain category of the original image is obtained, and the image feature is adjusted according to the domain category to obtain the target image feature, and the preset image noise is added to the target image feature to obtain the noisy image feature, and the noisy image feature is denoised based on the target text feature to obtain the target content feature, and the target content feature is feature decoded to obtain the target content feature corresponding to the original image.

[0232] The specific implementation of the above operations can be found in the previous embodiments, which will not be described in detail here.

[0233] The computer-readable storage medium may include: a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0234] Since the instructions stored in the computer-readable storage medium can execute the steps in any image generation method provided in the embodiments of the present invention, the beneficial effects that can be achieved by any image generation method provided in the embodiments of the present invention can be achieved. Please refer to the previous embodiments for details and will not be repeated here.

[0235] According to one aspect of the present application, a computer program product or a computer program is provided, the computer program product or the computer program includes computer instructions, and the computer instructions are stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the methods provided in various optional implementations of the above-mentioned image generation aspect or image customization editing aspect.

[0236] The above is a detailed introduction to an image generation method, device, electronic device and computer-readable storage medium provided by an embodiment of the present invention. Specific examples are used herein to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation methods and application scopes. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. An image generation method, characterized in that: include: Acquire an original image and an image description text corresponding to the original image, and segment the image description text to obtain a text word set; Extracting features from the original image, and generating at least one negative word based on the extracted image features, wherein the negative word represents content deleted from the original image during the image generation process; Performing feature extraction on the text word set and the negative words respectively to obtain positive text features corresponding to the text word set and negative text features corresponding to the negative words; Using the negative text features to adjust the positive text features to obtain target text features; A target image corresponding to the original image is generated based on the target text feature and the image feature.

2. The image generation method according to claim 1, characterized in that: The step of adjusting the positive text feature by using the negative text feature to obtain the target text feature includes: Obtain feature fusion parameters corresponding to the preset text feature space; According to the feature fusion parameter, the negative text feature and the positive text feature are fused to obtain the target text feature.

3. The image generation method according to claim 2, characterized in that: The step of fusing the negative text feature with the positive text feature according to the feature fusion parameter to obtain the target text feature includes: Calculating a feature difference between the negative text feature and the positive text feature; The feature fusion parameter is fused with the feature difference to obtain an adjusted positive text feature; The adjusted positive text feature is fused with the negative text feature to obtain the target text feature.

4. The image generation method according to claim 1, characterized in that: The step of generating a target image corresponding to the original image based on the target text feature and the image feature includes: Obtaining a domain category of the original image, and adjusting the image features according to the domain category to obtain target image features; Adding a preset image noise to the target image feature to obtain a noisy image feature, and denoising the noisy image feature based on the target text feature to obtain a target content feature; Feature decoding is performed on the target content feature to obtain a target image corresponding to the original image.

5. The image generation method according to claim 4, characterized in that: The step of adjusting the image features according to the domain category to obtain target image features includes: When the domain category is a target domain category, using the image feature as a target image feature; When the domain category is a domain category other than the target domain category, the feature value of the image feature is set to zero to obtain the target image feature.

6. The image generation method according to claim 1, characterized in that: The obtaining of the original image and the image description text corresponding to the original image includes: receiving an image generation request, wherein the image generation request carries an original image and image generation information corresponding to the original image; When the image generation information includes current image description text, using the current image description text as the image description text corresponding to the original image; When the image generation information does not include the current image description text, an image description text corresponding to the original image is generated based on the image generation information and the original image.

7. The image generation method according to claim 6, characterized in that: The step of generating an image description text corresponding to the original image based on the image generation information and the original image includes: Based on the image generation request, identifying image description information in the original image; The image description text is generated according to the image generation information and the image description information.

8. The image generation method according to claim 7, characterized in that: The step of identifying image description information in the original image based on the image generation request includes: According to the image generation request, identifying image content in the original image to obtain content description text; Based on the content description text, determining an image title corresponding to the original image; The image attribute information of the original image is acquired, and the content description text, the image title and the image attribute information are used as the image description information.

9. The image generation method according to claim 1, characterized in that: The extracting features from the original image comprises: Removing the background image from the original image to obtain a main image; Using an image coding network of an image generation model to perform image coding on the subject image to obtain image features; The generating at least one negative word based on the extracted image features includes: using a negative word generation network of the image generation model to generate at least one negative word based on the image features.

10. The image generation method according to claim 9, characterized in that: Before the image encoding network using the image generation model performs image encoding on the subject image to obtain image features, the method further includes: Obtaining an original image sample and a description text sample corresponding to the original image sample; Using a preset image generation model, generating a candidate image according to the original image sample, the description text sample and the preset image noise sample; Based on the preset image noise sample and the candidate image, a noise loss is determined, and the preset image generation model is converged according to the noise loss to obtain the image generation model.

11. The image generation method according to claim 10, characterized in that: The obtaining of the original image sample and the description text sample corresponding to the original image sample comprises: Acquire an initial original image sample, and remove the background image from the initial original image sample to obtain an original image sample; Identifying image description information in the original image template to obtain sample image description information; A description text sample corresponding to the original image sample is generated according to the sample image description information.

12. The image generation method according to claim 10, characterized in that: The method of using a preset image generation model to generate a candidate image according to the original image sample, the description text sample and the preset image noise sample includes: Using a preset image generation model to perform feature extraction on the original image sample and the description text sample to obtain a sample feature set, wherein the sample feature set includes sample image features and sample text features; Adding the preset image noise sample to the sample image feature to obtain the sample image feature after adding noise; Based on the sample text features, denoising the sample image features after adding noise to obtain candidate content features and predicted noise features corresponding to the candidate content features, and feature decoding the candidate content features to obtain a candidate image; The determining of the noise loss based on the preset image noise sample and the candidate image includes: determining the noise loss based on the predicted noise feature and the noise feature of the preset image noise sample.

13. The image generation method according to claim 12, characterized in that: The adopting a preset image generation model to perform feature extraction on the original image sample and the description text sample to obtain a sample feature set includes: Using a preset image generation model to characterize the original image sample, and generating at least one sample negative word based on the extracted initial sample image features; Extracting text features from the description text sample and the sample negative words respectively, and fusing the extracted text features to obtain sample text features; According to the sample domain category of the original image sample, the initial sample image feature is adjusted to obtain a sample image feature, and the sample text feature and the sample image feature are used as a sample feature set.

14. The image generation method according to claim 13, characterized in that: The step of adjusting the initial sample image features according to the sample domain category of the original image sample to obtain the sample image features includes: When the sample domain category is the target domain category, adjusting the initial sample image feature according to the sample object identifier marked in the original image sample to obtain the sample image feature; When the sample domain category is a domain category other than the target domain category, the feature value of the initial sample image feature is set to zero to obtain the sample image feature.

15. The image generation method according to claim 14, characterized in that: The adjusting the initial sample image features according to the sample object identifier marked in the original image sample to obtain the sample image features includes: Predicting an object identifier of an object in the original image sample according to the initial sample image feature to obtain a prediction probability, wherein the prediction probability includes a probability that the object identifier is the sample object identifier; When the predicted probability is less than a preset probability threshold, obtaining an average sample image feature corresponding to the sample object identifier, and using the average sample image feature as the sample image feature; When the predicted probability is greater than or equal to the preset probability threshold, the initial sample image feature is used as the sample image feature.

16. An image generating device, characterized in that: include: An acquisition unit, used to acquire an original image and an image description text corresponding to the original image, and segment the image description text to obtain a text word set; A first extraction unit is used to extract features from the original image, and generate at least one negative word based on the extracted image features, wherein the negative word represents content deleted from the original image during the image generation process; A second extraction unit is used to perform feature extraction on the text word set and the negative words respectively to obtain positive text features corresponding to the text word set and negative text features corresponding to the negative words; An adjustment unit, used to adjust the positive text feature using the negative text feature to obtain a target text feature; A generating unit is used to generate a target image corresponding to the original image based on the target text feature and the image feature.

17. An electronic device, characterized in that: It comprises a processor and a memory, wherein the memory stores an application program, and the processor is used to run the application program in the memory to execute the steps in the image generating method according to any one of claims 1 to 15.

18. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a plurality of instructions, and the instructions are suitable for being loaded by a processor to execute the steps in the image generating method according to any one of claims 1 to 15.

19. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the steps in the image generation method according to any one of claims 1 to 15 are implemented.

Images