Image generation method, medium, computer device and program product

Abstract

The invention discloses an image generation method, a medium, computer equipment and a program product. The method comprises the following steps: acquiring an original image and visual description information corresponding to the original image; performing style migration on the original image through a pre-trained style migration model to obtain a target image of a preset style; wherein the style migration model comprises a plurality of feature extraction modules, and the plurality of feature extraction modules comprise a color feature extraction module used for extracting color features and a non-color feature extraction module used for extracting other visual features except the color features; the color feature extraction module is used for extracting color features of the target object based on the original image and the visual description information, and the non-color feature extraction module is used for extracting other visual features of the target object based on the visual description information; the style migration model is used for performing style migration on the original image based on the color features and the other visual features to obtain the target image.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an image generation method, medium, computer device and program product. Background Art

[0002] Style transfer aims to transform an input image into an output image with a preset style. Style transfer is essentially achieved by changing the pixel values ​​of the input image. Therefore, the color of the target object in the output image obtained by style transfer may be inconsistent with the color of the target object in the input image. In related technologies, it is difficult to improve the consistency of the color of the target object in the output image with the color of the target object in the input image while following the preset style. Summary of the invention

[0003] In a first aspect, an embodiment of the present application provides an image generation method, the method comprising: obtaining an original image and visual description information corresponding to the original image; performing style transfer on the original image through a pre-trained style transfer model to obtain a target image of a preset style; wherein the style transfer model comprises a plurality of feature extraction modules, the plurality of feature extraction modules comprising a color feature extraction module for extracting color features and a non-color feature extraction module for extracting visual features other than color features; the color feature extraction module is used to extract the color features of the target object based on the original image and the visual description information, and the non-color feature extraction module is used to extract other visual features of the target object based on the visual description information; the style transfer model is used to perform style transfer on the original image based on the color features and the other visual features to obtain the target image.

[0004] In a second aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method described in any embodiment of the present application.

[0005] In a third aspect, an embodiment of the present application provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method described in any embodiment of the present application when executing the program.

[0006] In a fourth aspect, an embodiment of the present application provides a computer program product, including a computer program, which implements the method described in any embodiment of the present application when executed by a processor.

[0007] In the embodiment of the present application, the applicant found that the functions of different feature extraction modules in the style transfer model are different. Some feature extraction modules are used to extract color features, while other feature extraction modules are used to extract visual features other than color features. If the original image is injected into each feature extraction module in multiple feature extraction modules during style transfer, each feature extraction module will learn the visual features of the original image from multiple dimensions, which ultimately makes it difficult to make the target image output by the style transfer model follow the preset style well. Therefore, the embodiment of the present disclosure only injects the original image into the color feature extraction module used to extract color features among multiple feature extraction modules, and does not inject it into the non-color feature extraction module used to extract other visual features other than color features for feature extraction. In this way, on the one hand, the color features of the original image are learned during feature extraction, thereby improving the color consistency of the target object in the target image and the target object in the original image; on the other hand, since the original image is only referenced when color features are extracted, the original image is not referenced when other visual features are extracted, which avoids the influence of other visual features of the original image on the feature extraction process, so that the target image can follow the preset style well.

[0008] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] The drawings herein are incorporated into the specification and constitute a part of the present application. These drawings illustrate embodiments consistent with the present application and are used together with the specification to illustrate the technical solutions of the present application.

[0010] Figure 1 It is a schematic diagram of an application scenario of an embodiment of the present application.

[0011] Figure 2 It is a flowchart of the image generation method of an embodiment of the present application.

[0012] Figure 3 It is a schematic diagram of the training process of the first multimodal large language model of an embodiment of the present application.

[0013] Figure 4 It is a schematic diagram of the process of acquiring sample image-text pairs in an embodiment of the present application.

[0014] Figure 5 It is a structural diagram of the style transfer model of an embodiment of the present application.

[0015] Figure 6 It is a block diagram of an image generating device according to an embodiment of the present application.

[0016] Figure 7It is a schematic diagram of a computer device according to an embodiment of the present application. DETAILED DESCRIPTION

[0017] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0018] The terms used in this application are only for the purpose of describing specific embodiments, and are not intended to limit the application. The singular forms of "a", "said" and "the" used in this application and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used in this article refers to and includes any or all possible combinations of one or more associated listed items. In addition, the term "at least one" herein represents any combination of at least two of any one or more of a variety of.

[0019] It should be understood that although the terms first, second, third, etc. may be used in the present application to describe various information, these information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of the present application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0020] In order to enable those skilled in the art to better understand the technical solutions in the embodiments of the present application and to make the above-mentioned purposes, features and advantages of the embodiments of the present application more obvious and understandable, the technical solutions in the embodiments of the present application are further described in detail below in conjunction with the accompanying drawings.

[0021] Style transfer aims to convert an image into another image with a preset style. For example, a cartoon-style image can be converted into an embroidery-style image; or a real-life image can be converted into a cartoon-style image; or other types of style transfer can be performed. Style transfer can be achieved by pre-training a style transfer model. Figure 1 As shown in , the content image and the style image can be input into the style transfer model. The content image is the original image whose content information is to be retained, and the style image is used to define the artistic style that is to be applied to the content image. The style transfer model can convert the content image into the style defined by the style image. Figure 1As shown, the content image is a real photo, and the style image is a cartoon-style image. By inputting the content image and the style image into the style transfer model, a target image output by the style transfer model can be obtained. The style of the target image is the same as the style defined by the style image, both of which are cartoon styles, and the image content of the target image is the same as the image content in the content image. For example, if the content image is an image of a puppy, the target image is an image of a cartoon-style puppy.

[0022] Style transfer is essentially achieved by changing the pixel values ​​of the input image. Therefore, the color of the target object in the output image obtained by style transfer may be inconsistent with the color of the target object in the input image. In the related art, it is difficult to improve the consistency of the color of the target object in the output image with the color of the target object in the input image on the basis of following the preset style.

[0023] Based on this, the present application provides an image generation method, see Figure 2 , the method comprising:

[0024] Step S12: Acquire the original image and visual description information corresponding to the original image;

[0025] Step S14: performing style transfer on the original image through a pre-trained style transfer model to obtain a target image of a preset style;

[0026] Among them, the style transfer model includes multiple feature extraction modules, and the multiple feature extraction modules include a color feature extraction module for extracting color features and a non-color feature extraction module for extracting visual features other than color features; the color feature extraction module is used to extract the color features of the target object based on the original image and the visual description information, and the non-color feature extraction module is used to extract other visual features of the target object based on the visual description information; the style transfer model performs style transfer on the original image based on the color features extracted by the color feature extraction module and the other visual features extracted by the non-color feature extraction module to obtain the target image.

[0027] The present application simultaneously inputs the visual description information corresponding to the original image into a color feature extraction module for extracting color features and a non-color feature extraction module for extracting visual features other than color features, so that both the color feature extraction module and the non-color feature extraction module can learn the corresponding visual features from the visual description information; at the same time, the original image is input into the color feature extraction module in a direction other than the non-color feature extraction module, so that the color feature extraction module can refer to the original image when extracting color features, while the non-color feature extraction module will not refer to the original image when extracting other visual features. In this way, the color features of the original image can be better learned, thereby improving the color consistency between the target image and the target object in the original image; at the same time, it avoids the influence of other visual features of the original image on the feature extraction process, so that the target image can better follow the preset style.

[0028] The specific implementation method of the embodiment of the present application is described below by way of example.

[0029] In step S12, the original image may be an image of any style, which may be obtained by photographing an image capturing device, drawing by drawing software, synthesizing by AI, or obtaining by other means. The number of original images may be greater than or equal to 1. When the number of original images is greater than 1, each original image may be independently style-transferred. Alternatively, multiple original images may be first fused to obtain a fused image, and then the fused image may be style-transferred. Alternatively, style transfer may be performed independently on each original image to obtain a target image corresponding to each original image, and then image fusion may be performed on the target image corresponding to each original image. The above-mentioned image fusion may be a fusion of image content, such as fusing the image content in multiple images into one image; it may also be a fusion of image pixel values, such as weighted averaging of the pixel values ​​of corresponding pixels in multiple images; it may also be image fusion in other dimensions, which will not be listed here one by one.

[0030] The visual description information corresponding to the original image is used to describe the visual features of the original image. The visual features described by the visual description information may include but are not limited to at least one of the following: color features, texture features, shape features, transparency features, spatial features, etc. Optionally, the visual description information may include information for describing the overall visual features of the original image, and may also include information for describing the visual features of the target object in the original image. The target object may be an image subject in the original image, such as an object with the largest number of pixels occupied in the original image, an object of a specified category, an object at a specified composition position in the original image, an object with the largest number in the original image, or an object determined based on other conditions. In the case where the visual description information includes information for describing the visual features of the target object in the original image, the visual features of the target object described therein may include the visual features of the target object as a whole, and may also include the visual features of a local area of ​​the target object. For example, in the case where the target object is an animal, the visual features of the target object as a whole may include the color, action, and expression of the target object as a whole; the visual features of a local area of ​​the target object may include the color and shape of the target object's eyes, the length of the target object's tail, the texture of the target object's fur, etc.

[0031] The visual description information may be text information. The visual description information may be extracted from the original image by a pre-trained first multimodal large language model (MLLM). The multimodal large language model is capable of understanding data of multiple modalities (such as text, images, etc.). In this embodiment, the input of the multimodal large language model may be an image, and the output may be text information.

[0032] In some embodiments, the first sample image can be first encoded by a pre-trained image encoder to obtain image features of the first sample image, and then the image features of the first sample image can be input into the second multimodal large language model, so that the second multimodal large language model outputs visual description information of the first sample image based on the image features of the first sample image. Furthermore, since the dimension of the image features output by the image encoder may be different from the feature dimension supported by the second multimodal large language model, before the image features of the first sample image are input into the second multimodal large language model, the image feature dimension of the first sample image can also be aligned with the feature dimension supported by the second multimodal large language model. The alignment of feature dimensions can be achieved by a multi-layer perceptron.

[0033] In some embodiments, the first multimodal large language model is trained based on sample image-text pairs and sample visual question-answering information. The sample image-text pair includes a first sample image and sample visual description information corresponding to the first sample image (the sample visual description information may also be text information), and the sample visual question-answering information includes a second sample image, a question about the second sample image, and an answer to the question. By using image-text pairs in the training process, the first multimodal large language model can learn the relationship between objects and scenes in the image and the corresponding text descriptions. This helps the first multimodal large language model understand the content and context of the image, so that it can be more accurate when generating visual description information. By using sample visual question-answering information in the training process, the first multimodal large language model can learn to understand the content of the image and the ability to answer specific questions based on the content, thereby improving the reasoning ability and context understanding ability of the first multimodal large language model when processing multimodal information. See. Figure 3 , the training process is as follows:

[0034] Obtain sample data including sample image-text pairs and sample visual question-answering information, obtain visual description information of a first sample image output by a pre-trained multimodal large language model, and obtain an answer output by the pre-trained multimodal large language model based on a second sample image and a question, and based on the difference between the visual description information output by the pre-trained multimodal large language model and the sample visual description information (hereinafter referred to as the first difference), and the difference between the answer output by the pre-trained multimodal large language model and the answer included in the sample visual question-answering information (hereinafter referred to as the second difference), fine-tune the pre-trained multimodal large language model to obtain a first multimodal large language model.

[0035] Specifically, during the model fine-tuning process, the first sample image in the sample image-text pair can be encoded by an image encoder to obtain the features of the first sample image, and the features are input into the pre-trained multimodal large language model. When processing the sample visual question-answering information, the second sample image in the sample visual question-answering information can be encoded by an image encoder to obtain the features of the second sample image, and the features are input into the pre-trained multimodal large language model. The question in the sample visual question-answering information can also be divided into word units by a word unit division module to obtain multiple word units corresponding to the question, and the multiple word units corresponding to the question are input into the pre-trained multimodal large language model. The visual description information and answers output by the pre-trained multimodal large language model, as well as the answers in the sample visual question-answering information, can also be divided into word units by a word unit division module. Based on the word units corresponding to the visual description information output by the pre-trained multimodal large language model and the word units corresponding to the sample visual description information, the above-mentioned first difference can be determined. Based on the word units corresponding to the answers output by the pre-trained multimodal large language model and the word units corresponding to the answers included in the sample visual question-answering information, the above-mentioned second difference can be determined.

[0036] The above-mentioned first difference can reflect the ability of the pre-trained multimodal large language model to understand the first sample image. The more accurate the visual description information output by the pre-trained multimodal large language model for the first sample image, the smaller the first difference, indicating that the pre-trained multimodal large language model has a stronger ability to understand the first sample image. The above-mentioned second difference can reflect the reasoning ability, text generation ability and degree of understanding of the context of the pre-trained multimodal large language model. The more accurate the answer output by the pre-trained multimodal large language model for the question corresponding to the second sample image, the smaller the second difference, indicating that the reasoning ability, text generation ability and degree of understanding of the context of the pre-trained multimodal large language model are stronger.

[0037] The first multimodal large language model obtained by fine-tuning in the above manner can be applicable to a variety of preset styles. That is, no matter what style of the target image required by the user is, the visual description information of the original image can be extracted through the first multimodal large language model. There is no need to train the first multimodal large language model for multiple styles separately, thereby reducing the training complexity.

[0038] In some embodiments, the first sample image in the sample image-text pair can be obtained and annotated to obtain the sample visual description information in the sample image-text pair. However, the annotation cost of this method is high. In order to reduce the annotation cost, the sample image-text pair can be automatically generated by the second multimodal large language model. Figure 4 The specific method of automatically generating sample image-text pairs is illustrated with examples.

[0039] First, visual description information of the first sample image output by the second multimodal large language model can be obtained. Among them, the second multimodal large language model can be a multimodal large language model with a different structure or type from the first multimodal large language model. The visual description information of the first sample image can be extracted through the second multimodal large language model. The visual description information of the first sample image can be similar to the visual description information of the original image. For details, please refer to the aforementioned embodiment, which will not be repeated here.

[0040] Then, an intermediate image is generated based on the visual description information output by the second multimodal large language model. In this step, the visual description information output by the second multimodal large language model can be restored to an intermediate image through an image generation model (such as a generative adversarial network (GAN) or a variational autoencoder (VAE)). There may be certain differences between the visual features of the restored intermediate image and the visual features of the first sample image. Therefore, target description information can be obtained, which is used to describe the inconsistent visual features between the intermediate image and the first sample image. Specifically, both the intermediate image and the first sample image can be input into the second multimodal large language model to obtain the target description information through the second multimodal large language model.

[0041] Then, the similarity between the intermediate image and the first sample image is determined based on the target description information and the visual description information output by the second multimodal large language model. For example, the above similarity may be a similarity score. The number of inconsistent visual features of the intermediate image and the first sample image may be determined based on the target description information, and the total number of visual features of the first sample image may be determined based on the visual description information output by the second multimodal large language model, and the similarity score between the intermediate image and the first sample image may be determined based on the number of inconsistent visual features of the intermediate image and the first sample image and the above ratio. Of course, other methods may also be used to determine the similarity between the intermediate image and the first sample image, which is not limited in this application.

[0042] If the determined similarity is greater than the preset similarity threshold, it means that the visual description information output by the second multimodal large language model has a high degree of match with the first sample image. Therefore, the first sample image can be highly restored based on the visual description information. Therefore, the first sample image and the visual description information output by the second multimodal large language model can be determined as a sample image-text pair. Otherwise, the step of obtaining the visual description information of the first sample image output by the pre-trained second multimodal large language model can be returned to regenerate the visual description information of the first sample image. Specifically, the visual description information, target description information, and the similarity score between the first sample image and the intermediate image generated by the second multimodal large language model in the previous round of dialogue can be passed to the second multimodal large language model in the form of a historical dialogue, so that the second multimodal large language model regenerates the visual description information of the first sample image based on the above information.

[0043] In step S14, the style of the original image may be transferred using a pre-trained style transfer model to obtain a target image of a preset style.

[0044] The style transfer model may include a color feature extraction module for extracting color features and a non-color feature extraction module for extracting other visual features (such as shape features, texture features, etc.) other than color features. The description information corresponding to the original image can be input into the color feature extraction module and the non-color feature extraction module, so that the visual features of the original image in various dimensions (including color features and non-color features) can be extracted. At the same time, the original image itself can also be input into the style transfer model so that the style transfer model extracts visual features from the original image. However, the inventors have found that if the original image is input into all feature extraction modules, the style transfer model will refer to the visual features of the original image in all dimensions when performing style transfer, resulting in the output target image being close to the original image in all feature dimensions, thereby reducing the consistency of the target image with the preset style.

[0045] Therefore, the present application only directs the input of the original image into the color feature extraction module, and not into the non-color feature extraction module. The color feature extraction module can extract color features based on the original image and its visual description information, while the non-color feature extraction module only extracts non-color features based on the visual description information of the original image. Ultimately, the style transfer model can output a target image of a preset style based on the color features extracted by the color feature extraction module and the non-color features extracted by the non-color feature extraction module. By adopting the above method, the original image is only referenced when color features are extracted, and the original image is not referenced when other visual features are extracted, thereby avoiding the influence of other visual features of the original image on the feature extraction process, so that the target image can better follow the preset style.

[0046] Combine the following Figure 5 The specific implementation method of each feature extraction module is illustrated with examples.

[0047] In some embodiments, the i-th feature extraction module among multiple feature extraction modules can perform feature extraction based on the following method: if the i-th feature extraction module is a color feature extraction module, the visual description information and the original image are fused with the features extracted from the input latent vector of the i-th feature extraction module to obtain the output features of the i-th feature extraction module; if the i-th feature extraction module is a non-color feature extraction module, the visual description information is fused with the features extracted from the input latent vector of the i-th feature extraction module to obtain the output features of the i-th feature extraction module. In the above method, the original image can be selectively injected into the color feature extraction module instead of injecting the original image into each feature extraction module in the style transfer model.

[0048] In some embodiments, each feature extraction module may use an attention mechanism to perform feature extraction. Specifically, the input latent vector of the i-th feature extraction module may be obtained, and the input latent vector of the i-th feature extraction module may be self-attention processed to obtain the first intermediate vector of the i-th feature extraction module. Among them, the first intermediate vector of the i-1-th feature extraction module may be used as the input latent vector of the i-th feature extraction module among multiple feature extraction modules, the first intermediate vector of the i-th feature extraction module may be used as the input latent vector of the i+1-th feature extraction module among multiple feature extraction modules, and so on. The input latent vector of the first feature extraction module is the original latent vector.

[0049] For the visual description information, the text vector corresponding to the visual description information can be obtained, and the text vector and the first intermediate vector of the i-th feature extraction module can be cross-attention processed to obtain the second intermediate vector of the i-th feature extraction module. In the above embodiment, the latent vector is a vector of a preset dimension, which represents the position or feature of the input data in the latent space (LatentSpace). By performing self-attention processing on the input latent vector, it is possible to focus on different areas of the input image and reweight the features of these areas to highlight important parts and suppress irrelevant information. This helps the style transfer model learn the relationship between the various parts of the image, thereby improving the quality of the generated image. By performing cross-attention processing, the style transfer model can combine information from different modalities (including the original image and visual description information) to better capture the visual features of the image. Figure 5As shown, each feature extraction module includes three functional modules, which are respectively represented by yellow, green and red blocks in the figure. Among them, the yellow block represents the function for performing the above-mentioned self-attention processing to obtain the first intermediate vector, the red block represents the function for performing the cross-attention processing to obtain the second intermediate vector, and the green block represents the function for performing the cross-attention processing to obtain the third intermediate vector.

[0050] If the i-th feature extraction module is a color feature extraction module, the feature vector corresponding to the original image and the first intermediate vector of the i-th feature extraction module can be cross-attention processed to obtain the third intermediate vector of the i-th feature extraction module, and the second intermediate vector and the third intermediate vector are fused as the output features of the i-th feature extraction module. The output features of the i-th feature extraction module can be output to the i+1-th feature extraction module. If the i-th feature extraction module is a non-color feature extraction module, the second intermediate vector is directly used as the output feature of the i-th feature extraction module, and the output features of the i-th feature extraction module are output to the i+1-th feature extraction module. In this way, the original image is injected into the color feature extraction module in a directional manner. See more Figure 5 , the original image can be directed to the specified feature extraction module, which is equivalent to setting a corresponding "switch" for each feature extraction module. By controlling the switch to be closed, the original image can be injected into the feature extraction module; by controlling the switch to be disconnected, the original image can be not injected into the feature extraction module.

[0051] In some embodiments, the second intermediate vector can be weighted based on a preset text weight to obtain a weighted second intermediate vector, and the third intermediate vector can be weighted based on a preset image weight to obtain a weighted third intermediate vector. Then, the weighted second intermediate vector and the weighted third intermediate vector are summed. Among them, the weights corresponding to the second intermediate vector and the third intermediate vector can be adjusted according to actual needs. Through the above-mentioned weighted processing, the influence ratio of the visual description information and the original image on the feature extraction result can be adjusted, so that the user can flexibly choose to perform feature extraction mainly based on the visual description information, or to perform feature extraction mainly based on the original image.

[0052] In some embodiments, the self-attention processing can be implemented in the following manner: edge recognition can be performed on the target object in the original image to obtain the edge features of the target object, the edge features can be fused with the input latent vector of the i-th feature extraction module to obtain a fused latent vector, and the fused latent vector can be self-attention processed. Since the edge features can reflect the shape and outline of the target object in the original image, self-attention processing based on the edge features can keep the shape and outline of the target object consistent before and after style transfer. Furthermore, before edge recognition, the original image can also be segmented to remove the background area in the original image. By removing the background area, the influence of the background area on edge recognition can be reduced, thereby improving the accuracy of edge recognition.

[0053] In some embodiments, see Figure 5 , the above-mentioned multiple feature extraction modules include multiple encoding modules for feature encoding and multiple decoding modules for feature decoding, and the multiple encoding modules correspond to the multiple decoding modules one by one. The color feature extraction module may include at least one encoding module and at least one decoding module, and the non-color feature extraction module may also include at least one encoding module and at least one decoding module. Among them, if the i-th feature extraction module is one of the multiple encoding modules, the input latent vector of the i-th feature extraction module is the output latent vector of the i-1-th feature extraction module among the multiple feature extraction modules. If the i-th feature extraction module is one of the multiple decoding modules, the input latent vector of the i-th feature extraction module is obtained by fusing the output latent vector of the encoding module corresponding to the i-th feature extraction module and the output latent vector of the i-1-th feature extraction module among the multiple feature extraction modules. The encoding module is usually responsible for extracting the contextual features of the input data. In this embodiment, by fusing the output latent vector of the encoding module into the decoding module, the contextual features extracted by the encoding module can be injected into the decoding module, so that the decoding module generates a more coherent output with contextual understanding, improves the feature extraction effect, and thus improves the style transfer effect.

[0054] In practical applications, there may be multiple styles commonly used in style transfer. All of the commonly used styles can be used as candidate styles, and multiple candidate styles can be pre-generated to correspond to multiple candidate style transfer models. Each style transfer model generated can include the color feature extraction module and the non-color feature extraction module in the aforementioned embodiment. The specific implementation method is detailed in the aforementioned embodiment and will not be repeated here. In practical applications, the preset style selected by the user from multiple candidate styles can be obtained, and the original image can be style transferred through the candidate style transfer model corresponding to the selected preset style, so as to adapt to different style transfer requirements.

[0055] The solution of the present application can be widely used in the style transfer of various original images. The following takes a specific application scenario as an example to illustrate the solution of the present application. In this application scenario, the original image is a pet wallpaper image. Through data enhancement (using the original image and its visual description information as input data for style transfer), multimodal large language model fine-tuning, and directional feature injection (only injecting the color feature extraction module into the original image) technology, the present application can improve the consistency of the pet's fur color in the target image and the pet's fur color in the original image (that is, the consistency of the color features), improve the quality of the target image, and maintain the compatibility of the preset style in style transfer. The specific scheme is as follows:

[0056] In the process of style transfer, the SD-Unet model is used as the style transfer model. First, the edge recognition of the original image is performed and injected into the decoding process of the SD-Unet model through Control-net, so that the pet in the generated target image is consistent with the pet in the original image in shape. In the scheme of fur color consistency, the information input to the SD-Unet model is enhanced in two parts: 1) The fur color in the original image is understood more fully by fine-tuning the first multimodal large language model based on the Llava structure, thereby enhancing the fur color-related control information input to the SD-Unet model; 2) Using the pre-trained cross-attention of IPAdapter, the 3rd and 5th feature extraction modules of the SD-Unet model (i.e., the color feature extraction module in the aforementioned embodiment) are directed injected, and the color features of the target object in the original image are maintained to the greatest extent during the style transfer process while reducing the influence of other visual features in the original image on the style transfer process.

[0057] In order to allow the first multimodal large language model to have an accurate understanding of the subject's coat color, this solution uses data augmentation methods to re-supervise fine-tuning of the model to enhance its coat color understanding ability. Specifically, based on the multimodal capabilities of the GPT4-o model (i.e., the second multimodal large language model), a batch of high-quality pet coat color description image-text pair data (i.e., the sample image-text pairs in the aforementioned embodiment) is distilled in combination with thought chain prompts and multiple rounds of self-feedback iteration technology, and the first multimodal large language model is fine-tuned based on the above-mentioned image-text pair data. In the thought chain prompt technology, this application structuredly defines all semantically describable visual feature dimensions of pet coat color based on prior observations. In the multiple rounds of self-feedback iteration technology, the deviation of the generated visual description information is gradually corrected.

[0058] The first multimodal large language model after data enhancement and fine-tuning has a better understanding of the fur color of the main pet (i.e., the target object) in the original image. This application further uses it to generate detailed visual description information as control information to enhance the subject consistency maintenance of the style transfer model.

[0059] In the process of style transfer, the preset styles are implemented based on different style Lora models. Since the various preset styles have different effects on the fur color of the subject pet, when using the crossAttention pre-trained by IPAdapter, the feature extraction module of the SD-Unet model is flexibly selected for directional injection, thereby adjusting the style transfer effect while ensuring the consistency of the subject. Taking the StableDiffusionXL model as an example, the injection control is performed on the 11-layer BasicTransformerBlock in the SD-Unet model, and the weights of each layer can be adjusted.

[0060] This application controls the color consistency of the target object in the original image from two aspects: 1) Through detailed visual description information, the global color and layout information of the target object is injected into the style transfer model, and the text prompts are consistent with the data distribution during the pre-training of the style transfer model, so there is good raw image quality and harmony between different attributes; 2) Directly inject more fine-grained image information of the target object through the original image, and at the same time, through the model-oriented injection and reasoning process, precise control of the consistency of the target object is achieved, and conflicts with other visual features are avoided.

[0061] See also Figure 6 The present application also provides an image generating device, the device comprising:

[0062] An acquisition module 202 is used to acquire an original image and visual description information corresponding to the original image;

[0063] A style transfer module 204 is used to transfer the style of the original image through a pre-trained style transfer model to obtain a target image of a preset style;

[0064] Among them, the style transfer model includes multiple feature extraction modules, and the multiple feature extraction modules include a color feature extraction module for extracting color features and a non-color feature extraction module for extracting visual features other than color features; the color feature extraction module is used to extract the color features of the target object based on the original image and the visual description information, and the non-color feature extraction module is used to extract other visual features of the target object based on the visual description information; the style transfer model performs style transfer on the original image based on the color features extracted by the color feature extraction module and the other visual features extracted by the non-color feature extraction module to obtain the target image.

[0065] The functions or modules included in the device provided in the present application can be used to execute the method described in the above method embodiment. Its specific implementation can refer to the description of the above method embodiment. For the sake of brevity, it will not be repeated here.

[0066] An embodiment of the present application further provides a computer device, which comprises at least a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the method described in any of the foregoing embodiments is implemented.

[0067] Figure 7 A more specific hardware structure diagram of a computer device provided in an embodiment of the present application is shown, and the device may include: a processor 302, a memory 304, an input / output interface 306, a communication interface 308, and a bus 310. The processor 302, the memory 304, the input / output interface 306, and the communication interface 308 are connected to each other in communication within the device through the bus 310.

[0068] The processor 302 can be implemented by a general central processing unit (CPU), a microprocessor, an application specific integrated circuit (ASIC), or one or more integrated circuits, etc., for executing relevant programs to implement the technical solutions provided in the embodiments of the present application. The processor 302 can also include a graphics card, which can be a Nvidia titan X graphics card or a 1080Ti graphics card, etc.

[0069] The memory 304 can be implemented in the form of a read-only memory (ROM), a random access memory (RAM), a static storage device, a dynamic storage device, etc. The memory 304 can store an operating system and other application programs. When the technical solution provided in the embodiment of the present application is implemented by software or firmware, the relevant program code is stored in the memory 304 and called and executed by the processor 302.

[0070] The input / output interface 306 is used to connect the input / output module to realize information input and output. The input / output module can be configured in the device as a component (not shown in the figure), or it can be externally connected to the device to provide corresponding functions. The input device may include a keyboard, a mouse, a touch screen, a microphone, various sensors, etc., and the output device may include a display, a speaker, a vibrator, an indicator light, etc.

[0071] The communication interface 308 is used to connect a communication module (not shown) to realize communication interaction between the device and other devices. The communication module can realize communication through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0072] The bus 310 comprises a pathway for transmitting information between the various components of the device, such as the processor 302 , the memory 304 , the input / output interface 306 , and the communication interface 308 .

[0073] It should be noted that, although the above device only shows the processor 302, the memory 304, the input / output interface 306, the communication interface 308 and the bus 310, in the specific implementation process, the device may also include other components necessary for normal operation. In addition, it can be understood by those skilled in the art that the above device may also only include the components necessary for implementing the embodiment of the present application, and does not necessarily include all the components shown in the figure.

[0074] An embodiment of the present application provides a computer program product, including a computer program, which implements the method described in any embodiment of the present application when executed by a processor.

[0075] An embodiment of the present application further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method described in any of the foregoing embodiments.

[0076] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computer device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0077] Each embodiment in the present application is described in a progressive manner, and the same and similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts refer to the partial description of the method embodiment. The device embodiment described above is merely schematic, wherein the modules described as separate components may or may not be physically separated, and the functions of each module can be implemented in the same one or more software and / or hardware when implementing the embodiment scheme of the present application. It is also possible to select some or all of the modules according to actual needs to achieve the purpose of the embodiment scheme. Those of ordinary skill in the art can understand and implement it without paying creative work.

[0078] The above is only a specific implementation of the embodiment of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the embodiment of the present application. These improvements and modifications should also be regarded as the protection scope of the embodiment of the present application.

Claims

1. A method for generating an image, the method comprising: Acquire an original image and visual description information corresponding to the original image; Performing style transfer on the original image using a pre-trained style transfer model to obtain a target image with a preset style; The style migration model includes a plurality of feature extraction modules, and the plurality of feature extraction modules include a color feature extraction module for extracting color features and a non-color feature extraction module for extracting other visual features other than color features; the color feature extraction module is used to extract the color features of the target object based on the original image and the visual description information, and the non-color feature extraction module is used to extract other visual features of the target object based on the visual description information; The style transfer model is used to perform style transfer on the original image based on the color feature and the other visual features to obtain the target image.

2. According to the method of claim 1, the step of obtaining the original image and the visual description information corresponding to the original image comprises: Visual description information corresponding to the original image is extracted through a pre-trained first multimodal large language model.

3. According to the method of claim 2, the first multimodal large language model is obtained based on the following method: Acquire sample data, wherein the sample data includes a sample image-text pair and sample visual question-answering information, wherein the sample image-text pair includes a first sample image and sample visual description information corresponding to the first sample image, and the sample visual question-answering information includes a second sample image, a question about the second sample image, and an answer to the question; Obtaining visual description information of the first sample image output by a pre-trained multimodal large language model, and obtaining an answer output by the pre-trained multimodal large language model based on the second sample image and the question; Based on the difference between the visual description information output by the pre-trained multimodal large language model and the sample visual description information, and the difference between the answer output by the pre-trained multimodal large language model and the answer included in the sample visual question and answer information, the pre-trained multimodal large language model is fine-tuned to obtain the first multimodal large language model.

4. The method according to claim 3, further comprising: Obtaining visual description information of the first sample image output by a second multimodal large language model; generating an intermediate image based on the visual description information output by the second multimodal large language model; Acquire target description information, where the target description information is used to describe inconsistent visual features between the intermediate image and the first sample image; Determining a similarity between the intermediate image and the first sample image based on the target description information and the visual description information output by the second multimodal large language model; If the similarity is greater than a preset similarity threshold, the first sample image and the visual description information output by the second multimodal large language model are determined as the sample image-text pair; otherwise, the process returns to the step of obtaining the visual description information of the first sample image output by the pre-trained second multimodal large language model.

5. The method according to claim 4, wherein the second multimodal large language model outputs visual description information of the first sample image based on image features of the first sample image; wherein The image feature of the first sample image is obtained by performing image encoding on the first sample image by a pre-trained image encoder.

6. The method according to claim 1, wherein the i-th feature extraction module among the plurality of feature extraction modules performs feature extraction based on the following manner: If the i-th feature extraction module is the color feature extraction module, the visual description information and the original image are fused with the features extracted from the input latent vector of the i-th feature extraction module to obtain the output features of the i-th feature extraction module; If the i-th feature extraction module is the non-color feature extraction module, the visual description information is fused with the features extracted from the input latent vector of the i-th feature extraction module to obtain the output features of the i-th feature extraction module.

7. According to the method of claim 6, the feature extracted from the input latent vector of the i-th feature extraction module is a first intermediate vector of the i-th feature extraction module, and the first intermediate vector of the i-th feature extraction module is obtained by performing self-attention processing on the input latent vector of the i-th feature extraction module; The step of fusing the visual description information and the original image with the features extracted from the input latent vector of the i-th feature extraction module to obtain the output features of the i-th feature extraction module includes: Obtaining a text vector corresponding to the visual description information, performing cross-attention processing on the text vector and the first intermediate vector of the i-th feature extraction module, and obtaining a second intermediate vector of the i-th feature extraction module; Performing cross attention processing on the feature vector corresponding to the original image and the first intermediate vector of the i-th feature extraction module to obtain a third intermediate vector of the i-th feature extraction module; The second intermediate vector and the third intermediate vector are fused, and a fusion result is used as an output feature of the i-th feature extraction module.

8. According to the method of claim 6, the feature extracted from the input latent vector of the i-th feature extraction module is a first intermediate vector of the i-th feature extraction module, and the first intermediate vector of the i-th feature extraction module is obtained by performing self-attention processing on the input latent vector of the i-th feature extraction module; The step of fusing the visual description information with the features extracted from the input latent vector of the i-th feature extraction module to obtain the output features of the i-th feature extraction module comprises: Obtaining a text vector corresponding to the visual description information, performing cross-attention processing on the text vector and the first intermediate vector of the i-th feature extraction module, and obtaining a second intermediate vector of the i-th feature extraction module; The second intermediate vector is determined as the output feature of the i-th feature extraction module.

9. The method according to claim 7, wherein the fusing of the second intermediate vector and the third intermediate vector comprises: Weighting the second intermediate vector based on a preset text weight to obtain a weighted second intermediate vector; Weighting the third intermediate vector based on a preset image weight to obtain a weighted third intermediate vector; The weighted second intermediate vector and the weighted third intermediate vector are summed.

10. The method according to claim 7 or 8, wherein the step of performing self-attention processing on the input latent vector of the i-th feature extraction module comprises: Performing edge recognition on the target object in the original image to obtain edge features of the target object; Fusing the edge feature with the input latent vector of the i-th feature extraction module to obtain a fused latent vector; Perform self-attention processing on the fused latent vector.

11. The method according to claim 10, wherein the plurality of feature extraction modules include a plurality of encoding modules for feature encoding and a plurality of decoding modules for feature decoding, and the plurality of encoding modules correspond to the plurality of decoding modules in a one-to-one manner; wherein, If the i-th feature extraction module is one of the multiple encoding modules, the input latent vector of the i-th feature extraction module is the output latent vector of the i-1-th feature extraction module among the multiple feature extraction modules; If the i-th feature extraction module is one of the multiple decoding modules, the input latent vector of the i-th feature extraction module is obtained by fusing the output latent vector of the encoding module corresponding to the i-th feature extraction module and the output latent vector of the i-1-th feature extraction module among the multiple feature extraction modules.

12. The method according to claim 1, wherein the performing style transfer on the original image using a pre-trained style transfer model comprises: Acquire the preset style selected by the user from a plurality of candidate styles; wherein the plurality of candidate styles correspond to a plurality of candidate style transfer models respectively; The style of the original image is transferred using the candidate style transfer model corresponding to the selected preset style.

13. A computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the method according to any one of claims 1 to 12.

14. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method according to any one of claims 1 to 12 when executing the computer program.

15. A computer program product, comprising a computer program, which implements the method according to any one of claims 1 to 12 when executed by a processor.

Images