Image generation method and device, computer device and storage medium

Abstract

The present disclosure provides an image generation method, device, computer equipment and storage medium. The method comprises: obtaining an initial image and an initial text; determining a segmentation scheme according to the size of the initial image, segmenting the initial image according to the segmentation scheme to generate at least one segmented image; using an image encoder to extract features of the initial image and the at least one segmented image respectively to obtain global features and local features, and fusing the global features and the local features to obtain an image encoding result; and inputting the image encoding result and the initial text into a preset image generation model to generate a target image.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and in particular to an image generation method, apparatus, computer device, and storage medium. Background Technology

[0002] Image generation refers to the generation of new images through computer algorithms and models. With the popularization and development of computer technology, users' requirements for image generation are becoming more and more specific and diverse.

[0003] Personalized text-to-image generation, also known as pre-trained large-scale text-to-image generation model, extends text-conditional image generation to text + image-conditional image generation. It enables the generation of new images containing elements of a given text, given one or more images.

[0004] However, current processing methods require significant compression of the input image, leading to the loss of considerable detail. Furthermore, the compression process causes distortion of some image information due to changes in image size. It is evident that current image generation schemes have considerable room for improvement in terms of detail reproduction. Summary of the Invention

[0005] In view of this, the present disclosure proposes an image generation method, apparatus, computer device, and storage medium to solve or partially solve the above-mentioned problems.

[0006] To achieve the above objectives, the first aspect of this disclosure provides an image generation method, comprising:

[0007] Obtain the initial image and initial text;

[0008] A segmentation scheme is determined based on the size of the initial image, and the initial image is segmented according to the segmentation scheme to generate at least one segmented image;

[0009] The initial image and the at least one segmented image are subjected to feature extraction using an image encoder to obtain global features and local features. The global features and the local features are then fused to obtain the image encoding result.

[0010] The image encoding result and the initial text are input into a preset image generation model to generate the target image.

[0011] A second aspect of this disclosure provides an image generation apparatus, comprising:

[0012] The first module is used to obtain the initial image and initial text;

[0013] The second module is used to determine a segmentation scheme based on the size of the initial image, and to segment the initial image according to the segmentation scheme to generate at least one segmented image;

[0014] The third module is used to extract features from the initial image and the at least one segmented image using an image encoder to obtain global features and local features, and to fuse the global features and local features to obtain an image encoding result.

[0015] The fourth module is used to input the image encoding result and the initial text into a preset image generation model to generate the target image.

[0016] A third aspect of this disclosure provides a computer device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the program, implements the method described in the first aspect.

[0017] In a fourth aspect, this disclosure provides a non-transitory computer-readable storage medium storing computer instructions for causing a computer to perform the method as described in the first aspect.

[0018] As can be seen from the above, the image generation method, apparatus, computer equipment, and storage medium provided in this disclosure first segment the initial image, allowing for segmentation according to the size requirements of the image encoder, ensuring the size of the segmented image is suitable for the encoder's needs. Then, the image encoder directly extracts features from the segmented image to obtain local features. Simultaneously, the image encoder extracts features from the initial image to obtain global features. The local features are then fused into the global features to form an image encoding result for input into a preset image generation model. This ensures that the image encoding result, while maintaining the number of features, enhances the correlation between local features using global features and preserves the detailed features of the initial image using local features. This solves the problem of size changes during compression, enabling the image encoding result to more accurately represent image information. Ultimately, it improves the level of detail in the original image information during image generation, enhancing image generation efficiency and user experience. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments or related technologies of this disclosure, the accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the accompanying drawings described below are only embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1A schematic diagram of an exemplary system 100 according to an embodiment of the present disclosure is shown.

[0021] Figure 2 A flowchart illustrating an exemplary method 200 provided in an embodiment of this disclosure is shown.

[0022] Figure 3 A schematic diagram showing the effect comparison between the initial image input and the output image of the exemplary method 200 provided in this embodiment of the disclosure.

[0023] Figure 4 A flowchart illustrating an exemplary method 200 in a specific scenario provided by an embodiment of this disclosure is shown.

[0024] Figure 5 A schematic diagram of an exemplary apparatus 500 provided in an embodiment of this disclosure is shown.

[0025] Figure 6 A schematic diagram of an exemplary computer device 600 provided in an embodiment of the present disclosure is shown. Detailed Implementation

[0026] To make the objectives, technical solutions, and advantages of this specification clearer, the following detailed description is provided in conjunction with specific embodiments and the accompanying drawings.

[0027] It should be noted that, unless otherwise defined, the technical or scientific terms used in the embodiments of this disclosure should have the ordinary meaning understood by one of ordinary skill in the art to which this disclosure pertains. The terms "first," "second," and similar words used in the examples of this disclosure do not indicate any order, quantity, or importance, but are merely used to distinguish different components. Terms such as "comprising" or "including" mean that the element or object preceding the word encompasses the elements or objects listed following the word and their equivalents, without excluding other elements or objects. Terms such as "connected" or "linked" are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. Terms such as "upper," "lower," "left," and "right" are used only to indicate relative positional relationships; when the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0028] It is understood that before using the technical solutions disclosed in the various embodiments of this disclosure, users should be informed of the types, scope of use, and usage scenarios of the personal information involved in this disclosure in an appropriate manner in accordance with relevant laws and regulations, and user authorization should be obtained.

[0029] For example, upon receiving a user's active request, a prompt message is sent to the user to explicitly inform them that the requested operation will require the acquisition and use of the user's personal information. This allows the user to independently choose whether to provide personal information to the software or hardware, such as the electronic device, application, server, or storage medium performing the operations of this disclosed technical solution, based on the prompt message.

[0030] As an optional but non-limiting implementation, in response to a user's active request, sending a prompt message to the user can be done via a pop-up window, where the prompt message can be presented in text format. Furthermore, the pop-up window can also include a selection control allowing the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0031] It is understood that the above notification and user authorization process are merely illustrative and do not constitute a limitation on the implementation of this disclosure. Other methods that comply with relevant laws and regulations may also be applied to the implementation of this disclosure.

[0032] It is understood that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) shall comply with the requirements of relevant laws, regulations and related provisions.

[0033] To make the objectives, technical solutions, and advantages of this specification clearer, the following detailed description is provided in conjunction with specific embodiments and the accompanying drawings.

[0034] Figure 1 A schematic diagram of an exemplary system 100 according to an embodiment of the present disclosure is shown. System 100 may be a system for implementing image generation.

[0035] like Figure 1 As shown, taking the joint execution of an object display method by a terminal device and a server as an example, system 100 may include a terminal device 102, a server 104, and a database server 106. The terminal device 102 and server 104 are connected via a network, such as a wired or wireless network. Optionally, the apparatus for implementing image generation may be integrated into the terminal device 102. The database server 106 and server 104 are connected via a network, such as a wired or wireless network. The database server 106 may store various data related to the execution of the image generation method, such as the base image, image parameters, image generation algorithm, etc.

[0036] The terminal device 102 can have various applications (APPs) installed, such as image processing applications, video conferencing applications, lifestyle service applications, reading applications, video applications, social networking applications, payment applications, web browsers, and instant messaging tools. These applications can all be used to generate images and / or display the generated images. As an optional example, the applications (APPs) installed on the terminal device 102 can be downloaded and installed from the server 104.

[0037] The terminal device 102 here can be either hardware or software. When the terminal device 102 is hardware, it can be various electronic devices with a display screen, including but not limited to smartphones, tablets, e-book readers, MP3 players, laptops, and desktop computers (PCs). When the terminal device 102 is software, it can be installed in the electronic devices listed above. It can be implemented as multiple software programs or software modules (e.g., to provide distributed services) or as a single software program or software module. No specific limitations are made here.

[0038] Server 104 can be a server that provides various services. For example, it could be a backend server that supports various applications displayed on terminal device 102. Database server 106 can also be a database server that provides various services. It is understood that if server 104 can implement the relevant functions of database server 106, then database server 106 may not need to be set up in system 100.

[0039] The server 104 and database server 106 here can be either hardware or software. When they are hardware, they can be implemented as a distributed server cluster consisting of multiple servers, or as a single server. When they are software, they can be implemented as multiple software programs or software modules (e.g., used to provide distributed services), or as a single software program or software module. No specific limitations are made here.

[0040] It should be noted that the image generation method provided in this embodiment can be executed by system 100. Specifically, it can be executed interactively between terminal device 102, server 104, and database server 106. It is understood that if terminal device 102 possesses the functions required by server 104 and database server 106 to execute the image generation method, it can also be executed solely by terminal device 102. It should be understood that... Figure 1The number of terminal devices 102, servers 104, database servers 106, and users 108 shown in the diagram is merely illustrative. Any number of terminal devices, users, servers, and database servers can be included depending on actual needs.

[0041] In an exemplary application scenario, user 108 can input instructions to create an image through terminal device 102. Server 104 can provide image generation services to user 108 based on these instructions and display the operation interface of the image generation service on the page through terminal device 102.

[0042] As described in the background section, in some examples, users can utilize AI (Artificial Intelligence) to generate images, such as applying Generative Artificial Intelligence (AIGC). Image generation refers to the generation of new images through computer algorithms and models, which may be entirely fictional, artistic creations, or modifications and enhancements of existing images.

[0043] Among the various image generation tools that utilize AIGC, there is usually an interface to display the image and corresponding operation functions to the user. The user can use these operation functions to provide AIGC with more specific references and emphases, such as providing the tool with reference images, reference text, setting the emphasis when referencing images, setting the degree of reference, and so on.

[0044] In some examples, an image generation tool can be software or a program with image generation capabilities. However, this is not specifically limited. In particular, any tool that has image generation capabilities and intelligently generates images through interaction with the operator can be considered an image generation tool.

[0045] In a more specific scenario, user 108 can utilize an image processing application (APP) to generate images using AIGC. In one example, the user can input any image they obtain or input as the raw image into the image processing application. After completing the corresponding settings and inputting raw text containing key information for generating the image, the application will use a corresponding image generation model to transform and generate the image. This image generation model is primarily a large, foundational model for image generation, and its structure can be stacked GANs, diffusion models, UNet, etc. Then, to generate images of specific styles based on this foundational model, specific constraints need to be provided. This can be achieved by setting plugins (which can be understood as smaller models providing constraints) on the foundational model. These plugins provide the necessary constraints to the foundational model, ultimately resulting in images with different styles. Depending on the specific scenario, there can be various plugins set for the foundational model, such as ControlNet models, IP-preserving or ID-preserving models, etc. In a more specific scenario, this solution focuses on personalized text-to-image generation. Based on a pre-trained large-scale text-to-image model, it extends text-conditional image generation to text + image-conditional image generation. This allows for the generation of new images containing elements of one or more given images, controlled by a given text. Specific applications include: story creation: maintaining consistency in the appearance of characters within images while using text to drive the generation of new actions, expressions, and backgrounds; image stylization: preserving the content information of the subject and background of an image while generating new styles through text control; and image editing: maintaining local elements within an image while enabling local modifications through text control. It is clear that preserving image features plays a crucial role in personalized text-to-image generation, and image encoding determines the upper limit of feature preservation. Extracted image features need to perceive the overall abstract semantics to better interact with text signals for text-driven processing, and also need to perceive image details to achieve better feature preservation and detail restoration.

[0046] However, in some examples, a personalized text-to-image method can employ a single encoder with a fixed low resolution, such as IPadapter. The initial input image is then downsampled (resized) to a fixed size of 224*224. Next, 256 image tokens are extracted using an image encoder in a multimodal contrastive learning pre-trained model. These tokens are then passed through a feature mapper to align with the feature space of the larger text-to-image model.

[0047] However, the applicant discovered the following issues when implementing the above example solution: First, regarding resolution, on the one hand, the original resolution of the input image is generally much larger than 224*224, and the significant resolution compression leads to the loss of image detail information. On the other hand, the original aspect ratio of the input image is also quite diverse; downsampling to a square will cause distortion of image information. For example, downsampling a tall, thin person and then extracting overall semantic features may result in information about a short, stout person. The reason for these problems may depend on the characteristics of the image encoder in the pre-trained contrastive learning model, which can only support a fixed resolution and has not yet been properly addressed in personalized text-generated images. Second, regarding features, the image encoder in the contrastive learning pre-trained model is generally trained with another text encoder using contrastive loss. The training objective is the alignment of the text and image feature spaces. Using this image encoder can obtain relatively good overall abstract semantic information, but there is still considerable room for improvement in preserving and restoring image detail features.

[0048] In light of the above-mentioned practical situation, this disclosure provides an image generation method. First, the initial image is segmented according to the size requirements of the image encoder, ensuring the segmented image size is suitable for the encoder's needs. Then, the image encoder directly extracts features from the segmented image to obtain local features. Simultaneously, the image encoder extracts features from the initial image to obtain global features. The local features are then fused into the global features to form an image encoding result for input into a preset image generation model. This ensures that the image encoding result, while maintaining the number of features, enhances the correlation between local features using global features and preserves the detailed features of the initial image. This solves the problem of size changes during compression, enabling the image encoding result to more accurately represent image information. Ultimately, this improves the level of detail in the original image information during image generation, enhancing image generation efficiency and user experience.

[0049] Figure 2 A flowchart illustrating an exemplary method 200 provided according to embodiments of the present disclosure is shown. Optionally, method 200 can be used to generate images, particularly personalized text-based images. Method 200 can be... Figure 1 This can be implemented by terminal device 102, or by... Figure 1 The system 100 is implemented.

[0050] like Figure 2 As shown in the embodiments of this disclosure, the image generation method proposed by example may specifically include the following steps.

[0051] Step 202: Obtain the initial image and initial text.

[0052] In this step, the initial text and initial image are both initial data used for image generation. This text or image can be provided by the user through a corresponding port when using the image generation tool, or it can be imported in batches through tables or links. In some examples, user 108 can upload the initial text or initial image to the image generation tool on server 104 using terminal device 102, or user 108 can control the image generation tool on server 104 to obtain the initial text or initial image from the appropriate channel through operations on terminal device 102. Of course, in other examples, user 108 can also obtain the corresponding initial text or initial image from server 104 using terminal device 102. This example does not specifically limit this.

[0053] For example, such as Figure 3 As shown, the initial image can be in the form of the two images on the left, or the user can freely choose the initial image according to their needs; similarly, the initial text can be... Figure 3 As shown, it is a descriptive text formed by a paragraph or multiple prompts, used to limit the scene, screen structure, and included elements that the user wants to generate.

[0054] Step 204: Determine a segmentation scheme based on the size of the initial image, and segment the initial image according to the segmentation scheme to generate at least one segmented image.

[0055] In this step, after the initial image is determined, a segmentation scheme needs to be determined based on its size. Since the image encoder involved in the subsequent steps has certain size requirements for the input image data, in order to make the initial image meet the needs of the image encoder, in addition to scaling the initial image, it can also be segmented into the size required by the image encoder and then input into the image encoder part by part. Of course, if multiple image encoders are set, multiple parts can be encoded simultaneously.

[0056] Furthermore, in some examples, since the required input image size for the image encoder is known (this required image size can be referred to as the target size), the initial image can be directly segmented according to the target size. For example, starting from one endpoint of the initial image, the image can be segmented sequentially according to the target size to form a segmentation scheme. The resulting segmented local images are then called segmented images.

[0057] However, since the size of the initial input image is not fixed, and the size of the initial image is not necessarily proportional to the target size, this segmentation method is not friendly to the image edges. Sometimes an edge image may contain very little of the initial image content. In this case, it may be necessary to scale the edge image with large deformation, causing a large change in the image content. This will also lead to the loss or distortion of details in this part.

[0058] Therefore, in some examples, after determining the target size, it can be compared with the size of the initial image. Based on the comparison result, the number and arrangement of the segmented images can be determined, and then the initial image can be uniformly segmented according to the number and arrangement. Specifically, image size is generally represented in the form of length * width. Assuming the target size is 224 * 244, the ratio between the initial image and the target size can be determined in both the length and width directions. The ratio can then be rounded to the nearest integer. For example, if the initial image size is 1024 * 512, the length ratio is 4.57, rounded to 4; the width ratio is 2.098, rounded to 2. This determines that there are 8 segmented images, arranged in two rows and four columns. Then, the initial image can be uniformly segmented according to the determined number and arrangement, ensuring that the segmented images are all the same size. In this case, it is segmented into 8 256 * 256 image blocks (i.e., segmented images), thus determining the segmentation scheme. The segmented image obtained in this way has a size equal to or close to the target size. Even if scaling adjustments are needed, the adjustment range is very limited, and the impact on image details is minimal. That is, in some examples, determining the segmentation scheme based on the size of the initial image includes: determining the target size; determining the number and arrangement of the segmented images based on the ratio between the size of the initial image and the target size; and uniformly dividing the initial image according to the number and arrangement to generate the segmentation scheme.

[0059] In other examples, the number of segmented images can be determined beforehand based on specific needs, such as a minimum segment of 1 and a maximum segment of 6. Then, based on these two segmentations, all feasible schemes can be generated and recorded as preset schemes in a preset scheme set. Specifically, taking a minimum segment of 1 and a maximum segment of 6 as an example, the preset schemes are (1,1), (1,2), (1,3), (1,4), (1,5), (1,6), (2,1)(2,2), (2,3), (3,1), (3,2), (4,1), (5,1), (6,1). In each preset scheme, the first number represents the number of rows, and the second number represents the number of columns. After obtaining the dimensions of an initial image, a preset scheme can be selected from a set of preset schemes based on the aspect ratio of the target size and the aspect ratio of the initial image. The method of aspect ratio comparison can be the same as or similar to that in the previous embodiment. Then, based on the comparison result, a preset scheme with the same ratio is selected from the set of preset schemes as the segmentation scheme. That is, in some examples, determining the segmentation scheme based on the dimensions of the initial image includes: determining the target size, a set minimum segmentation number, and a set maximum segmentation number; generating a set of preset schemes based on the minimum segmentation number and the maximum segmentation number; and selecting from the set of preset schemes based on the aspect ratio of the target size and the aspect ratio of the initial image size, thereby selecting the segmentation scheme.

[0060] Furthermore, in some examples, multiple preset schemes in the preset scheme set may have the same aspect ratio. For example, in a scenario where the minimum segmentation is 1 and the maximum segmentation is 8, preset schemes (2, 1) and (4, 2) may appear, which are essentially the same in terms of aspect ratio. Therefore, if multiple preset schemes can be selected from the preset scheme set based on the aspect ratio, the image size after segmenting the initial image according to these preset schemes can be determined. Then, the preset scheme whose segmented image size is closest to the target size is selected as the segmentation scheme. For example, taking the initial image size of 1024*512 in the aforementioned example, according to the preset schemes (2, 1) and (4, 2), the segmented image sizes are 512*512 and 256*256, respectively. Therefore, the preset scheme closer to the target size of 224*244, namely (4, 2), is selected as the segmentation scheme. In some examples, the selection from the set of preset schemes includes: in response to being able to select at least two preset schemes from the set of preset schemes based on the aspect ratio, determining the image size after segmentation using the at least two preset schemes; and selecting the preset scheme whose segmented image size is closest to the target size as the segmentation scheme.

[0061] In some examples, after uniformly segmenting the initial image according to the aforementioned segmentation scheme, at least one segmented image is generated. The size of these segmented images may not necessarily meet the target size requirement. If the size of the segmented image does not meet the target size, it can be scaled according to the target size (in most scenarios, this is downsampling) to adjust the image size to meet the target size requirement. That is, in some examples, generating at least one segmented image includes: determining whether the size of any image segmented according to the segmentation scheme is consistent with the target size; in response to a mismatch, scaling the size of the any segmented image according to the target size, and using the scaled image as the at least one segmented image.

[0062] Step 206: Use an image encoder to extract features from the initial image and the at least one segmented image to obtain global features and local features. Then, fuse the global features and the local features to obtain the image encoding result.

[0063] In this step, an image encoder is used to extract features from both the initial image and the segmented image. Global features are extracted from the initial image, and local features are extracted from the segmented image. Since global features cannot accurately capture the details of image features, and local features have weak correlations, affecting the extraction of overall semantic information, feature fusion is necessary. For the specific feature fusion method, since both are feature extractions from the initial image, they will have some correlation at the pixel level. For example, although a global feature may correspond to a scaled version of the initial image, it must also indirectly correspond to at least one pixel in the initial image. Similarly, each local feature must directly or indirectly correspond to at least one pixel in the initial image. Therefore, the relationship between global and local features can be established based on whether the corresponding pixels are the same. After establishing the relationship, appropriate feature fusion plugins, such as the Token mixer feature fusion tool, can be used to fuse the two features. After fusion, local features can be attached to global features, using global features to control the number of features and improve the correlation between local features; global features, in turn, use local features to enhance the description of image details. The resulting image encoding is the feature data after mixing.

[0064] In some examples, building upon the feature extraction described above, to further achieve multi-level feature extraction and improve the perception of image details while ensuring the accuracy of semantic information, the image encoder for feature extraction can be configured in multiple ways. Specifically, it can include at least an image encoder based on a multimodal contrastive learning pre-trained model for extracting abstract semantic information, and an image encoder based on a data-interpolation neural network for image detail perception. This allows the contrastive learning pre-trained model to extract high-order abstract semantic information from the corresponding image, and the data-interpolation neural network to extract low-order image detail information. Specifically, the multimodal contrastive learning pre-trained model image encoder can be used to extract features from the initial image to obtain global abstract semantic features; the data-interpolation neural network image encoder can be used to extract features from the initial image to obtain global image detail features. Similarly, the multimodal contrastive learning pre-trained model image encoder can be used to extract features from at least one segmented image to obtain local abstract semantic features; the data-interpolation neural network image encoder can be used to extract features from at least one segmented image to obtain local image detail features. Finally, for global abstract semantic features and image detail features, or for local abstract semantic features and image detail features, the two are directly merged by concatenation to complete feature extraction. That is, in some examples, the image encoder includes at least an image encoder of a multimodal contrastive learning pre-trained model and an image encoder of a data-interpolation neural network; the feature extraction of the initial image and the at least one segmented image using the image encoder includes: using the image encoder of the multimodal contrastive learning pre-trained model and the image encoder of the data-interpolation neural network to extract features from the initial image or the at least one segmented image respectively, to obtain abstract semantic features and image detail features; and concatenating the abstract semantic features with the corresponding image detail features to complete the feature extraction.

[0065] Furthermore, abstract semantic features and image detail features can be combined in different dimensions depending on the specific scenario. To ensure the best effect after combination, the abstract semantic features and image detail features can be combined along the color channel dimension, i.e., combined along the channel dimension. Of course, during combination, global abstract semantic features are combined with global image detail features, while local abstract semantic features are combined with local image detail features. That is, in some examples, combining the abstract semantic features with the corresponding image detail features includes combining the abstract semantic features with the image detail features along the color channel dimension.

[0066] Furthermore, regarding the initial image or global features, since the size of the initial image may not meet the requirements of the image encoder—specifically, it may not meet the requirements of the image encoder of the multimodal contrastive learning pre-trained model or the image encoder of the data-interpolation neural network—the initial image can be scaled up to the required size according to the specific image encoder. Then, global feature extraction is performed. That is, in some examples, before feature extraction from the initial image, the method further includes scaling the initial image according to the image size required by the image encoder of the multimodal contrastive learning pre-trained model or the image encoder of the data-interpolation neural network.

[0067] In some examples, for the fusion of global and local features, in addition to the methods shown in the previous examples, a relationship can be established based on the feature similarity between the two to complete the fusion. Specifically, for any global feature vector in the global features, the similarity between each local feature vector and the global feature vector can be calculated. This similarity can then be used as a weight to establish a relationship between each local feature vector and the global feature vector, thus forming a matrix of the global feature vector and all local feature vectors. This process continues until all global feature vectors have a similar matrix. Once all matrices are established, the fusion of global and local features is considered complete. This fusion method maintains the same number of features as the global features but adds detailed information about the local features on top of the global features. Specifically, this fusion can be achieved using an attention mechanism. A token mixer can be constructed using an attention layer and a linear mapping layer (MLP) to obtain the final image encoding result. That is, in some embodiments, the feature fusion of the global features and the local features includes: determining the similarity between any first feature included in the global features and each second feature included in the local features; and establishing a connection between the any first feature and each second feature using the similarity as a weight, thereby performing the feature fusion. Here, the first feature is a specific feature in the global features, and the second feature is a specific feature in the local features.

[0068] Step 208: Input the image encoding result and the initial text into a preset image generation model to generate the target image.

[0069] In this step, the obtained image encoding result can be input along with the initial text into the corresponding image generation model. The image generation model then performs image generation to obtain the target image. The specific image generation model can be a T2I model or other image generation models. Figure 3 The image shown is a schematic diagram of the input initial image and initial text, and the generated target image. The image below each initial text is the target image generated based on the corresponding initial image and initial text.

[0070] Finally, the generated target image can be displayed on the corresponding device to provide feedback to the operator. Of course, in other examples, the output method of the target image is not limited to display; it can also be used to store, display, use, or further process the target image. The specific output method of the target image can be flexibly selected according to different application scenarios and implementation needs.

[0071] For example, in an application scenario where the method in this example is executed on a single device, the target image can be directly displayed on the display component (monitor, projector, etc.) of the current device, so that the operator of the current device can directly see the content of the target image on the display component.

[0072] For example, in an application scenario where the method in this example is executed on a system composed of multiple devices, the target image can be sent to other pre-defined devices within the system, i.e., synchronization terminals, as receivers, via any data communication method (wired connection, NFC, Bluetooth, Wi-Fi, cellular network, etc.), so that the synchronization terminals can perform subsequent processing. Optionally, the synchronization terminal can be a pre-defined server, which is generally located in the cloud and serves as a data processing and storage center, capable of storing and distributing the target image; wherein, the receivers of the distribution are the terminal devices, and the owners or operators of these terminal devices can be operators (users) of the image generation operation, maintainers of the image generation tool, image supervisors, etc.

[0073] For example, in an application scenario where the method in this example is executed on a system consisting of multiple devices, the target image can be directly sent to a preset terminal device through any data communication method. The terminal device can be one or more of the devices listed in the preceding paragraphs.

[0074] In some examples, the various models mentioned in the scheme can be further jointly trained based on their own pre-training. That is, the image encoder, image generation model, and feature fusion mixer can be jointly trained after pre-training. The main purpose of joint training is to further optimize the parameters of the feature fusion mixer and the adapter layer of the image generation model. In other words, an adapter will be set in the image generation model of this scheme, and the parameters of the adapter are also the optimization target of joint training. Therefore, in the specific joint training process, the first stage is to freeze the pre-training parameters of the image generation model (e.g., the T2I model) and the image encoder, and only optimize the parameters of the token mixer to align the extracted image encoding results with the text space of the original image generation model, so as to facilitate information interaction and achieve better control capabilities. The second stage is to freeze the pre-training parameters of the image generation model and the image encoder again, but simultaneously optimize the parameters of the adapter in the token mixer and the image generation model to obtain the overall preservation effect. That is, in some examples, the image encoder, the image generation model, and the feature fusion fusion fusion unit are trained in a joint training manner; wherein, the image generation model includes at least one adapter layer; the training process includes: freezing the parameters of the image encoder and the image generation model; optimizing the parameters of the fusion unit separately; and then jointly training the at least one adapter layer and the parameter-optimized fusion unit.

[0075] In a specific application scenario, the implementation of this solution can be summarized as follows: First, considering multi-level resolution, existing pre-trained image encoders only support fixed resolutions. A block-based strategy can be used to enable encoding of images with arbitrary aspect ratios and high resolution. For example, assuming an input image of 1024*512, using a multimodal contrastive learning pre-trained model as a feature extractor, one approach is to directly downsample the image to 224*244 before extracting features. This leads to the aforementioned problems. Therefore, a strategy can be used to intelligently divide the image into eight 256*256 image blocks, and then downsample each 256*256 block to 224*244 before extracting features. Second, considering multi-level feature extraction, in addition to using an image encoder with a multimodal contrastive learning pre-trained model that has good abstract semantic understanding, an additional image encoder with a data-interpolation neural network that has good image detail perception can be added. The features of both are concatenated along the channel dimension. Finally, considering multi-level feature fusion, the increased number of image patches after high resolution leads to a corresponding increase in the number of features generated after image encoding. This potentially increases the computational load of the attention layer and reduces the correlation between different image patches, affecting the extraction of overall semantic information. Based on this, a global image patch can be added on top of the local image patches. Similarly, the features of the local image patches are fused into the global image features through attention, maintaining a small number of global features without losing local feature information, and strengthening the correlation between local image patches through the global image.

[0076] In specific applications, such as Figure 4 As shown, the input includes an initial image and an initial text. The initial image is processed through two branches: 1) Local feature extraction branch: the initial image is processed by a dynamic pachify strategy to obtain segmented images according to a determined segmentation scheme. Multiple segmented images are then processed by an image encoder to obtain local features. 2) Global feature extraction branch: the initial image is directly scaled globally to obtain a size suitable for the image encoder, and then processed by the image encoder to obtain global features. Afterwards, the local and global features are fused at multiple levels by a feature fusion unit (tokenmixer) to output the final image encoding result. Finally, the image encoding result and the initial text are injected into a pre-trained image generation model (e.g., a T2I model) to generate a personalized text-to-image result, i.e., the target image.

[0077] As can be seen from the above embodiments, the image generation method provided in this disclosure first segments the initial image, allowing for segmentation according to the size requirements of the image encoder, ensuring the size of the segmented image is suitable for the encoder's needs. Then, the image encoder directly extracts features from the segmented image to obtain local features. Simultaneously, the image encoder extracts features from the initial image to obtain global features. The local features are then fused into the global features to form an image encoding result for input into a preset image generation model. This ensures that the image encoding result, while maintaining the number of features, enhances the correlation between local features using global features and preserves the detailed features of the initial image, thus resolving the impact of size changes during compression and enabling the image encoding result to more accurately represent image information. Ultimately, this improves the level of detail in the original image information during image generation, enhancing image generation efficiency and user experience.

[0078] It should be noted that the method of this disclosure embodiment can be executed by a single device, such as a computer or server. The method of this disclosure embodiment can also be applied in a distributed scenario, where multiple devices cooperate to complete the task. In such a distributed scenario, one of these devices may execute only one or more steps of the method of this disclosure embodiment, and the multiple devices will interact with each other to complete the method described.

[0079] It should be noted that the above description describes specific embodiments of this disclosure. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in a different order than that shown in the above embodiments and still achieve the desired results. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0080] Based on the same technical concept, corresponding to any of the above embodiments, this disclosure also provides an image generation apparatus 500. Figure 5 A schematic diagram of an exemplary apparatus 500 provided in an embodiment of this disclosure is shown. For example... Figure 5 As shown, the device 500 can be used to implement method 200 and may further include the following modules.

[0081] The first module 510 is used to acquire the initial image and initial text.

[0082] The second module 520 is used to determine a segmentation scheme based on the size of the initial image, and to segment the initial image according to the segmentation scheme to generate at least one segmented image.

[0083] The third module 530 is used to extract features from the initial image and the at least one segmented image using an image encoder to obtain global features and local features, and to fuse the global features and local features to obtain an image encoding result.

[0084] The fourth module 540 is used to input the image encoding result and the initial text into a preset image generation model to generate a target image.

[0085] In some exemplary embodiments, the second module 520 is further configured to:

[0086] Determine the target size, the minimum and maximum number of segments to be cut;

[0087] A preset scheme set is generated based on the minimum and maximum segmentation.

[0088] The segmentation scheme is selected from the preset scheme set based on the aspect ratio of the target size and the aspect ratio of the initial image size.

[0089] In some exemplary embodiments, the second module 520 is further configured to:

[0090] In response to the ability to select at least two preset schemes from the preset scheme set based on the aspect ratio, the image size after segmentation using the at least two preset schemes is determined;

[0091] The preset scheme whose image size after segmentation is closest to the target size is selected as the segmentation scheme.

[0092] In some exemplary embodiments, the second module 520 is further configured to:

[0093] Determine the target size;

[0094] The number and arrangement of the segmented images are determined based on the ratio between the size of the initial image and the target size.

[0095] The initial image is evenly divided according to the number and arrangement to generate the segmentation scheme.

[0096] In some exemplary embodiments, the second module 520 is further configured to:

[0097] Determine whether the size of any image segmented according to the segmentation scheme is consistent with the target size;

[0098] In response to the inconsistency between the two, the image after any segmentation is scaled according to the target size, and the scaled image is used as the at least one segmented image.

[0099] In some exemplary embodiments, the image encoder includes at least an image encoder based on a multimodal contrastive learning pre-trained model and an image encoder based on a data-interpolation neural network;

[0100] The third module 530 is also used for:

[0101] The image encoder of the multimodal contrastive learning pre-trained model and the image encoder of the data-interpolation neural network are used to extract features from the initial image or the at least one segmented image to obtain abstract semantic features and image detail features.

[0102] The abstract semantic features are concatenated with the corresponding image detail features to complete the feature extraction.

[0103] In some exemplary embodiments, the third module 530 is further configured to:

[0104] The abstract semantic features and the image detail features are concatenated in the color channel dimension.

[0105] In some exemplary embodiments, the third module 530 is further configured to:

[0106] The initial image is scaled according to the image encoder of the multimodal contrastive learning pre-trained model or the image encoder of the data-interpolation neural network.

[0107] In some exemplary embodiments, the third module 530 is further configured to:

[0108] Determine the similarity between any first feature contained in the global features and each second feature contained in the local features;

[0109] Using the similarity as a weight, a connection is established between any first feature and each second feature, thereby performing feature fusion.

[0110] In some exemplary embodiments, the image encoder, the image generation model, and the feature fusion fusion unit are trained through joint training; wherein the image generation model includes at least one adapter layer; the training process includes:

[0111] The parameters of the image encoder and the image generation model are frozen.

[0112] The parameters of the fusion unit are optimized separately.

[0113] Then, the at least one adapter layer and the parameter-optimized fusion machine are jointly trained.

[0114] For ease of description, the above apparatus is described in terms of its functions, divided into various modules. Of course, in implementing this disclosure, the functions of each module can be implemented in one or more software and / or hardware.

[0115] The apparatus of the above embodiments is used to implement the corresponding method 200 in any of the foregoing embodiments, and has the beneficial effects of the corresponding method 200 embodiments, which will not be repeated here.

[0116] Based on the same technical concept, corresponding to any of the above embodiments, this disclosure also provides a computer device for implementing the above method 200. Figure 6 A schematic diagram of the hardware structure of an exemplary computer device 600 provided in an embodiment of this disclosure is shown. The computer device 600 can be used to implement... Figure 1 Terminal device 102. In some scenarios, this computer device 600 can also be used to implement... Figure 1 Server 104, Database Server 106.

[0117] like Figure 6 As shown, the computer device 600 may include: a processor 602, a memory 604, a network module 606, a peripheral interface 608, and a bus 610. The processor 602, memory 604, network module 606, and peripheral interface 608 are interconnected within the computer device 600 via the bus 610.

[0118] Processor 602 may be a central processing unit (CPU), image processor, neural network processor (NPU), microcontroller (MCU), programmable logic device, digital signal processor (DSP), application-specific integrated circuit (ASIC), or one or more integrated circuits. Processor 602 can be used to perform functions related to the techniques described in this disclosure. In some embodiments, processor 602 may also include multiple processors integrated as a single logic component. For example, such as... Figure 6 As shown, processor 602 may include multiple processors 602a, 602b and 602c.

[0119] Memory 604 can be configured to store data (e.g., instructions, computer code, etc.). Figure 6As shown, the data stored in memory 604 may include program instructions (e.g., program instructions for implementing method 200 of embodiments of this disclosure) and data to be processed (e.g., the memory may store configuration files of other modules, etc.). Processor 602 may also access the program instructions and data stored in memory 604 and execute the program instructions to operate on the data to be processed. Memory 604 may include volatile or non-volatile storage devices. In some embodiments, memory 604 may include random access memory (RAM), read-only memory (ROM), optical disk, magnetic disk, hard disk, solid-state drive (SSD), flash memory, memory stick, etc.

[0120] Network interface 606 can be configured to provide communication with other external devices to computer device 600 via a network. This network can be any wired or wireless network capable of transmitting and receiving data. For example, the network can be a wired network, a local wireless network (e.g., Bluetooth, WiFi, Near Field Communication (NFC), etc.), a cellular network, the Internet, or a combination thereof. It is understood that the type of network is not limited to the specific examples described above.

[0121] The peripheral interface 608 can be configured to connect the computer device 600 to one or more peripheral devices to enable information input and output. For example, peripheral devices may include input devices such as keyboards, mice, touchpads, touch screens, microphones, and various sensors, as well as output devices such as displays, speakers, vibrators, and indicator lights.

[0122] Bus 610 can be configured to transfer information between various components of computer device 600 (e.g., processor 602, memory 604, network interface 606, and peripheral interface 608), such as internal buses (e.g., processor-memory bus), external buses (USB port, PCI-E bus), etc.

[0123] It should be noted that although the architecture of the computer device 600 described above only shows the processor 602, memory 604, network interface 606, peripheral interface 608, and bus 610, in specific implementations, the architecture of the computer device 600 may also include other components necessary for normal operation. Furthermore, those skilled in the art will understand that the architecture of the computer device 600 described above may only include the components necessary for implementing the embodiments of this disclosure, and does not necessarily include all the components shown in the figures.

[0124] Based on the same technical concept, corresponding to the methods of any of the above embodiments, this disclosure also provides a non-transitory computer-readable storage medium storing computer instructions for causing the computer to perform the method 200 as described in any of the above embodiments.

[0125] The computer-readable medium of this embodiment includes permanent and non-permanent, removable and non-removable media, and information storage can be implemented by any method or technology. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transfer medium that can be used to store information accessible by a computing device.

[0126] The computer instructions stored in the storage medium of the above embodiments are used to cause the computer to perform the method 200 as described in any of the above embodiments, and have the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0127] Based on the same technical concept, corresponding to the method 200 in any of the above embodiments, this disclosure also provides a computer program product, including computer program instructions, which, when executed on a computer, cause the computer to perform the method 200 as described in any of the above embodiments. In some embodiments, the computer program instructions may be executed by one or more processors of the computer to cause the computer and / or the processor to perform the method 200. Corresponding to the execution entity for each step in each embodiment of the method 200, the processor performing the corresponding step may belong to the corresponding execution entity.

[0128] The computer program product of the above embodiments is used to cause the computer and / or the processor to perform the method 200 as described in any of the above embodiments, and has the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0129] Those skilled in the art should understand that the discussion of any of the above embodiments is merely exemplary and is not intended to imply that the scope of this disclosure (including the claims) is limited to these examples; within the framework of this disclosure, the technical features of the above embodiments or different embodiments can also be combined, the steps can be implemented in any order, and there are many other variations of different aspects of the embodiments of this disclosure as described above, which are not provided in detail for the sake of brevity.

[0130] Additionally, to simplify the description and discussion, and to avoid obscuring the embodiments of this disclosure, the provided drawings may or may not show well-known power / ground connections to integrated circuit (IC) chips and other components. Furthermore, the apparatus may be shown in block diagram form to avoid obscuring the embodiments of this disclosure, and this also takes into account the fact that the details of implementation of these block diagram apparatuses are highly dependent on the platform on which the embodiments of this disclosure will be implemented (i.e., these details should be fully understood by those skilled in the art). While specific details (e.g., circuitry) have been set forth to describe exemplary embodiments of this disclosure, it will be apparent to those skilled in the art that the embodiments of this disclosure may be implemented without these specific details or with variations thereof. Therefore, these descriptions should be considered illustrative rather than restrictive.

[0131] Although this disclosure has been described in conjunction with specific embodiments thereof, many substitutions, modifications, and variations of these embodiments will be apparent to those skilled in the art from the foregoing description. For example, other memory architectures (e.g., dynamic RAM (DRAM)) may be used with the embodiments discussed.

[0132] This disclosure is intended to cover all such substitutions, modifications, and variations that fall within the broad scope of the appended claims. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. An image generation method, comprising: Obtain the initial image and initial text; A segmentation scheme is determined based on the size of the initial image, and the initial image is segmented according to the segmentation scheme to generate at least one segmented image; The initial image and the at least one segmented image are subjected to feature extraction using an image encoder to obtain global features and local features. The global features and the local features are then fused to obtain the image encoding result. The image encoding result and the initial text are input into a preset image generation model to generate the target image.

2. The method of claim 1, wherein, The step of determining the segmentation scheme based on the size of the initial image includes: Determine the target size, the minimum and maximum number of segments to be cut; A preset scheme set is generated based on the minimum and maximum segmentation. The segmentation scheme is selected from the preset scheme set based on the aspect ratio of the target size and the aspect ratio of the initial image size.

3. The method of claim 2, wherein, The selection from the preset set of schemes includes: In response to the ability to select at least two preset schemes from the preset scheme set based on the aspect ratio, the image size after segmentation using the at least two preset schemes is determined; The preset scheme whose image size after segmentation is closest to the target size is selected as the segmentation scheme.

4. The method of claim 1, wherein, The step of determining the segmentation scheme based on the size of the initial image includes: Determine the target size; The number and arrangement of the segmented images are determined based on the ratio between the size of the initial image and the target size. The initial image is evenly divided according to the number and arrangement to generate the segmentation scheme.

5. The method according to claim 2 or 4, wherein, The generation of at least one segmented image includes: Determine whether the size of any image segmented according to the segmentation scheme is consistent with the target size; In response to the inconsistency between the two, the image after any segmentation is scaled according to the target size, and the scaled image is used as the at least one segmented image.

6. The method according to claim 1, wherein, The image encoder includes at least an image encoder based on a multimodal contrastive learning pre-trained model and an image encoder based on a data-interpolation neural network; The step of extracting features from the initial image and the at least one segmented image using an image encoder includes: The image encoder of the multimodal contrastive learning pre-trained model and the image encoder of the data-interpolation neural network are used to extract features from the initial image or the at least one segmented image to obtain abstract semantic features and image detail features. The abstract semantic features are concatenated with the corresponding image detail features to complete the feature extraction.

7. The method according to claim 6, wherein, The step of concatenating the abstract semantic features with the corresponding image detail features includes: The abstract semantic features and the image detail features are concatenated in the color channel dimension.

8. The method according to claim 6, wherein, Before performing feature extraction on the initial image, the method further includes: The initial image is scaled according to the image encoder of the multimodal contrastive learning pre-trained model or the image encoder of the data-interpolation neural network.

9. The method according to claim 1, wherein, The feature fusion of the global features and the local features includes: Determine the similarity between any first feature contained in the global features and each second feature contained in the local features; Using the similarity as a weight, a connection is established between any first feature and each second feature, thereby performing feature fusion.

10. The method according to claim 1, wherein, The image encoder, the image generation model, and the feature fusion fusion unit are trained jointly; wherein the image generation model includes at least one adapter layer; the training process includes: The parameters of the image encoder and the image generation model are frozen. The parameters of the fusion unit are optimized separately. Then, the at least one adapter layer and the parameter-optimized fusion machine are jointly trained.

11. An image generation apparatus, comprising: The first module is used to obtain the initial image and initial text; The second module is used to determine a segmentation scheme based on the size of the initial image, and to segment the initial image according to the segmentation scheme to generate at least one segmented image; The third module is used to extract features from the initial image and the at least one segmented image using an image encoder to obtain global features and local features, and to fuse the global features and local features to obtain an image encoding result. The fourth module is used to input the image encoding result and the initial text into a preset image generation model to generate the target image.

12. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein, When the processor executes the program, it implements the method as described in any one of claims 1 to 10.

13. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method as described in any one of claims 1 to 10.

Images