Image synthesis method, image synthesis model training method, equipment and product

Abstract

The invention provides an image synthesis method, an image synthesis model training method, equipment and a product. The method comprises the following steps: acquiring image style prompt information and a user face image; inputting the user face image and the image style indication information into an image synthesis model to generate a synthesis image with a specified style; wherein the image synthesis model is obtained by training a style training sample constructed by image style elements bound with element prompt information; the image style element is an element which is obtained by segmenting the original image and is used for representing the image style.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and particularly to image synthesis, methods, devices, and products for training an image synthesis model. Background Art

[0002] With the development of image synthesis technology, especially in combination with artificial intelligence models, the image synthesis ability has been significantly improved.

[0003] In some work scenarios, when performing image synthesis, many detailed image elements are involved in the synthesis. For example, when synthesizing a work photo, different image elements such as a specified work uniform, company logo, and a tie with a specified pattern need to be synthesized into the same image. However, during the synthesis, it is easy to have a deviation in the synthesis position of the image elements or the selected image elements are inaccurate. Ultimately, the effect of the synthesized image cannot meet the user's requirements. Summary of the Invention

[0004] The present disclosure provides an image synthesis method, a method for training an image synthesis model, a device, and a product.

[0005] According to a first aspect of the present disclosure, an image synthesis method is provided. The method specifically includes: obtaining image style prompt information and a user's facial image; inputting the user's facial image and the image style indication information into an image synthesis model to generate a synthesized image with a specified style; wherein, the image synthesis model is obtained after being trained with a style training sample constructed by using an image style element bound with element prompt information; and the image style element is an element obtained by segmenting an original image and used to represent the image style.

[0006] Based on the above content, it can be known that when training the image synthesis model, the original image used for training is segmented into multiple image style elements. Corresponding element prompt information is assigned to each image style element, thereby establishing a binding relationship between the image style element and the element prompt information. Furthermore, a style training sample is constructed by using the image style element and the element prompt information. The image synthesis model obtained by training with the style training sample can accurately realize the combination of different image style elements. During subsequent image synthesis, multiple image style elements can be accurately synthesized into the same synthesized image according to the received prompt information.

[0007] Before inputting the user's facial image and the image style indication information into the image synthesis model according to at least one embodiment of the present disclosure, it further includes: if the image style prompt information contains element prompt information for a second image style element, then using a detection model to determine the position of the smeared image element included in the first image style element; so as to stack the second image style element at the position of the smeared image element when using the image synthesis model to synthesize an image.

[0008] According to a second aspect of the present disclosure, there is provided a method for training an image synthesis model. The method specifically includes: obtaining image style elements for image synthesis from original images; binding element hint information to each image style element; constructing style training samples using the image style elements bound with the element hint information; and training a model using the style training samples to obtain an image synthesis model, so as to synthesize a synthesized image of a specified style using the image synthesis model.

[0009] Obtaining image style elements for image synthesis from original images according to at least one embodiment of the present disclosure includes: segmenting at least one original image to obtain a plurality of image elements; and saving the image elements other than the face images as the image style elements.

[0010] After segmenting at least one original image to obtain a plurality of image elements according to at least one embodiment of the present disclosure, the method further includes: if the image style elements include a first image style element and a second image style element whose positional relationship is an up-and-down stacking relationship, determining the positional relationship between the second image style element and the first image style element; wherein, the second image style element is stacked on the first image style element, or the second image style element is embedded in the first image style element and the second image style element is smaller than the first image style element; and smearing the position where the second image style element is located in the first image style element according to the size of the second image style element to obtain a smeared image element.

[0011] Binding element hint information to each image style element according to at least one embodiment of the present disclosure includes: classifying the image style elements and determining the class names; assigning numbers to a plurality of image style elements having the same class name; generating hint information using the class names and the numbers, and binding the hint information to the corresponding image style elements.

[0012] According to at least one embodiment of the present disclosure, the image synthesis model is constructed from a first basic model, a portrait low-rank matrix, and a style low-rank matrix; training the image synthesis model using the style training samples to obtain the image synthesis model includes: obtaining portrait training samples; training to obtain the portrait low-rank matrix using the portrait training samples and a second basic model; and training to obtain the style low-rank matrix using the style training samples and a third basic model; wherein, the second basic model is obtained by optimizing the first basic model using a portrait data set, and the third basic model is obtained by optimizing the first basic model using a style data set; and constructing the image synthesis model based on the first basic model, the portrait low-rank matrix, and the style low-rank matrix.

[0013] In at least one embodiment of the present disclosure, in response to a user's grouping request, groups are established using image style elements of different class names, and group hint information is assigned to the groups.

[0014] Based on the above, when training an image synthesis model, the original images used for training are segmented into multiple image style elements. Corresponding element hint information is assigned to each image style element, thereby establishing a binding relationship between the image style elements and the element hint information. Furthermore, style training samples are constructed using the image style elements and the element hint information. The image synthesis model trained using the style training samples can accurately combine different image style elements. During subsequent image synthesis, multiple image style elements can be accurately synthesized into the same synthesized image according to the received hint information.

[0015] According to a third aspect of the present disclosure, an electronic device is provided, including: a memory that stores execution instructions; and a processor that executes the execution instructions stored in the memory, such that the processor executes the method described in the first aspect or the second aspect of any one of the embodiments of the present disclosure.

[0016] According to a fourth aspect of the present disclosure, a readable storage medium is provided, in which execution instructions are stored, and when the execution instructions are executed by a processor, they are used to implement the method described in the first aspect or the second aspect of any one of the embodiments of the present disclosure.

[0017] According to a fifth aspect of the present disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements the method described in the first aspect or the second aspect of any one of the embodiments of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The drawings illustrate exemplary embodiments of the present disclosure and, together with the description, are used to explain the principles of the present disclosure. These drawings are included to provide a further understanding of the present disclosure and are included in this specification and form a part of this specification.

[0019] Figure 1 It is a schematic flowchart of a method for training an image synthesis model provided by the present disclosure.

[0020] Figure 2 It is a schematic diagram of smeared image elements illustrated by the present disclosure.

[0021] Figure 3 It is a schematic diagram of the model training process illustrated by the present disclosure.

[0022] Figure 4 It is a schematic flowchart of an image synthesis method provided by the present disclosure.

[0023] Figure 5 The structural schematic block diagram of an image synthesis model training device according to an embodiment of the present disclosure.

[0024] Figure 6 The structural schematic block diagram of an image synthesis device according to an embodiment of the present disclosure.

[0025] Figure 7 The structural schematic block diagram of an electronic device according to an embodiment of the present disclosure. Detailed implementation manners

[0026] The present disclosure will be further described in detail below with reference to the accompanying drawings and examples. It can be understood that the specific examples described herein are only used to explain the relevant content and do not limit the present disclosure. Additionally, it should be noted that for the convenience of description, only parts related to the present disclosure are shown in the accompanying drawings.

[0027] It should be noted that, without conflict, the embodiments in the present disclosure and the features in the embodiments can be combined with each other. The technical solutions of the present disclosure will be described in detail below with reference to the accompanying drawings and embodiments.

[0028] The field of portrait generation is one of the current hottest fields of Artificial Intelligence Generated Content (AIGC). One of the scenarios is the synthesis of work photos. In the prior art, when training a model for work photo synthesis, usually a whole image is used as a training sample and input into the image synthesis model for training. As a result, although the trained image synthesis model can complete the image synthesis task, the synthesis effect cannot meet the user's requirements. For example, the color of the suit in the synthesized image is inaccurate, the pattern of the tie in the synthesized work photo is incorrect, and the position of the company logo in the synthesis is incorrect, etc. Therefore, there is an urgent need for a solution that can achieve accurate image synthesis.

[0029] For the convenience of description and to make the technical solutions of the specific implementation manners of the present disclosure easier to understand, before describing the image synthesis method implemented in the present disclosure, the technical terms involved in the specific implementation manners of the present disclosure are explained as follows.

[0030] An image style element refers to some basic image elements segmented from a complete image. For example, the suit, tie, etc. segmented from a work photo image are all basic image elements. These image elements can be used to synthesize various different styles of images.

[0031] The base model refers to a generative model, such as the Diffusion model, the transform model, etc. This base model can be fine-tuned using a dataset as needed. Moreover, this base model can also be embedded with a corresponding low-rank matrix (such as a style low-rank matrix).

[0032] The LoRA model is a technique specifically used for fine-tuning large language models, which is referred to as a low-rank matrix in the following content. Different low-rank matrices with different functions need to be specifically trained using different training samples and base models. Its core idea is to reduce the number of parameters to be adjusted by introducing a low-rank matrix without changing the structure of the original large language model (i.e., the base model mentioned later), thereby reducing the cost of fine-tuning while maintaining or nearly maintaining the original performance of the model.

[0033] Figure 1 It is a schematic flowchart of a method for training an image synthesis model provided by this disclosure. As Figure 1 shown, the method includes steps 101 to 104. Among them, this method can be executed by an electronic device such as a server (local server or cloud server).

[0034] Specifically, Figure 1 the method shown includes: Step 101: Obtain image style elements for image synthesis from the original image.

[0035] It should be noted that the original image mentioned here can be understood as an image containing the basic elements required for image synthesis. For example, a complete ID photo, in which the tie, suit, company logo, etc. can all be image style elements. In addition, work uniforms can also be an image style element, and the company logo or emblem printed on the work uniform can also be an image style element. The tie colors, patterns, suit colors, and company emblems selected for ID photos of different companies are different.

[0036] In different application scenarios, different original images are selected, and the image style elements segmented from the original image are also different. To enable the image synthesis model to have a better synthesis effect, it is necessary to train the model using the corresponding image style elements (such as fine-tuning the base model or training the LoRA model).

[0037] Step 102: Bind element prompt information to each image style element.

[0038] The prompt information (Prompt) mentioned here is a text or statement fragment used to trigger and guide an artificial intelligence language model to generate a specific output. These text inputs can be in any form such as questions, descriptions, keywords, sentence fragments, etc., aiming to help the model understand and execute the user's instructions or tasks.

[0039] When establishing the training samples, binding element hint information to the image style elements can provide the context of the input information for the model, helping the model understand the one-to-one binding relationship between the image style elements and the element hint information. And in the subsequent inference process, after the image synthesis model receives the user's Prompt, it can accurately understand which image style element the user needs.

[0040] Step 103: Construct style training samples using the image style elements bound with element hint information.

[0041] When constructing the training samples, instead of directly using an entire image containing many image style elements as the training sample, the segmented image style elements are used as the training sample. This training sample has better training effects when training the model.

[0042] In practical applications, the training samples can be grouped according to different types of image style elements. For example, all the segmented tie images are grouped as one group, and different ties are distinguished by the element hint information within the group; all the segmented suit images are grouped as one group, and different suits are distinguished by the element hint information within the group. Then, the style training samples are constructed jointly by multiple groups.

[0043] Step 104: Use the style training samples to train the model to obtain an image synthesis model, so as to use the image synthesis model to synthesize a synthetic image of a specified style.

[0044] The model to be trained here can be a generative model, such as a Diffusion model, a Transformer model, etc. After training with these basic models, an image synthesis model is obtained.

[0045] In an alternative solution, the model to be trained can be a Diffusion model. During training, a style low-rank matrix (style LoRA) can be trained using the style training samples. The style low-rank matrix mentioned here can be obtained by training based on the basic model. To improve the training effect, the basic model can also be fine-tuned, that is, the first basic model can be fine-tuned using the style fine-tuning dataset to obtain a third basic model for training the style low-rank matrix. If a portrait low-rank matrix is also needed when synthesizing the image, the first basic model can be fine-tuned using the portrait fine-tuning dataset to obtain a second basic model. Furthermore, a portrait low-rank matrix can be obtained by training using the portrait training set and the second basic model. The specific training process will be described in detail in the subsequent embodiments and will not be repeated here.

[0046] Through this method, the original image is segmented into multiple image style elements, and the segmented image style elements are associated with corresponding element prompt information. This allows the model to accurately understand the one-to-one correspondence between image style elements and element prompt information during training, and accurately distinguish between different image style elements of the same type. When the trained model is subsequently used for image synthesis, it can accurately combine multiple image style elements into a single composite image based on the received prompt information.

[0047] In one or more embodiments of the present disclosure, obtaining image style elements for image synthesis from original images includes: segmenting at least one original image to obtain multiple image elements; and saving image elements other than facial images as image style elements.

[0048] There are many ways to segment the original image. For example, edge detection algorithms (such as Sobel and Canny) can be used to detect edges in the image. Based on this edge information, morphological operations (such as dilation, erosion, opening, and closing) can be combined to extract the outlines of elements such as ties and suits.

[0049] Alternatively, deep learning models can be used to identify and segment the original image. When performing image segmentation, users can specify the segmentation type. For example, they can specify to segment out a tie, suit, company emblem, brooch, or hairstyle. If two stylistic elements in the original image are connected side by side, direct segmentation is possible. For example, when segmenting a head image from a suit, due to the distinct shape and texture features, traditional image processing methods (such as edge detection and threshold segmentation) or deep learning-based semantic segmentation methods can be used. These methods are generally able to quickly identify and segment the head and neck and shoulders (that is, the suit image including the neck).

[0050] If two image style elements are stacked one on top of the other (for example, the second image style element is stacked on top of the first), the second image style element needs to be extracted from the first. For example, when extracting a tie from an image of a suit, due to the similarity between the tie and the suit, a more advanced segmentation method is required. For example, instance segmentation can be used, as it can distinguish different instances within the same category. In addition, methods such as shape priors, color space transformations, or conditional random fields can be considered to improve segmentation accuracy.

[0051] After splitting an image into multiple image style elements, during the model training process, the model can perform more refined control over each image style element. This helps to achieve more precise effects in image synthesis tasks. By splitting the image elements, the model can focus more on learning the key features of each element, thereby reducing the interference of redundant information. In addition, after splitting the image into multiple elements, each element can be trained separately. This helps to accelerate the training process and improve the convergence speed of the model.

[0052] Through the above method, after accurately segmenting the original image, the required image elements are obtained. The image elements obtained by segmentation not only contain the image style elements required by the user, but also contain some useless image elements, such as the face image element and background image element in the original image, etc. By performing a more fine-grained split of the original image, it is convenient to establish a one-to-one binding relationship between the image style elements and the element hint information subsequently. Thereby, the model training effect can be effectively improved.

[0053] In one or more embodiments of the present disclosure, after segmenting at least one original image to obtain multiple image elements, it further includes: if the image style elements include a first image style element and a second image style element whose positional relationship is an up-and-down stacked relationship, determining the positional relationship between the second image style element and the first image style element; wherein, the second image style element is stacked on the first image style element, or, the second image style element is embedded in the first image style element, and the second image style element is smaller than the first image style element; obtaining a smeared image element by smearing at the position of the second image style element in the first image style element according to the size of the second image style element.

[0054] In practical applications, some prominent image style elements can be directly cropped or segmented using the method described above. However, for some less prominent image style elements (which can be understood as relatively small image style elements, such as company emblems, brooches, etc.), the direct segmentation effect is relatively poor, or, when synthesizing some small image style elements subsequently, there are often problems such as deviation in the synthesis position, blurring after synthesis, and even possible synthesis errors. Therefore, in order to improve the synthesis effect of the image synthesis model on non-prominent image style elements, it can be assisted by adding smeared image elements.

[0055] Specifically, the first image style element is a relatively large image element, and the second image style element is a relatively small image element stacked on the first image style element, or, the second image style element is an image element embedded in the first image style element.

[0056] To meet the requirement of replacing the second image style element in the subsequent image synthesis process (i.e., in the subsequent synthesis process, the matching relationship between the first image style element and the second image style element can be adjusted as needed), the second image style element needs to be separated from the first image style element.

[0057] During training, a detection model (such as the Yolo V3 detection model) can be used to detect the relative position relationship between the first image style element and the second image style element. In other words, mark the position of the second image style element in the first image style element.

[0058] Next, a smeared image element is generated at the marked position in the first image style element. This smeared image element should exactly cover the original second image style element. To facilitate the subsequent recognition of this smeared image element, there should be an obvious color difference between the smeared image element and the first image style element. In practical applications, there is no strict limit on the shape of the smeared image element.

[0059] For example, as Figure 2 is a schematic diagram of the smeared image element illustrated in this disclosure. As can be seen from Figure 2 , the first image style element is a suit, and the second image style element is the company logo decorated on the suit. The company logo is relatively small and occupies a fixed position on the collar of the suit in the first image style element. In the subsequent image synthesis, the user may choose to match the suit in the first image style element with other company logos. Therefore, the second image style element in the first image style element needs to be modified into a smeared image element, such as Figure 2 shown as a smeared rectangular smeared image element. To help the model better understand the meaning of the smeared image element and its role in the subsequent image synthesis during the training process, element hint information with a unique identification function can be bound to the smeared image element. For example, the corresponding element hint information can be marked with the first letters of the pinyin as TMYS - 1.

[0060] By the above method, the smeared image element can be added to help the image synthesis model accurately identify the position of the smeared image element in the first image style element. The smeared image element can be directly marked in the first image style element, ensuring the accurate splicing of the second image style element and the first image style element during the subsequent image synthesis.

[0061] In one or more embodiments of this disclosure, binding element hint information to each image style element includes: classifying the image style element and determining the class name; assigning numbers to multiple image style elements with the same class name; generating hint information using the class name and number, and binding the hint information to the corresponding image style element.

[0062] In practical applications, in order to help the model better understand and distinguish each image style element, unique element hint information can be added to each image style element. By binding the element hint information to the image style element, it can help the model accurately distinguish different image style elements with relatively high similarity.

[0063] When binding the element hint information, the image style elements can be classified first. For example, they can be classified into tie types, suit types, company logo types, brooch types, lapel pin types, etc. Further, for each type of marked class name, for example, the class names can be: tie, suit, company logo, brooch, lapel pin.

[0064] Since tie types, suit types, company logo types, brooch types, lapel pin types, etc. have their own unique colors, patterns, designs, etc., further, multiple image style elements corresponding to the same class name can be numbered. For example, the numbers for tie types are A1, A2, A3... The numbers for suit types are B1, B2, B3, etc.

[0065] Then, the classification name and the number are combined together as the element hint information. For example, the element hint information for tie types obtained can be: tie A1, tie A2, etc.; the element hint information for suit types can be: suit B1, suit B2, etc. In the subsequent image synthesis process, they can be freely combined as needed. For example, it can be tie A1 paired with suit B1, or tie A1 paired with suit B2.

[0066] Through this binding method, it can help the model distinguish different types of image style elements and accurately distinguish the same type of image style elements.

[0067] By allowing the model to be trained to learn the detailed Prompt (element hint information) and establish a corresponding binding relationship with the image style element, the model can more accurately understand the semantic content of the image. This helps improve the accuracy of the model in the image synthesis task. Moreover, during the training process, the model can learn the one-to-one correspondence between the features of the image style element and the Prompt, rather than just memorizing specific images. This enables the model to still accurately understand and process when facing new and unseen images. Through more refined Prompt annotation, users can more precisely control the output of the image synthesis model. For example, in the image synthesis task, users can adjust the Prompt to change the detailed features such as the color, shape, pattern, and dark grid of the synthesized image.

[0068] In one or more embodiments of the present disclosure, the image synthesis model is constructed from a first basic model, a portrait low-rank matrix, and a style low-rank matrix; training the image synthesis model using style training samples to obtain the image synthesis model, including: obtaining portrait training samples; training to obtain the portrait low-rank matrix using the portrait training samples and a second basic model; and, training to obtain the style low-rank matrix using the style training samples and a third basic model; wherein, the second basic model is obtained by optimizing the first basic model using a portrait dataset, and the third basic model is obtained by optimizing the first basic model using a style dataset; constructing the image synthesis model based on the first basic model, the portrait low-rank matrix, and the style low-rank matrix.

[0069] Fine-tune the first basic model using the portrait fine-tuning dataset and the style fine-tuning dataset to obtain the second basic model and the third basic model.

[0070] When fine-tuning the first basic model, in order to meet different subsequent training requirements, different datasets are specifically selected for fine-tuning. Specifically, in order to meet the subsequent image synthesis requirements, many portrait images are collected to jointly form the portrait fine-tuning dataset. Then, the first basic model is fine-tuned using the portrait fine-tuning dataset to obtain the second basic model focused on image synthesis.

[0071] At the same time, in order to meet the subsequent synthetic image style requirements, many different style images are collected to jointly form the style fine-tuning dataset. Then, the first basic model is fine-tuned using the style fine-tuning dataset to obtain the third basic model focused on synthesizing images of a specified style.

[0072] The second basic model and the third basic model obtained in the above manner are more targeted. That is, in order to facilitate better subsequent training to obtain the portrait low-rank matrix (portrait LoRA) and the style low-rank matrix (style LoRA), dedicated basic models are selected for training respectively. The low-rank matrices obtained by training have stronger image synthesis ability when combined with the first basic model.

[0073] When training the portrait low-rank matrix and the style low-rank matrix, corresponding training samples need to be prepared. That is, portrait training samples and style training samples are prepared respectively. Specifically, the portrait low-rank matrix is trained using the portrait training samples and the second basic model obtained by targeted fine-tuning. Since the second basic model is obtained by fine-tuning the first basic model using the portrait fine-tuning dataset, the portrait low-rank matrix obtained by training has better image synthesis ability than the portrait low-rank matrix obtained by directly training using the first basic model (that is, the synthesized portrait is more similar to the user).

[0074] The style low-rank matrix is trained using the third base model obtained by leveraging style training samples and targeted fine-tuning. Since the third base model is obtained by fine-tuning the first base model with a style fine-tuning dataset, the trained style low-rank matrix has better style image synthesis ability than the style low-rank matrix directly trained using the first base model (that is, the consistency of different user styles in the synthesized image is better).

[0075] In addition, during training, the portrait low-rank matrix and the style low-rank matrix are trained separately and each uses a different base model. This ensures that they do not interfere with each other during the training process, so that the two trained low-rank matrices have better pertinence and more balanced synthesis ability when synthesizing images.

[0076] Here, since both the portrait low-rank matrix and the style low-rank matrix are trained using fine-tuning models of the first base model, the portrait low-rank matrix and the style low-rank matrix have the same matrix dimensions and are both compatible with the first base model. In addition, since the first base model has not undergone any fine-tuning, when the first base model performs an image synthesis task, the overall synthesis ability is relatively balanced. The final image synthesis effect mainly depends on the two low-rank matrices. In other words, in the image synthesis model, the portrait low-rank matrix and the style low-rank matrix fully balance the synthesis effects of the portrait and the style during the image synthesis process, that is, making the face in the synthesized image more similar to the user and having better consistency in style among different users.

[0077] For example, as Figure 3 is a schematic diagram of the model training process illustrated in this disclosure. As can be seen from Figure 3 , first, the first base model is respectively fine-tuned using a portrait fine-tuning dataset and a style fine-tuning dataset to obtain two base models, namely the second base model and the third base model. On this basis, the portrait training samples and the style training samples can be further used for training to obtain the portrait low-rank matrix and the style low-rank matrix. Then, the portrait low-rank matrix and the style low-rank matrix can be inserted into the first base model. During the training process, the portrait low-rank matrix and the style low-rank matrix are trained separately to avoid mutual influence. When fusing, the portrait low-rank matrix and the style low-rank matrix are inserted into the first base model to ensure good compatibility and can be directly used without further debugging. Through the above solution, not only the training effect of the image synthesis model is improved, but also the training efficiency can be effectively improved.

[0078] It should be noted that the portrait low-rank matrix can be called portrait LoRA, and the style low-rank matrix can be called style LoRA.

[0079] The portrait training samples mentioned here are samples that need to be annotated. The annotation may involve key point localization of the face, expression classification, pose estimation, etc. These information are crucial for training a LoRA that can capture and enhance the key features in the portrait generation process. In addition, since LoRA aims to improve the model's generation ability, these data may also include high-quality generation targets, such as high-resolution portrait images, and corresponding low-resolution or stylized input images.

[0080] The style training samples mentioned here are samples that need to be annotated. The annotation may involve style classification, style feature extraction, style transfer targets, etc. These information are crucial for training a LoRA that can capture and transform style features while keeping the content information unchanged. In addition, since LoRA aims to improve the model's style transfer ability, these data may also include image pairs before and after style transfer, and corresponding style descriptions or labels.

[0081] During training, select a suitable base model, such as diffusion as the first base model, and then obtain the second base model through fine-tuning as the starting point for training the portrait LoRA. Input the preprocessed portrait training samples into the first base model for training. During the training process, the model optimizes its parameters through the backpropagation algorithm to minimize the loss function, thereby learning the general features of the portrait. After training is completed, the model will output a portrait low-rank matrix, that is, the portrait LoRA. This matrix is a compact representation of the key information about the portrait learned by the model during the training process. The portrait LoRA contains the encoding of the portrait features by the model and can be used for subsequent portrait generation, recognition, or editing tasks.

[0082] Through the above training method, on the basis of the second base model, use specific training strategies (such as contrastive learning, generative adversarial network, etc.) and annotated portrait data to train the portrait LoRA. The goal of the portrait LoRA is to capture and enhance the key features in the portrait generation process, so that highly similar and high-quality portraits can be generated during inference. It should be noted that the portrait low-rank matrix (that is, the portrait LoRA) obtained here should be able to be well compatible with the first base model, that is, have a matching parameter format, matrix dimension, and data type with the first base model.

[0083] The third base model is obtained by fine-tuning the same first base model and serves as the starting point for Style LoRA training. The preprocessed style training samples are input into the second base model for training. During the training process, the model optimizes its parameters through the backpropagation algorithm to capture style features and minimize the loss with style labels or descriptions. After training is completed, the model outputs a style low-rank matrix, namely Style LoRA. This matrix is a compact representation of the key information about the style learned by the model during the training process. Style LoRA contains the encoding of the style features by the model and can be used for subsequent tasks such as style transfer, artistic creation, or image editing. It should be noted that the portrait low-rank matrix (i.e., Portrait LoRA) obtained through training here should be well compatible with the first base model, that is, it has a matching parameter format, matrix dimension, and data type with the first base model.

[0084] Through the above training method, on the basis of the third base model, training Style LoRA using style transfer-related training methods and labeled style data. The goal of Style LoRA is to capture and transform the style features in the image while keeping the content information unchanged to achieve diverse style transfer effects.

[0085] In the synthesis model, the weights of Portrait LoRA and Style LoRA are combined with the weights of the first base model. This can be achieved through simple weight addition, weight interpolation, or more complex fusion strategies. During the subsequent application inference process, the weights of Portrait LoRA and Style LoRA can be adaptively adjusted according to the characteristics of the input image to achieve better generation effects. This can be achieved by introducing additional adaptive layers or attention mechanisms.

[0086] In the intermediate layer of the first base model, the features extracted by Portrait LoRA and Style LoRA can be fused. This can be achieved through methods such as feature concatenation, feature addition, or feature transformation to generate a synthetic image that contains both portrait and style features. After constructing the image synthesis model, a synthetic dataset containing portrait and style labels can be used for training and optimization. During the training process, appropriate loss functions and training strategies can be adopted to ensure that the model can learn the key features of the synthetic image and generate high-quality synthetic results.

[0087] In one or more embodiments of the present disclosure, in response to a user's grouping request, groupings are established using image style elements of different class names, and group prompt information is assigned to the groupings.

[0088] When training the model, it is also possible to group the image style elements. For example, group the suit A1, tie B2, and company logo C3 into the same group and name the group prompt information as Company A; group the suit A1, tie B3, and company logo C5 into the same group and name the group prompt information as Company B. In this way, it helps the model establish the matching relationships between different image style elements. Generally, there is only one image style element of the same class name in a group. If there are multiple ones, it is easy to cause synthesis errors. For example, when synthesizing a work photo, if the user specifies two ties, it will lead to synthesis errors in the synthesized image.

[0089] It should be noted that when establishing the grouping correspondence relationship, avoid adding the prompt information Prompt that restricts the matching relationship of the image style elements, so that the trained image synthesis model can freely combine the image style elements according to the specific instructions of the user. While meeting the requirement of quickly synthesizing through the group prompt information, it should also be able to meet the diverse matching needs of the user.

[0090] Based on the same idea, the present disclosure also provides an image synthesis method. As Figure 4 is a schematic flowchart of an image synthesis method provided by the present disclosure. As can be seen from Figure 4 it, the method specifically includes the following steps: Step 401: Obtain the image style prompt information and the user's facial image. Step 402: Input the user's facial image and the image style indication information into the image synthesis model to generate a synthesized image with a specified style; wherein, the image synthesis model is obtained by training with a style training sample constructed by image style elements bound with element prompt information; the image style elements are basic elements obtained by segmenting the original image and used to represent the image style.

[0091] When the user has an image synthesis requirement (for example, the user wants to generate a work photo that meets the standard requirements), the user can provide the server with the user's facial image taken by himself / herself. The user can also provide the required image style indication information in the form of a prompt word (Prompt). Further, the image synthesis model can perform the image synthesis task by using the user's facial image and the image style indication information.

[0092] The image style indication information mentioned here can be understood as the style of the image that the user finally wants to synthesize, such as various image styles like ID photos, portraits, Chinese styles, etc. More specifically, the ID photo can be further divided into work photos of Company A, work photos of Company B, and so on. It should be noted that the more detailed the image style indication information provided by the user, the more in line with the user's requirements the final synthesized image style will be.

[0093] In practical applications, the image style prompt information provided by the user can also include specific element prompt information. For example, the user provides a prompt to produce a work photo of Company A, but replace the company logo in the work photo with brooch C1. When the image synthesis model performs image synthesis, it will replace brooch A1 in the work photo with brooch C1.

[0094] In the inference stage of image synthesis, although the second base model and the third base model are used to train the portrait low-rank matrix (i.e., portrait LoRA) and the style low-rank matrix (i.e., style LoRA) respectively, when actually generating portraits and performing style conversion, it returns to the original first base model and combines these two LoRAs. The key to this approach is that by training LoRAs on different fine-tuning models respectively, it is possible to avoid task conflicts that may arise from training on the same base model (such as training portrait LoRA and style LoRA simultaneously based on the first base model), and at the same time, it can use the weight parameter adjustment of LoRA to fuse portrait and style features, and finally generate portrait images that are both highly similar and conform to a specific style.

[0095] Based on the above solution, by precisely controlling semantic information such as image style prompt information and element prompt information, the image synthesis model can generate images that more accurately meet the user's intention and reduce misunderstandings or deviations. Moreover, since the image synthesis model conducts in-depth learning on each detail during training, when synthesizing images, it can more precisely represent the image style elements of each part of the image, making the image more in line with the user's expectations.

[0096] In one or more embodiments of the present disclosure, before inputting the user's facial image and image style indication information into the image synthesis model, it further includes: if the image style prompt information contains element prompt information for the second image style element, then use the detection model to determine the position of the smeared image element included in the first image style element; so that when using the image synthesis model to synthesize an image, stack the second image style element at the position of the smeared image element.

[0097] The second image style element mentioned here is a relatively smaller element compared to the first image style element. When synthesizing an image, in order to ensure that the second image style element can be accurately stacked at the specified position on the first image style element, the detection model can be used to detect the position of the smeared image element in the first image style element.

[0098] Specifically, when performing image synthesis, the user will provide image style hint information. Some of this hint information may be relatively vague, such as synthesizing a work photo of Company A. Some can be more refined element hint information. For example, synthesizing a work photo of Company A and replacing the company logo with logo C1. In this case, during synthesis, the detection model will detect the smeared image elements on the suit, and then cover the position of the smeared image elements with logo C1.

[0099] Through the above solution, significant smeared image elements are marked in the first image style element, and the detection model can easily detect the position of the smeared image elements. Furthermore, according to the detected position, the second image style element can be accurately stacked into the first image style element. The use of smeared image elements can guide the precise splicing of image elements.

[0100] Based on any of the above embodiments, the present disclosure also provides an image synthesis model training device. Figure 5 It is a structural schematic block diagram of an image synthesis model training device according to an embodiment of the present disclosure. As Figure 5 shown, the image synthesis model training device includes: an acquisition module 51 for acquiring image style elements for image synthesis from the original image; a binding module 52 for binding element hint information to each image style element; a sample construction module 53 for constructing a style training sample using the image style elements bound with element hint information; and a training module 54 for performing model training using the style training sample to obtain an image synthesis model, so as to synthesize a synthesized image of a specified style using the image synthesis model.

[0101] The acquisition module 51 is used to segment at least one original image to obtain a plurality of image elements; and save the image elements except for the face images as image style elements.

[0102] Optionally, it further includes a smearing module 55 for determining the positional relationship between the second image style element and the first image style element if the image style element includes a first image style element and a second image style element with an up-and-down stacking positional relationship; wherein, the second image style element is stacked on the first image style element, or the second image style element is embedded in the first image style element, and the second image style element is smaller than the first image style element; and smearing the position of the second image style element in the first image style element according to the size of the second image style element to obtain smeared image elements.

[0103] The binding module 52 is used to classify the image style elements and determine the class name; assign numbers to multiple image style elements with the same class name; generate hint information using the class name and the numbers, and bind the hint information to the corresponding image style elements.

[0104] The image synthesis model is constructed from a first basic model, a low-rank portrait matrix, and a low-rank style matrix. A training module 54 is configured to obtain portrait training samples; train a low-rank portrait matrix using the portrait training samples and a second basic model; and, train a low-rank style matrix using style training samples and a third basic model; wherein, the second basic model is obtained by optimizing the first basic model using a portrait data set, and the third basic model is obtained by optimizing the first basic model using a style data set; the image synthesis model is constructed based on the first basic model, the low-rank portrait matrix, and the low-rank style matrix.

[0105] A binding module 52 is configured to, in response to a grouping request from a user, establish a group using image style elements of different class names and assign group prompt information to the group.

[0106] Based on any of the above embodiments, the present disclosure also provides an image synthesis device. Figure 6 It is a structural schematic block diagram of an image synthesis device according to an embodiment of the present disclosure. As Figure 6 shown, the image synthesis device includes: an acquisition module 61 configured to acquire image style prompt information and a user's facial image. A generation module 62 is configured to input the user's facial image and the image style indication information into an image synthesis model to generate a synthesized image with a specified style. Wherein, the image synthesis model is obtained after being trained using a style training sample constructed with image style elements bound with element prompt information; the image style elements are elements obtained by segmenting an original image and used to represent the image style.

[0107] Optionally, it further includes a detection module 63 configured to, if the image style prompt information contains element prompt information for a second image style element, determine the position of the smeared image element included in the first image style element using a detection model; so as to stack the second image style element at the position of the smeared image element when synthesizing an image using the image synthesis model.

[0108] The implementation processes of the functions and roles of each module in the above device are specifically detailed in the implementation processes of the corresponding steps in the above method, and will not be elaborated here.

[0109] The execution subject of the image synthesis method and the image synthesis model training method in the specific embodiments of the present disclosure may be an electronic device such as a server (including a local server or a cloud server).

[0110] Therefore, based on any of the above embodiments, the present disclosure also provides an electronic device, and this electronic device can execute the image synthesis method and the image synthesis model training method of any of the above embodiments described in the present disclosure.

[0111] The user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in this disclosure are all information and data authorized by the user or fully authorized by all parties. Moreover, the collection, use, and processing of relevant data need to comply with the relevant laws, regulations, and standards of relevant countries and regions, and corresponding operation entrances are provided for users to choose to authorize or reject.

[0112] Figure 7 It is a schematic block diagram of the structure of an electronic device according to an embodiment of the present disclosure.

[0113] The hardware structure of the electronic device 1000 can be implemented using a bus architecture. The bus architecture can include any number of interconnected buses and bridges, depending on the specific application of the hardware and overall design constraints. The bus 1100 connects various circuits including one or more processors 1200, a memory 1300, and / or hardware modules together. The bus 1100 can also connect various other circuits 1400 such as peripheral devices, voltage regulators, power management circuits, external antennas, etc.

[0114] The bus 1100 can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, an Extended Industry Standard Component (EISA) bus, or the like. The bus can be divided into an address bus, a data bus, a control bus, etc. For the sake of representation, only one connecting line is used in this figure, but it does not mean that there is only one bus or one type of bus.

[0115] The present disclosure also provides a readable storage medium in which a computer program is stored. When the computer program is executed by a processor, it is used to implement the above method. The "readable storage medium" can be any device that can contain, store, communicate, propagate, or transmit a program for use by an instruction execution system, apparatus, or device or in connection with these instruction execution systems, apparatuses, or devices. More specific examples of the readable storage medium include the following: an electrical connection part (electronic device) having one or more wirings, a portable computer disk cartridge (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable read-only memory (CDROM), etc.

[0116] The present disclosure also provides a computer program product. The method of the present disclosure can be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer programs or instructions. When the computer program or instructions are loaded and executed, the processes or functions of the present disclosure are executed in whole or in part.

[0117] The computer program or instructions can be stored in a readable storage medium, or transmitted from one readable storage medium to another. For example, the computer program or instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center in a wired or wireless manner. The readable storage medium can be any available medium that can be accessed, or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium, such as a floppy disk, hard disk, or magnetic tape; it can also be an optical medium, such as a digital video disc; or it can be a semiconductor medium, such as a solid state drive. The computer-readable storage medium can be a volatile or non-volatile storage medium, or can include both volatile and non-volatile types of storage media.

[0118] Those skilled in the art should understand that the embodiments of the present disclosure can be provided as a method, system, or computer program product. Therefore, the present disclosure can take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present disclosure can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0119] The present disclosure is described with reference to the flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to the present disclosure. It should be understood that each process and / or block in the flowchart and / or block diagram, and the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing method devices to generate a machine, such that the instructions executed by the processor of the computer or other programmable data processing method devices generate means for implementing the functions specified in one Figure 1 one process or multiple processes and / or blocks Figure 1 one block or multiple blocks.

[0120] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing method device to work in a specific manner, such that the instructions stored in the computer-readable memory produce a manufacture including an instruction device that implements the functions specified in one or more processes and / or blocks Figure 1 of the one or more processes and / or blocks Figure 1 specified in the one or more blocks or blocks.

[0121] These computer program instructions can also be loaded onto a computer or other programmable data processing method device, such that a series of operation steps are performed on the computer or other programmable device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in one or more processes and / or blocks Figure 1 of the one or more processes and / or blocks Figure 1 specified in the one or more blocks or blocks.

[0122] In the description of this specification, the descriptions with reference to the terms "one embodiment / way", "some embodiments / ways", "example", "specific example", or "some examples", etc. mean that the specific features, structures, or characteristics described in connection with the embodiment / way or example are included in at least one embodiment / way or example of the present disclosure. In this specification, the schematic expressions of the above terms do not necessarily refer to the same embodiment / way or example. Moreover, the specific features, structures, or characteristics described can be combined in a suitable manner in any one or more embodiments / ways or examples. In addition, without contradiction, those skilled in the art can combine and combine the different embodiments / ways or examples described in this specification and the features of different embodiments / ways or examples.

[0123] In addition, the terms "first" and "second" are only used for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly specifying the quantity of the indicated technical features. Thus, the features defined with "first" and "second" can explicitly or implicitly include at least one of the features. In the description of the present disclosure, "a plurality of" means at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0124] Those skilled in the art should understand that the above embodiments are only for clearly explaining the present disclosure and are not intended to limit the scope of the present disclosure. For those skilled in the art, other changes or variations can be made on the basis of the above disclosure, and these changes or variations are still within the scope of the present disclosure.

Claims

1. An image synthesis method, characterized in that, The method includes: Obtaining image style hint information and a user's facial image; Inputting the user's facial image and the image style indication information into an image synthesis model to generate a synthesized image with a specified style; Wherein, the image synthesis model is obtained after being trained with style training samples constructed by using image style elements bound with element hint information; the image style elements are elements obtained by segmenting an original image and used to represent the image style.

2. The method according to claim 1, characterized in that Before inputting the user's facial image and the image style indication information into the image synthesis model, it further includes: If the image style hint information contains element hint information for a second image style element, using a detection model to determine the position of the smeared image element included in the first image style element; so that when using the image synthesis model to synthesize an image, the second image style element is stacked at the position of the smeared image element.

3. A method for training an image synthesis model, characterized in that, The method includes: Obtaining image style elements for image synthesis from an original image; Binding element hint information to each of the image style elements; Constructing style training samples by using the image style elements bound with the element hint information; Training a model by using the style training samples to obtain an image synthesis model, so as to use the image synthesis model to synthesize a synthesized image with a specified style.

4. The method according to claim 3, characterized in that, The obtaining image style elements for image synthesis from an original image includes: Segmenting at least one original image to obtain a plurality of image elements; Saving the image elements other than the face image as the image style elements.

5. The method according to claim 4, characterized in that, After segmenting at least one original image to obtain a plurality of image elements, it further includes: If the image style elements include a first image style element and a second image style element whose positional relationship is an up-and-down stacking relationship, determining the positional relationship between the second image style element and the first image style element; wherein, the second image style element is stacked on the first image style element, or the second image style element is embedded in the first image style element and the second image style element is smaller than the first image style element; Smearing at the position of the second image style element in the first image style element according to the size of the second image style element to obtain a smeared image element.

6. The method according to claim 3, wherein The binding element hint information to each of the image style elements includes: Classifying the image style elements and determining the class name; Assigning numbers to a plurality of the image style elements having the same class name; Generating the hint information by using the class name and the numbers, and binding the hint information to the corresponding image style elements.

7. The method according to claim 3, characterized in that, The image synthesis model is constructed by a first basic model, a portrait low-rank matrix, and a style low-rank matrix; The training the image synthesis model by using the style training samples to obtain an image synthesis model includes: Obtaining portrait training samples; Training a portrait low-rank matrix using the portrait training samples and the second basic model; and training a style low-rank matrix using the style training samples and the third basic model; wherein the second basic model is obtained by optimizing the first basic model using a portrait dataset, and the third basic model is obtained by optimizing the first basic model using a style dataset; Constructing an image synthesis model based on the first basic model, the portrait low-rank matrix, and the style low-rank matrix.

8. The method according to claim 6, characterized in that Further comprising: In response to a user's grouping request, establishing groups using image style elements of different class names and assigning group prompt information to the groups.

9. An electronic device, characterized in that, Comprising: A memory that stores execution instructions; And A processor that executes the execution instructions stored in the memory, causing the processor to execute the method according to any one of claims 1 to 8.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the method according to any one of claims 1 to 8.

Images