Training method of image generation model, image generation method and device thereof

Abstract

The invention provides a training method of an image generation model, an image generation method and a device thereof, and relates to the field of image processing, in particular to the field of artificial intelligence. According to the specific implementation scheme, a style reference image set is obtained, wherein the style reference image set comprises style reference images; obtaining a first prompt word corresponding to the style reference image; based on the style reference image set and the first cue word, performing low-rank adaptive LoRA fine tuning training on a pre-trained base large model to obtain a target LoRA adapter; and determining a target LoRA adapter and a pre-trained image processing adapter IP-Adapter module as an auxiliary model structure, and coupling the auxiliary model structure with the base large model to obtain a target image generation model.

Description

Technical Field

[0001] The present disclosure relates to the field of image processing, in particular to the field of artificial intelligence, and specifically to a training method for an image generation model, an image generation method and a device thereof. Background Art

[0002] With the development of image processing technology, pre-trained image generation models have achieved remarkable results in image generation tasks. These models are pre-trained with large-scale datasets and can generate high-quality images. However, in practical applications, users often want to generate images with specific styles, such as specific photo styles, photography styles, etc.

[0003] In order to meet this demand, the pre-trained base model needs to be further optimized and customized. The traditional solution is to retrain the entire model, but this is not only time-consuming and labor-intensive, but may also cause the model's performance to deteriorate on other tasks. Summary of the invention

[0004] The present disclosure provides a training method for an image generation model, an image generation method and a device thereof.

[0005] According to one aspect of the present disclosure, a training method for an image generation model is provided, by obtaining a style reference image set, the style reference image set including a style reference image; obtaining a first prompt word corresponding to the style reference image; based on the style reference image set and the first prompt word, performing low-rank adaptive LoRA fine-tuning training on a pre-trained base large model to obtain a target LoRA adapter; determining the target LoRA adapter and the pre-trained image processing adapter IP-Adapter module as an auxiliary model structure, coupling them with the base large model, to obtain a target image generation model.

[0006] This application uses LoRA technology to greatly reduce the time and computing resource requirements for fine-tuning training of the base large model; the coupled target image generation model combines the generation capability of the base large model, the style adjustment capability of the target LoRA adapter and the feature interaction capability of the IP-Adapter module, and the generated image has the expected style effect and higher quality.

[0007] According to another aspect of the present disclosure, an image generation method is provided, including: obtaining an original image uploaded by a user to be subjected to stylized image generation; obtaining a second prompt word input by the user; inputting the original image and the second prompt word into a target image generation model to obtain a stylized image output by the target image generation model; wherein the target image generation model is generated by training using the image generation model training method introduced above.

[0008] According to another aspect of the present disclosure, a training device for an image generation model is provided, including a first acquisition module for acquiring a style reference image set, the style reference image set including a style reference image; a second acquisition module for acquiring a first prompt word corresponding to the style reference image; a fine-tuning training module for performing low-rank adaptive LoRA fine-tuning training on a pre-trained base large model based on the style reference image set and the first prompt word to obtain a target LoRA adapter; and a model generation module for determining the target LoRA adapter and the pre-trained image processing adapter IP-Adapter module as an auxiliary model structure, coupling them with the base large model to obtain a target image generation model.

[0009] According to another aspect of the present disclosure, an image generation device is provided, including: a first acquisition module, used to acquire an original image uploaded by a user to be subjected to stylized image generation; a second acquisition module, used to acquire a second prompt word input by the user; an image generation module, used to input the original image and the second prompt word into a target image generation model, and acquire a stylized image output by the target image generation model; wherein the target image generation model is trained and generated using the training device of the image generation model introduced above.

[0010] According to another aspect of the present disclosure, there is provided an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the above-mentioned image generation model training method or image generation method.

[0011] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute the above-mentioned image generation model training method or image generation method.

[0012] According to another aspect of the present disclosure, a computer program product is provided, including a computer program, which implements the above-mentioned image generation model training method or image generation method when executed by a processor.

[0013] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0015] Figure 1It is a schematic diagram of an exemplary implementation of a training method for an image generation model according to an exemplary embodiment of the present disclosure.

[0016] Figure 2 It is a schematic diagram of an exemplary implementation of a training method for an image generation model according to an exemplary embodiment of the present disclosure.

[0017] Figure 3 It is a schematic diagram of an exemplary implementation of a training method for an image generation model according to an exemplary embodiment of the present disclosure.

[0018] Figure 4 is a schematic diagram of an exemplary implementation of an image generating method according to an exemplary embodiment of the present disclosure.

[0019] Figure 5 is a schematic diagram of an exemplary implementation of an image generating method according to an exemplary embodiment of the present disclosure.

[0020] Figure 6 A training device for an image generation model according to an exemplary embodiment of the present disclosure.

[0021] Figure 7 is a schematic diagram of an image generating device according to an exemplary embodiment of the present disclosure.

[0022] Figure 8 is a schematic diagram of an electronic device according to an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION

[0023] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0024] Image processing is a technique that uses computers to analyze images to achieve the desired results. It is also called image processing. Image processing generally refers to digital image processing. A digital image is a large two-dimensional array obtained by shooting with industrial cameras, video cameras, scanners and other equipment. The elements of the array are called pixels, and their values ​​are called grayscale values. Image processing technology generally includes three parts: image compression, enhancement and restoration, matching, description and recognition.

[0025] Deep Learning (DL) is a new research direction in the field of Machine Learning (ML). It is introduced into machine learning to make it closer to its original goal - artificial intelligence. Deep learning is the internal laws and representation levels of learning sample data. The information obtained in the learning process is of great help in the interpretation of data such as text, images and sounds. Its ultimate goal is to enable machines to have analytical learning capabilities like humans and to recognize data such as text, images and sounds. Deep learning is a complex machine learning algorithm that has achieved results in speech and image recognition that far exceed previous related technologies.

[0026] Artificial Intelligence (AI) is a discipline that studies how computers can simulate certain thought processes and intelligent behaviors of human beings (such as learning, reasoning, thinking, planning, etc.). It includes both hardware-level technologies and software-level technologies. AI hardware technologies generally include computer vision technology, speech recognition technology, natural language processing technology, as well as learning / deep learning, big data processing technology, knowledge graph technology, and other aspects.

[0027] In the technical solution disclosed herein, the acquisition, storage and application of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0028] Figure 1 is a schematic diagram of an exemplary implementation of a training method for an image generation model shown in the present application, such as Figure 1 As shown, the training method of the image generation model includes the following steps:

[0029] S101: Acquire a style reference image set, where the style reference image set includes style reference images.

[0030] In some embodiments, the style reference image set may be generated based on the style reference images uploaded online by the user after data augmentation.

[0031] In some embodiments, the style reference image set may be a set of images in a certain vertical field (such as portraits, landscapes, and vehicles). For example, if the style reference image set is a set of children's images, each child image in the style reference image set may be regarded as a style reference image.

[0032] S102: Obtain a first prompt word corresponding to the style reference image.

[0033] The first prompt word is used to describe the image style, image content or image color of the style reference image. For example, the first prompt word may be "retro style", "modern simplicity", "dark tone", "round face" and the like.

[0034] Optionally, the first prompt word may be implemented based on manual annotation, or may be generated through automatic annotation (eg, an image annotation model).

[0035] S103, based on the style reference image set and the first prompt word, perform low-rank adaptive LoRA fine-tuning training on the pre-trained base large model to obtain a target LoRA adapter.

[0036] Among them, the pre-trained base large model refers to the image generation model that has been previously trained.

[0037] Insert low-rank matrices into the key layers of the pre-trained large base model, and based on the style reference image set and the first prompt word obtained above, only perform low-rank adaptation (LoRA) fine-tuning on these low-rank matrices. During the LoRA fine-tuning process, keep the parameters of the pre-trained large base model unchanged, and only update the parameters of the low-rank matrix. After the fine-tuning training is completed, the target LoRA adapter obtained contains information about a specific style.

[0038] S104, determining the target LoRA adapter and the pre-trained image processing adapter IP-Adapter module as the auxiliary model structure, coupling them with the base large model, and obtaining the target image generation model.

[0039] The generated target LoRA adapter and the pre-trained Image Processing Adapter (IP-Adapter) module are coupled with the base large model to form a complete target image generation model. Among them, the base large model is responsible for image generation, the target LoRA adapter is responsible for style adjustment, and the IP-Adapter module is responsible for feature interaction and enhancement.

[0040] Among them, the IP-Adapter module can be generated in advance in combination with the base large model training, and can be directly coupled to the base large model here.

[0041] In some embodiments, if the style reference image set is a child style image, then after the above-mentioned LoRA fine-tuning training, a target LoRA adapter corresponding to the child style can be obtained. If the target LoRA adapter corresponding to the child style and the pre-trained IP-Adapter module are coupled with the base large model, a target image generation model for generating child style images can be obtained.

[0042] In some embodiments, if the style reference image set is anime-style images, then after the above-mentioned LoRA fine-tuning training, a target LoRA adapter corresponding to the anime style can be obtained. If the target LoRA adapter corresponding to the anime style and the pre-trained IP-Adapter module are coupled with the base large model, a target image generation model for generating anime-style images can be obtained.

[0043] The embodiment of the present application proposes a training method for an image generation model, by obtaining a style reference image set, the style reference image set includes a style reference image; obtaining a first prompt word corresponding to the style reference image; based on the style reference image set and the first prompt word, performing low-rank adaptive LoRA fine-tuning training on a pre-trained base large model to obtain a target LoRA adapter; determining the target LoRA adapter and the pre-trained image processing adapter IP-Adapter module as auxiliary model structures, coupling with the base large model, and obtaining a target image generation model. The present application greatly reduces the time and computing resource requirements for fine-tuning training of the base large model through LoRA technology; the coupled target image generation model combines the generation capability of the base large model, the style adjustment capability of the target LoRA adapter, and the feature interaction capability of the IP-Adapter module, and the generated image has the expected style effect and higher quality.

[0044] Furthermore, the IP-Adapter module is used to extract image features and text features after obtaining the original image and prompt words input by the user, and perform cross-attention interaction on the image features and text features to obtain a cross-feature representation, wherein the cross-feature representation is used to input the base large model.

[0045] Figure 2 is a schematic diagram of an exemplary implementation of a training method for an image generation model shown in the present application, such as Figure 2 As shown, the training method of the image generation model includes the following steps:

[0046] S201, dividing the first sample image set according to the person vertical category, and obtaining a second sample image set corresponding to each person vertical category.

[0047] A first sample image set is obtained, where the first sample image set may include first sample images with different character styles.

[0048] Perform person recognition and segmentation on each first sample image in the first sample image set to obtain the person in the first sample image; identify the person vertical category to which the person in the first sample image belongs; divide the first sample image set according to the identified person vertical categories to obtain a second sample image set corresponding to each person vertical category.

[0049] Among them, the character vertical category refers to the vertical field of characters, for example, it may include the children's vertical field, the adult male vertical field, the adult female vertical field, the elderly vertical field, the athlete vertical field, etc.

[0050] In the present application, the first sample image set is divided according to the vertical category of characters, mainly to make the subsequently generated target image generation model have the characteristics of a specific vertical field of characters, such as avoiding the adultization of children's images, so that the stylized images generated by the target image generation model are more in line with reality.

[0051] S202, for each person vertical category, perform body feature recognition on each second sample image in the second sample image set corresponding to the person vertical category, where the body features include facial features and limb features.

[0052] Among them, facial features usually include facial proportions, deciduous teeth features, facial curve features, iris color features, eye features, eyebrow features, nose features, mouth features, ear features, face shape features, skin features, beard features, hair features, skin color features, etc.

[0053] Among them, limb features usually include limb posture, body proportions, skin features, spine features, height features, neck features, shoulder features, arm features, hand features, torso features, leg features, foot features, etc.

[0054] S203, obtaining reference body features corresponding to the character vertical category.

[0055] Define a reference body feature for each character vertical as a standard for comparison.

[0056] For example, for the child vertical category, the reference body features may be child facial features (such as lighter eyebrows, rounder face, etc.) and child limb features (such as body proportions that are more in line with children, spinal condition, etc.).

[0057] For example, for the male athlete vertical, the reference body features may be features with athletic facial features (e.g., a strong jawline, clear brows and eyes) and common athletic body shapes (e.g., a more muscular upper body, a tighter abdomen).

[0058] S204, matching the identified body features with the reference body features, and selecting a third sample image set corresponding to the person vertical category from the second sample image set as a style reference image set.

[0059] By matching the identified body features with the reference body features, each style reference image in the final screened style reference image set can meet the desired character vertical category standard.

[0060] S205: Obtain a first prompt word corresponding to the style reference image.

[0061] As an achievable method, a description text of the style reference image is obtained; keywords are extracted from the description text, and a first prompt word corresponding to the style reference image is generated according to the extracted keywords.

[0062] As another feasible method, style features of the style reference image are extracted through a style transfer network; and the style features are converted into a first prompt word corresponding to the style reference image.

[0063] The first prompt word generated above is subsequently used in LoRA fine-tuning training, so that the target image generation model finally obtained can generate images that are more in line with the expected image style based on the text prompts.

[0064] Furthermore, after obtaining the first prompt word, the first prompt word may be deduplicated to avoid duplication of the first prompt word.

[0065] S206, based on the style reference image set and the first prompt word, perform low-rank adaptive LoRA fine-tuning training on the pre-trained base large model to obtain a target LoRA adapter.

[0066] S207, determine the target LoRA adapter and the pre-trained image processing adapter IP-Adapter module as the auxiliary model structure, couple them with the base large model, and obtain the target image generation model.

[0067] Regarding the specific implementation of steps S206 to S207, reference may be made to the specific introduction of the relevant parts in the above embodiments, which will not be elaborated here.

[0068] If the style reference image sets corresponding to the plurality of character vertical categories are obtained as described above, steps S206 to S207 are respectively performed based on the style reference image sets corresponding to each character vertical category to obtain the target image generation model corresponding to each character vertical category.

[0069] The embodiment of the present application obtains a set of style reference images that meet the desired character vertical style from a first sample image set through body feature comparison, so that the final target image generation model can generate images that are more in line with the desired character vertical style; through LoRA technology, the time and computing resource requirements for fine-tuning training of the base large model are greatly reduced; the coupled target image generation model combines the generation capability of the base large model, the style adjustment capability of the target LoRA adapter and the feature interaction capability of the IP-Adapter module, and the generated image has the expected character vertical style and is of higher quality.

[0070] Optionally, in the above embodiment, the second sample image set corresponding to each character vertical category obtained after dividing the first sample image set according to the character vertical categories may be directly used as the style reference image set corresponding to the character vertical category.

[0071] Figure 3 is a schematic diagram of an exemplary implementation of a training method for an image generation model shown in the present application, such as Figure 3 As shown, the training method of the image generation model includes the following steps:

[0072] S301, obtaining an image uploaded by a user to a reference image area as a style reference image.

[0073] S302: Perform image augmentation on the style reference image to obtain a set of augmented style reference images.

[0074] In addition to obtaining a style reference image set that conforms to the desired character vertical style from the first sample image set as mentioned above, the present application also proposes a method for obtaining a style reference image set, that is, when the user uses the image generation technology online, the image uploaded by the user to the reference image area (the reference image area is an area on the display interface specially designed for users to upload style reference images) is used as a style reference image, and then the style reference image is augmented to obtain a style reference image set after augmentation.

[0075] The style reference image may be processed by at least one of geometric transformation, noise addition, affine transformation, and perspective transformation to obtain an augmented image generated after the processing, and the style reference image and the augmented image may form a style reference image set.

[0076] S303: Obtain a first prompt word corresponding to the style reference image.

[0077] Optionally, only the first prompt word of the style reference image uploaded by the user to the reference image area may be obtained, without obtaining the first prompt word of the augmented image.

[0078] Optionally, the style reference images and augmented images included in the style reference image set may be regarded as style reference images for subsequent model fine-tuning training, and then the first prompt word corresponding to each style reference image included in the style reference image set is obtained.

[0079] Regarding the method of obtaining the first prompt word, reference may be made to the detailed introduction of the relevant part in the above embodiment, which will not be described again here.

[0080] S304, based on the style reference image set and the first prompt word, perform low-rank adaptive LoRA fine-tuning training on the pre-trained base large model to obtain a target LoRA adapter.

[0081] S305, determine the target LoRA adapter and the pre-trained image processing adapter IP-Adapter module as the auxiliary model structure, couple them with the base large model, and obtain the target image generation model.

[0082] Regarding the specific implementation of steps S303 to S305, reference may be made to the specific introduction of the relevant parts in the above embodiments, which will not be elaborated here.

[0083] The embodiment of the present application obtains the image uploaded by the user to the reference image area as the style reference image, and performs subsequent LoRA fine-tuning training based on the style reference image set generated after augmenting the style reference image data, so that the final target image generation model can generate stylized images according to user needs, thereby realizing the generation of a target image generation model with a customized style and improving user satisfaction; through LoRA technology, the time and computing resource requirements for fine-tuning training of the base large model are greatly reduced; the coupled target image generation model combines the generation capability of the base large model, the style adjustment capability of the target LoRA adapter and the feature interaction capability of the IP-Adapter module, and the generated image has the expected character vertical style and higher quality.

[0084] Figure 4 is a schematic diagram of an exemplary implementation of an image generation method shown in the present application, such as Figure 4 As shown, the image generation method comprises the following steps:

[0085] S401, obtaining an original image uploaded by a user to be subjected to stylized image generation.

[0086] S402: Acquire a second prompt word input by the user.

[0087] The second prompt word may be a positive prompt word or a negative prompt word.

[0088] S403: Input the original image and the second prompt word into a target image generation model to obtain a stylized image output by the target image generation model.

[0089] The target image generation model is generated by training using the image generation model training method introduced in the above embodiment.

[0090] In some embodiments, when the user generates a stylized image on the stylized image generation software, he uploads the original image to be stylized and uploads the style reference image in the reference image area, and enters the second prompt word. Then the user can click the image generation button, and then the stylized image generation software uses the introduction part of the third embodiment above to perform data augmentation on the style reference image to generate a style reference image set, and combines the style reference image set with the first prompt word corresponding to the style reference image to perform LoRA fine-tuning training on the pre-trained base large model, and the target LoRA adapter obtained based on the LoRA fine-tuning training and the pre-trained image processing adapter IP-Adapter module are used as auxiliary model structures, which are coupled with the base large model to obtain the target image generation model (that is, the target image generation model here is online training). After the target image generation model is generated by online training, the stylized image generation software inputs the original image to be stylized and the second prompt word into the target image generation model to obtain the stylized image output by the target image generation model.

[0091] The embodiment of the present application can stylize the original image to be generated into a stylized image based on the target image generation model generated above, and output a stylized image that meets the user's needs, thereby achieving rapid realization of user-defined style images and improving user satisfaction.

[0092] Figure 5 is a schematic diagram of an exemplary implementation of an image generation method shown in the present application, such as Figure 5 As shown, the image generation method comprises the following steps:

[0093] S501, determining a target image style corresponding to a target image generation model.

[0094] After the target image generation model is generated by training based on the image generation model training method introduced in the above embodiment, the target image style corresponding to the target image generation model may be determined.

[0095] S502: Generate a target image style selection box based on the target image style, and display the target image style selection box on a display interface.

[0096] For example, if the style reference image set used when training the target image generation model is a child style image, then the target image style corresponding to the target image generation model generated by the training is determined to be a child style, and a child style selection box is generated and displayed on the display interface.

[0097] S503, obtaining an original image uploaded by a user for stylized image generation, and obtaining a third prompt word input by the user.

[0098] The third prompt word may be a positive prompt word or a negative prompt word.

[0099] S504 , in response to detecting a user selecting a target image style selection box, inputting the original image and the third prompt word into a target image generation model, and acquiring a stylized image output by the target image generation model.

[0100] Exemplarily, if the user uploads the original image to be stylized image generated, enters the third prompt word, selects the child style selection box on the display interface, and submits the style image generation button, the original image to be stylized image generated and the third prompt word are input into the target image generation model generated by training the style reference image set based on the child style, and the child style stylized image output by the target image generation model is obtained.

[0101] In an embodiment of the present application, the target image style corresponding to the target image generation model generated by the above training is used as a candidate style style for the user to choose. Providing multiple candidate style styles can effectively increase the diversity of user choices and improve user experience.

[0102] Figure 6 is a training device for an image generation model shown in this application, such as Figure 6 As shown, the training device 600 of the image generation model includes a first acquisition module 601, a second acquisition module 602, a fine-tuning training module 603 and a model generation module 604, wherein:

[0103] The first acquisition module 601 is used to acquire a style reference image set, where the style reference image set includes style reference images.

[0104] The second acquisition module 602 is used to acquire a first prompt word corresponding to the style reference image.

[0105] The fine-tuning training module 603 is used to perform low-rank adaptive LoRA fine-tuning training on the pre-trained base large model based on the style reference image set and the first prompt word to obtain a target LoRA adapter.

[0106] The model generation module 604 is used to determine the target LoRA adapter and the pre-trained image processing adapter IP-Adapter module as an auxiliary model structure, and couple them with the base large model to obtain the target image generation model.

[0107] Through the LoRA technology, this device greatly reduces the time and computing resource requirements for fine-tuning the training of the base large model; the coupled target image generation model combines the generation capability of the base large model, the style adjustment capability of the target LoRA adapter and the feature interaction capability of the IP-Adapter module, and the generated image has the expected style effect and higher quality.

[0108] Furthermore, the IP-Adapter module in the model generation module 604 is used to extract image features and text features, and perform cross-attention interaction on the image features and text features to obtain a cross-feature representation, wherein the cross-feature representation is used to input the base large model.

[0109] Furthermore, the first acquisition module 601 is also used to: divide the first sample image set according to the character vertical category, and obtain the second sample image set corresponding to each character vertical category; for each character vertical category, perform body feature recognition on each second sample image in the second sample image set corresponding to the character vertical category, the body features include facial features and limb features; obtain reference body features corresponding to the character vertical category; match the identified body features with the reference body features, and filter out the third sample image set corresponding to the character vertical category from the second sample image set as the style reference image set.

[0110] Furthermore, the first acquisition module 601 is also used to: perform person recognition and segmentation on each first sample image in the first sample image set to obtain the person in the first sample image; identify the person vertical category to which the person in the first sample image belongs; divide the first sample image set according to the identified person vertical categories to obtain the second sample image set corresponding to each person vertical category.

[0111] Furthermore, the first acquisition module 601 is further used to: acquire an image uploaded by a user to the reference image area as a style reference image; and perform image augmentation on the style reference image to acquire a set of augmented style reference images.

[0112] Furthermore, the first acquisition module 601 is also used to: perform at least one of geometric transformation, noise addition, affine transformation, and perspective transformation on the style reference image, obtain the augmented image generated after the processing, and form a style reference image set from the style reference image and the augmented image.

[0113] Furthermore, the second acquisition module 602 is further configured to: acquire a description text of the style reference image; extract keywords from the description text, and generate a first prompt word corresponding to the style reference image according to the extracted keywords.

[0114] Furthermore, the second acquisition module 602 is further configured to: extract style features of the style reference image through a style transfer network; and convert the style features into a first prompt word corresponding to the style reference image.

[0115] Furthermore, the second acquisition module 602 is further configured to: perform a deduplication process on the first prompt word.

[0116] Figure 7 is a schematic diagram of an image generating device shown in the present application, such as Figure 7 As shown, the image generating device 700 includes a first acquiring module 701, a second acquiring module 702 and an image generating module 703, wherein:

[0117] The first acquisition module 701 is used to acquire an original image uploaded by a user to be used for stylized image generation.

[0118] The second acquisition module 702 is used to acquire a second prompt word input by the user.

[0119] The image generation module 703 is used to input the original image and the second prompt word into the target image generation model to obtain the stylized image output by the target image generation model.

[0120] The target image generation model is generated by training using a training device for the image generation model as in the above example.

[0121] The device can stylize the original image to be stylized based on the target image generation model generated above, output a stylized image that meets user needs, achieve rapid realization of user-defined style images, and improve user satisfaction.

[0122] Furthermore, the image generating device 700 also includes: a determination module, used to determine the target image style corresponding to the target image generating model; a display module, used to generate a target image style selection box based on the target image style, and display the target image style selection box on the display interface; a third acquisition module, used to obtain a third prompt word input by the user; the image generating module 703 is also used to input the original image and the third prompt word into the target image generating model in response to monitoring the user's selection operation on the target image style selection box, and obtain the stylized image output by the target image generating model.

[0123] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0124] Figure 8A schematic block diagram of an example electronic device 800 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0125] like Figure 8 As shown, the device 800 includes a computing unit 801, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 802 or a computer program loaded from a storage unit 808 into a random access memory (RAM) 803. In the RAM 803, various programs and data required for the operation of the device 800 can also be stored. The computing unit 801, the ROM 802, and the RAM 803 are connected to each other via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0126] A number of components in the device 800 are connected to the I / O interface 805, including: an input unit 806, such as a keyboard, a mouse, etc.; an output unit 807, such as various types of displays, speakers, etc.; a storage unit 808, such as a disk, an optical disk, etc.; and a communication unit 809, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 809 allows the device 800 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0127] The computing unit 801 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 801 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 801 performs the various methods and processes described above, such as the training method or image generation method of the image generation model. For example, in some embodiments, the training method or image generation method of the image generation model may be implemented as a computer software program, which is tangibly included in a machine-readable medium, such as a storage unit 808. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 800 via the ROM 802 and / or the communication unit 809. When the computer program is loaded into the RAM 803 and executed by the computing unit 801, one or more steps of the training method or image generation method of the image generation model described above may be executed. Alternatively, in other embodiments, the computing unit 801 may be configured to execute the image generation model training method or the image generation method in any other appropriate manner (eg, by means of firmware).

[0128] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0129] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0130] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0131] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0132] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0133] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0134] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0135] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A training method for an image generation model, wherein: The method comprises: Acquire a style reference image set, wherein the style reference image set includes style reference images; Obtaining a first prompt word corresponding to the style reference image; Based on the style reference image set and the first prompt word, performing low-rank adaptive LoRA fine-tuning training on the pre-trained base large model to obtain a target LoRA adapter; The target LoRA adapter and the pre-trained image processing adapter IP-Adapter module are determined as auxiliary model structures, which are coupled with the base large model to obtain a target image generation model.

2. The method according to claim 1, wherein: The method further comprises: The IP-Adapter module is used to extract image features and text features, and perform cross-attention interaction on the image features and the text features to obtain a cross-feature representation, wherein the cross-feature representation is used to input the base large model.

3. The method according to claim 2, wherein: The acquiring of a style reference image set includes: Divide the first sample image set according to the character vertical categories, and obtain a second sample image set corresponding to each character vertical category; For each of the character vertical categories, performing body feature recognition on each second sample image in the second sample image set corresponding to the character vertical category, the body features including facial features and limb features; Obtain reference body features corresponding to the character vertical category; The identified body features are matched with the reference body features, and a third sample image set corresponding to the person vertical category is selected from the second sample image set as a style reference image set.

4. The method according to claim 3, wherein: The step of dividing the first sample image set according to the character vertical categories and obtaining a second sample image set corresponding to each character vertical category includes: Performing person recognition segmentation on each first sample image in the first sample image set to obtain the person in the first sample image; Identifying the person in the first sample image into a person vertical category; The first sample image set is divided according to the identified character vertical categories, and a second sample image set corresponding to each character vertical category is obtained.

5. The method according to claim 2, wherein: The acquiring of a style reference image set includes: Obtain an image uploaded by the user to the reference image area as a style reference image; Perform image augmentation on the style reference image to obtain a set of augmented style reference images.

6. The method according to claim 5, wherein: The performing image augmentation on the style reference image to obtain a set of augmented style reference images includes: The style reference image is processed by at least one of geometric transformation, noise addition, affine transformation, and perspective transformation to obtain an augmented image generated after the processing, and the style reference image and the augmented image form a style reference image set.

7. The method according to claim 1, wherein: The obtaining of the first prompt word corresponding to the style reference image includes: Obtaining a description text of the style reference image; Keywords are extracted from the description text, and a first prompt word corresponding to the style reference image is generated according to the extracted keywords.

8. The method according to claim 1, wherein: The obtaining of the first prompt word corresponding to the style reference image includes: Extracting style features of the style reference image through a style transfer network; The style feature is converted into a first prompt word corresponding to the style reference image.

9. The method according to claim 4 or 6, wherein: After obtaining the first prompt word corresponding to the style reference image, the method further includes: The first prompt word is deduplicated.

10. A method for generating an image, wherein: The method comprises: Obtain the original image uploaded by the user for stylized image generation; Get the second prompt word input by the user; Inputting the original image and the second prompt word into a target image generation model to obtain a stylized image output by the target image generation model; The target image generation model is trained and generated using the method described in any one of claims 1 to 9.

11. The method according to claim 10, wherein: The method further comprises: Determining a target image style corresponding to the target image generation model; Generating a target image style selection box based on the target image style, and displaying the target image style selection box on a display interface; Obtain the third prompt word input by the user; In response to detecting a user's selection operation on the target image style selection box, the original image and the third prompt word are input into the target image generation model to obtain a stylized image output by the target image generation model.

12. A training device for an image generation model, wherein: The device comprises: A first acquisition module, configured to acquire a style reference image set, wherein the style reference image set includes style reference images; A second acquisition module, used to acquire a first prompt word corresponding to the style reference image; A fine-tuning training module, used to perform low-rank adaptive LoRA fine-tuning training on the pre-trained base large model based on the style reference image set and the first prompt word to obtain a target LoRA adapter; The model generation module is used to determine the target LoRA adapter and the pre-trained image processing adapter IP-Adapter module as auxiliary model structures, and couple them with the base large model to obtain a target image generation model.

13. The device according to claim 12, wherein: The IP-Adapter module is used to extract image features and text features, and perform cross-attention interaction on the image features and the text features to obtain a cross-feature representation, wherein the cross-feature representation is used to input the base large model.

14. The device according to claim 13, wherein: The first acquisition module is further used for: Divide the first sample image set according to the character vertical categories, and obtain a second sample image set corresponding to each character vertical category; For each of the character vertical categories, performing body feature recognition on each second sample image in the second sample image set corresponding to the character vertical category, the body features including facial features and limb features; Obtain reference body features corresponding to the character vertical category; The identified body features are matched with the reference body features, and a third sample image set corresponding to the person vertical category is selected from the second sample image set as a style reference image set.

15. The device according to claim 14, wherein: The first acquisition module is further used for: Performing person recognition segmentation on each first sample image in the first sample image set to obtain the person in the first sample image; Identifying the person in the first sample image into a person vertical category; The first sample image set is divided according to the identified character vertical categories, and a second sample image set corresponding to each character vertical category is obtained.

16. The device according to claim 13, wherein: The first acquisition module is further used for: Obtain an image uploaded by the user to the reference image area as a style reference image; Perform image augmentation on the style reference image to obtain a set of augmented style reference images.

17. The device according to claim 16, wherein: The first acquisition module is further used for: The style reference image is processed by at least one of geometric transformation, noise addition, affine transformation, and perspective transformation to obtain an augmented image generated after the processing, and the style reference image and the augmented image form a style reference image set.

18. The device according to claim 12, wherein: The second acquisition module is further used for: Obtaining a description text of the style reference image; Keywords are extracted from the description text, and a first prompt word corresponding to the style reference image is generated according to the extracted keywords.

19. The device according to claim 12, wherein: The second acquisition module is further used for: Extracting style features of the style reference image through a style transfer network; The style feature is converted into a first prompt word corresponding to the style reference image.

20. The device according to claim 15 or 17, wherein: The second acquisition module is further used for: The first prompt word is deduplicated.

21. An image generating device, wherein: The device comprises: A first acquisition module is used to acquire an original image uploaded by a user to be used for stylized image generation; A second acquisition module, used to acquire a second prompt word input by a user; An image generation module, used for inputting the original image and the second prompt word into a target image generation model to obtain a stylized image output by the target image generation model; Wherein, the target image generation model is trained and generated using the device described in any one of claims 12-20.

22. The device according to claim 21, wherein The device comprises: A determination module, used to determine the target image style corresponding to the target image generation model; A display module, used to generate a target image style selection box based on the target image style, and display the target image style selection box on a display interface; A third acquisition module, used to acquire a third prompt word input by a user; The image generation module is further used to input the original image and the third prompt word into the target image generation model in response to detecting a user's selection operation on the target image style selection box, and obtain a stylized image output by the target image generation model.

23. An electronic device, comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-9 or 10-11.

24. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-9 or 10-11.

25. A computer program product comprising a computer program which, when executed by a processor, implements the steps of the method according to any one of claims 1-9 or 10-11.

Images