Image processing method and device

Abstract

The embodiment of the invention provides an image processing method, and the method comprises the steps: obtaining a low-rank adaptation module and an initial image of a plurality of roles, and the initial image comprises a target role; determining a low-rank adaptation module of the target role from the low-rank adaptation modules of the multiple roles according to the initial image; according to the low-rank adaptation module of the target role, adjusting a pre-trained diffusion model to obtain a target diffusion model; and obtaining a target image of the target role based on the initial image and the target diffusion model. According to the technical scheme of the embodiment of the invention, the diffusion model can be finely adjusted through the low-rank adaptation module of the target role, and the diffusion model capable of accurately identifying and coloring the target role is obtained, so that accurate coloring of the image is realized, the consistency of roles, clothing props and color matching is ensured, the production efficiency and quality are greatly improved, and the production cost is reduced. The labor cost is reduced; and the user experience is optimized.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of artificial intelligence technology, and in particular, to an image processing method, apparatus, computer equipment, computer-readable storage medium, and computer program product. Background Art

[0002] With the rapid development of information technology, image creation has become an important way for people to express their creativity and has attracted more and more attention. Image creation includes multiple steps such as drawing, coloring, detail processing and post-editing. However, current image creation still has problems such as uncontrollable process and unstable results, resulting in inconsistent image quality, confusing content and poor browsing experience.

[0003] It should be noted that the above content is not necessarily prior art, nor is it intended to limit the scope of patent protection of this application. Summary of the invention

[0004] The embodiments of the present application provide an image processing method, apparatus, computer device, computer-readable storage medium, and computer program product to solve or alleviate one or more of the technical problems raised above.

[0005] One aspect of an embodiment of the present application provides an image processing method, the method comprising: Obtaining low-rank adaptation modules and initial images of multiple characters, wherein the initial images include a target character; Determining, according to the initial image, a low-rank adaptation module of the target character from the low-rank adaptation modules of the multiple characters; According to the low-rank adaptation module of the target role, the pre-trained diffusion model is adjusted to obtain a target diffusion model; A target image of the target character is acquired based on the initial image and the target diffusion model.

[0006] Optionally, the initial image includes: the initial image includes: an uncolored first initial image or a second initial image, the first initial image includes the structure of the target character, and the second initial image is obtained by enhancing the structure of the target character in the first initial image; the target image includes a colored image.

[0007] Optionally, determining the low-rank adaptation module of the target character from the low-rank adaptation modules of the multiple characters according to the initial image includes: Calling a labeler to process the initial image to obtain character information; According to the role information, the low-rank adaptation module of the target role is obtained from the low-rank adaptation modules of the multiple roles.

[0008] Optionally, the image processing method further includes: Inputting the character information and the initial image into a pre-trained control network to obtain control information through the control network; Correspondingly, based on the initial image and the target diffusion model, acquiring a target image of the target character includes: The initial image and the control information are input into the target diffusion model to acquire the target image through the target diffusion model.

[0009] Optionally, acquiring a target image of the target character based on the initial image and the target diffusion model includes: The initial image is input into the target diffusion model to obtain the target image through the target diffusion model.

[0010] Optionally, the low-rank adaptation module for each role is obtained by: Acquire a visual reference set of a character, the visual reference set comprising: a plurality of colored pictures of the character; Model training is performed using the visual reference set as training samples to obtain a low-rank adaptation module of the character.

[0011] Optionally, the image processing method further includes: Iterating the target image multiple times through the target diffusion model until the target image obtained by the last iteration meets a preset standard; In each iteration, the target image output by the target diffusion model in the previous iteration is used as the input of the target diffusion model in this iteration.

[0012] Another aspect of an embodiment of the present application provides an image processing device, the device comprising: A first acquisition module, used to acquire low-rank adaptation modules and initial images of multiple characters, wherein the initial image includes a target character; A determination module, configured to determine the low-rank adaptation module of the target character from the low-rank adaptation modules of the multiple characters according to the initial image; An adjustment module, used for adjusting the pre-trained diffusion model according to the low-rank adaptation module of the target role to obtain a target diffusion model; The second acquisition module is used to acquire a target image of the target character based on the initial image and the target diffusion model.

[0013] Another aspect of an embodiment of the present application provides a computer device, including: at least one processor; and a memory communicatively coupled to the at least one processor; Wherein: the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method as described above.

[0014] Another aspect of an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer instructions, and when the computer instructions are executed by a processor, the method described above is implemented.

[0015] Another aspect of an embodiment of the present application provides a computer program product, including a computer program, which implements the method described above when executed by a processor.

[0016] The above technical solution adopted in the embodiment of the present application may have the following advantages: According to the obtained initial image of the target character (such as a draft or a line drawing), the low-rank adaptation module of the target character is determined from the low-rank adaptation modules (LoRA) of multiple characters. The pre-trained diffusion model (SD) is adjusted by the low-rank adaptation module of the target character to obtain the target diffusion model. Based on the initial image and the target diffusion model, the target image of the target character (such as a coloring image) can be obtained. It can be seen that the embodiment of the present application fine-tunes the diffusion model through the low-rank adaptation module of the target character to obtain a diffusion model that can accurately identify and color the target character, thereby achieving accurate image coloring, ensuring the consistency of characters, costumes, props and colors, greatly improving production efficiency and quality, reducing labor costs and optimizing user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The accompanying drawings exemplarily illustrate the embodiments and constitute a part of the specification, and together with the text description of the specification, are used to explain the exemplary implementation of the embodiments. The embodiments shown are for illustrative purposes only and do not limit the scope of the claims. In all drawings, the same reference numerals refer to similar but not necessarily identical elements.

[0018] Figure 1 The flowchart of the image processing method according to the first embodiment of the present application is schematically shown; Figure 2 The newly added flow chart of the image processing method according to the first embodiment of the present application is schematically shown; Figure 3 The diagram schematically shows a training sample of a low-rank adaptation module according to the first embodiment of the present application; Figure 4 Schematically shows Figure 1 Flow chart of sub-steps of step S102; Figure 5 The following schematically shows the application effect diagram of the image processing method according to the first embodiment of the present application; Figure 6 The following schematically shows the application effect diagram of the image processing method according to the first embodiment of the present application; Figure 7 A block diagram schematically shows an image processing device according to the second embodiment of the present application; and Figure 8 The hardware architecture diagram of the computer device according to the third embodiment of the present application is schematically shown. DETAILED DESCRIPTION

[0019] In order to make the purpose, technical solutions and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application.

[0020] It should be noted that the descriptions involving "first", "second", etc. in the embodiments of the present application are only for descriptive purposes and cannot be understood as indicating or implying their relative importance or implicitly indicating the number of technical features indicated. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In addition, the technical solutions between the various embodiments can be combined with each other, but they must be based on the ability of ordinary technicians in the field to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such combination of technical solutions does not exist and is not within the scope of protection required by this application.

[0021] In the description of the present application, it should be understood that the numerical labels before the steps do not indicate the order in which the steps are executed, but are only used to facilitate the description of the present application and to distinguish each step, and therefore should not be understood as a limitation on the present application.

[0022] First, the following terms are explained: Draft: A preliminary drawing that is used to plan the overall layout, character positions, actions, and scene involvement of the comic. The draft can provide the basic framework and structure of the comic.

[0023] Outlining: Based on the rough draft, the outline and framework of the rough draft are outlined with clear lines to form a line drawing. Outlining ensures that the shape and position of the drawn content are accurate, preparing for the subsequent coloring process.

[0024] Coloring: Fill the line drawing with color based on the completed line drawing. By controlling the depth (brightness) and saturation of the color, the objects in the line drawing are expressed, giving the picture texture, vitality and vitality.

[0025] LoRA (Low-Rank Adaptation): Introduces a low-rank matrix to locally reparameterize the model to achieve efficient adaptive adjustment.

[0026] SD (Stable Diffusion) model: Diffusion model that can be used for AI image generation.

[0027] ControlNet: An extended model.

[0028] Secondly, in order to facilitate those skilled in the art to understand the technical solutions provided in the embodiments of the present application, the relevant technologies are described below: With the rapid development of information technology, image creation has become an important way for people to express their creativity and has attracted more and more attention. Image creation includes multiple steps such as drawing, coloring, detail processing and post-editing. However, current image creation still has problems such as uncontrollable process and unstable results, resulting in inconsistent image quality, confusing content and poor browsing experience.

[0029] To this end, an embodiment of the present application provides a technical solution for image processing. In this technical solution: (1) A LoRA (low-rank adaptation module) is trained separately for each character. By locally reparameterizing the SD model through LoRA, the model can be trained to accurately identify and color specific human images (roles) with less training data and video memory usage, thereby ensuring the consistency of the color of the specific character, overcoming the problems of inconsistent coloring and unstable quality in the raw images of traditional models. (2) The controllability of model training is further enhanced, the stability, interpretability, consistency and controllability of the model are improved, and the quality of the images generated by the model is significantly improved, which can directly achieve the effect of manual outlining and coloring. See below for details.

[0030] The technical solutions of the present application are described below through multiple embodiments. It should be noted that these embodiments can be implemented in a variety of different forms and should not be construed as being limited to the embodiments described here.

[0031] Embodiment 1 Figure 1 The flowchart of the image processing method according to the first embodiment of the present application is schematically shown.

[0032] like Figure 1 As shown, the image processing method may include steps S100 to S106, wherein: Step S100, obtaining low-rank adaptation modules and initial images of multiple characters, wherein the initial image includes a target character.

[0033] Step S102: determining, according to the initial image, a low-rank adaptation module of the target character from the low-rank adaptation modules of the multiple characters.

[0034] Step S104: adjusting the pre-trained diffusion model according to the low-rank adaptation module of the target role to obtain a target diffusion model.

[0035] Step S106: acquiring a target image of the target character based on the initial image and the target diffusion model.

[0036] The image processing method provided in this embodiment determines the low-rank adaptation module of the target character from the low-rank adaptation modules (LoRA) of multiple characters according to the acquired initial image of the target character (such as a draft or a line drawing). The pre-trained diffusion model (SD) is adjusted by the low-rank adaptation module of the target character to obtain the target diffusion model. Based on the initial image and the target diffusion model, the target image of the target character (such as a coloring image) can be obtained. It can be seen that the embodiment of the present application fine-tunes the diffusion model through the low-rank adaptation module of the target character to obtain a diffusion model that can accurately identify and color the target character, thereby achieving accurate coloring of the image, ensuring the consistency of the characters, costumes, props and colors, greatly improving production efficiency and quality, reducing labor costs and optimizing user experience.

[0037] The following combination Figure 1 , each step in steps S100~S106 and other optional steps are explained in detail.

[0038] Step S100 , obtaining low-rank adaptation modules and initial images of multiple characters, wherein the initial image includes a target character.

[0039] The target character can be a human, an object, a plant, a fantasy creature (such as a dragon, a unicorn, a goblin, etc.), or an inanimate object (such as a car, furniture, etc.). The initial image can include the target character. In an optional embodiment, the first initial image or the second initial image is not colored. The first initial image can include the basic structure of the target character (such as a draft, in which the general outline of the target character is drawn with rough lines). The second initial image can be obtained by strengthening the first initial image (such as a line drawing, obtained by outlining the outline of the target character on the enhanced draft). Taking a human character as an example, the initial image can include a human body, a hairstyle, clothing, props (such as accessories), etc. Obtaining the initial image can be to obtain a draft or line drawing of the target character to prepare for subsequent coloring work.

[0040] Step S102 , according to the initial image, determining the low-rank adaptation module of the target character from the low-rank adaptation modules of the multiple characters.

[0041] Each comic has its own unique style, and even in the same comic, the styles of different characters may be different. The style of a character is determined by elements such as hairstyle, clothing, props, color matching, and expression. The AI ​​in the related art uses a large number of comics as samples for training, which may cause confusion in the model's cognition of the characteristics of each character, thereby generating images inconsistent with the style of the target character, seriously affecting the user experience. If a dedicated coloring model is trained for each character separately, a large amount of training data and computing resources are required. To alleviate the above problems, this embodiment can train a low-rank adaptation module, i.e., LoRA, for each character separately. For example, comics that have been colored in the past can be analyzed to learn and understand the style of a specific character, thereby forming a low-rank adaptation module for a specific character. The low-rank adaptation module can be used as a plug-in for the SD model and trained and updated independently of the SD model. Among them, the SD model can be used for image processing, such as image coloring. The low-rank adaptation module can improve the adaptability of the SD model to a specific character and maintain the consistency of the character style (such as color matching).

[0042] The low-rank adaptation module can be trained in many ways, such as simulated samples, synthetic samples, sample enhancement (image rotation, scaling), etc. An exemplary scheme is provided below.

[0043] In an optional embodiment, if Figure 2 As shown, the low-rank adaptation module of the role can be obtained by the following steps: Step S200, obtaining a visual reference set of the character, wherein the visual reference set includes: a plurality of colored pictures of the character.

[0044] Step S202: Perform model training using the visual reference set as training samples to obtain a low-rank adaptation module for the character.

[0045] For example, a visual reference set of a character may be obtained, and the visual reference set may include multiple colored pictures of the character, or multiple screenshots of the character taken from past colored works, such as Figure 3 As shown in the figure. The visual reference set may include colored pictures of the character at different angles, different expressions, different postures, different scenes, and different parts of the character. The number of colored pictures or screenshots can be determined according to the actual situation, such as 10 to 20, 40 to 50, etc. The low-rank adaptation module of the character can be obtained by training the model with the visual reference set or the screenshot set as training samples.

[0046] In this embodiment, the low-rank adaptation module of the role is obtained through limited sample training, which can reduce the training data requirements and computing resource consumption. The low-rank adaptation module is subsequently applied to the fine-tuning of the diffusion model, and the computing resources are further saved through local reparameterization, and the adaptability of the diffusion model to specific roles is improved.

[0047] The above embodiment exemplarily introduces a method for training a LoRA for each role separately. The following further exemplifies how to accurately obtain the low-rank adaptation module corresponding to the target role from the low-rank adaptation modules of multiple roles.

[0048] In an optional embodiment, if Figure 4 As shown, step S102 may include: Step S400: calling a labeler to process the initial image to obtain character information.

[0049] Step S402: acquiring the low-rank adaptation module of the target role from the low-rank adaptation modules of the multiple roles according to the role information.

[0050] The tagger can be a pre-trained machine learning model or a tool with preset rules, which can be used for image processing, such as automatically identifying and tagging specific objects in the initial image. For example, the tagger can identify and analyze the initial image and add one or more tags (such as human, cat, car, etc.) to the target character in the initial image. The aforementioned taggers can constitute the character information. Figure 5 An initial image of a target character (a female line drawing) is shown as an example, and the character information of the target character obtained by the annotator can be expressed as "human, female, butterfly hair accessories, bangs, loose hair, open coat, butterfly clothing...". According to the acquired character information of the target character, the low-rank adaptation module of the target character can be quickly matched from the low-rank adaptation modules of multiple characters. In some embodiments, the role ID (role identification) and the role information can be pre-stored in a preset database in association with each other, and the role ID can be bound to the low-rank adaptation module corresponding to the role. After calling the annotator to process the initial image to obtain the character information, the character information can be used to query the preset database, and when the matching degree is greater than the preset threshold or a complete match, a unique role ID is determined. The low-rank adaptation module of the target character can be retrieved and accurately obtained through the unique role ID.

[0051] In this embodiment, the initial image is processed by a tagger to obtain multiple taggers. The taggers can be used to infer the prompt words (role information), and the low-rank adaptation module of the target role can be quickly obtained based on the prompt words, thereby ensuring the adaptability of the target diffusion model to the target role.

[0052] In some embodiments, the initial image may be input into a pre-trained classification model, and the classification result of the initial image may be obtained through the classification model. The classification result may be the character ID of the target character, or the name of the comic book or character to which the target character belongs. Based on the classification result, the low-rank adaptation module of the target character may be efficiently and accurately obtained from the low-rank adaptation modules of multiple characters.

[0053] The above-mentioned multiple embodiments exemplarily introduce how to train and obtain the low-rank adaptation module of the target role. The following exemplarily introduces how to fine-tune the diffusion model based on the low-rank adaptation module of the target role.

[0054] Step S104 , according to the low-rank adaptation module of the target role, the pre-trained diffusion model is adjusted to obtain the target diffusion model.

[0055] Exemplarily, the low-rank adaptation module of the target character may include adaptation parameters of the target character, which can be used to fine-tune the model. The diffusion model may be an SD model pre-trained on a large-scale dataset. Loading the pre-trained diffusion model, and loading the low-rank adaptation module of the target character into a specific layer of the diffusion model, and fusing the parameters of LoRA with the existing parameters of the diffusion model, the target diffusion model can be obtained. The stability, interpretability, consistency and controllability of the target diffusion model after fusing LoRA are effectively optimized. The target diffusion model can accurately identify the target character and color the target character, thereby ensuring the color consistency of the target character.

[0056] In this embodiment, the diffusion model is fine-tuned through the low-rank adaptation module to obtain a target diffusion model that is adapted to the coloring of the target character. There is no need to completely retrain the diffusion model, which can reduce resource consumption and ensure the consistency and controllability of the coloring.

[0057] Step S106 , based on the initial image and the target diffusion model, obtaining a target image of the target character.

[0058] The target image can be a colored image (or shading image) of the target character obtained by coloring the initial image using the Wensheng image and / or Tusheng image function of the target diffusion model. Figure 5 As shown, Figure 5 The left side is the initial image, and the right side is the target image. There are many ways to apply the target diffusion model, and different inputs can affect the quality of the output target image. Several exemplary solutions will be provided below.

[0059] In an optional embodiment, the step S106 may include: inputting the initial image into the target diffusion model to acquire the target image through the target diffusion model.

[0060] For example, the initial image can be directly input into the target diffusion model, and the target image can be quickly generated based on the image generation function of the target diffusion model. In this embodiment, by fusing the SD model of the target character corresponding to LoRA, the target character in the initial image can be quickly identified and the consistency of coloring can be ensured, thereby improving the stability and performance of the target image and reducing uncontrollable differences.

[0061] In an optional embodiment, the image processing method may further include: inputting the character information and the initial image into a pre-trained control network to obtain control information through the control network. Correspondingly, step S106 may include: inputting the initial image and the control information into the target diffusion model to obtain the target image through the target diffusion model.

[0062] The control network can be a pre-trained extended model for generating control information to adjust or guide the target diffusion model to generate a higher quality and more stable target image. Exemplarily, the character information and the initial image can be input into the control network at the same time, and the control network can combine the two to generate control information. Among them, the control information may include style, detail enhancement, specific parameters, etc. The control information and the initial image are simultaneously input into the target diffusion model, and the target diffusion model enhances the control ability of the draft or line draft according to the control information, so that the AI ​​(SD model) can accurately color based on the initial image (draft or line draft of the painter). In some embodiments, the Line Art model in the control network can also add specific parameters to the control information to control the target diffusion model to retain the original line draft in the generated target image. Since the target diffusion model has superimposed the LoRA of the target character, the final target image can be faithful to the original draft or original line draft, and the style consistency, color consistency, and costume prop consistency of the target character can be maintained.

[0063] In this embodiment, the control network is introduced to further enhance the coloring accuracy of the target diffusion model, which can effectively improve the quality of AI-generated images, approach the performance of manual outlining and coloring, and greatly improve the production efficiency of comic creation.

[0064] In some embodiments, if different colored clothing is required for the target character, this can be easily achieved by adjusting the tagger (character information) in the prompt word and then modifying the control information.

[0065] The above embodiments exemplarily introduce how the image processing method provided by the present application realizes image colorization. In practical applications, the above image processing method still has room for further optimization. An exemplary solution will be provided below.

[0066] In an optional embodiment, the image processing method may further include: performing multiple iterations of the target image through the target diffusion model until the target image obtained in the last iteration meets a preset standard. In each iteration, the target image output by the target diffusion model in the previous iteration is used as the input of the target diffusion model in the current iteration.

[0067] For example, the LoRA technology can be used to perform multiple image generation iterations. That is, the target image is iterated multiple times through the target diffusion model. In each iteration, the target image obtained in the previous iteration (the output of the target diffusion model) is used as the input of the target diffusion model of this iteration, until the target image obtained in the last iteration reaches a preset annotation, such as a preset threshold value of clarity. Figure 6 As shown, Figure 6 The left side is the target image without iterative processing. Figure 6 The right side is the target image after iterative processing. Comparing the two, we can see that iterative processing can strengthen the lines and details of the target image, further improving the quality of the target image.

[0068] In this embodiment, the target image is iterated multiple times using LoRA technology to further enhance the clarity of the lines and the overall quality of the target image, improve the consistency and naturalness of comic coloring, and further enhance the production efficiency of AI-assisted comic creation.

[0069] In order to make the present application easier to understand, an exemplary application is provided below.

[0070] S1: Train a LoRA for each character in each comic separately to obtain low-rank adaptation modules for multiple characters.

[0071] S2: Obtain low-rank adaptation modules and initial images of multiple characters, where the initial image includes a target character, and the initial image may be a draft or line drawing of the target character.

[0072] S3: The initial image is processed by the tagger to obtain multiple taggers. The tagger is used to infer the prompt word (role information). Based on the role information, the LoRA of the target role is called to locally re-parameterize the SD model (diffusion model) to obtain the target diffusion model.

[0073] S4: Input the initial image and character information into the pre-trained control network to generate control information through the control network.

[0074] The control information is used to enhance the control of the target diffusion model over the draft or line draft.

[0075] S5: Inputting the initial image and the control information into the target diffusion model to obtain a target image through the target diffusion model. The target image may be a colored image.

[0076] In this exemplary application: (1) A LoRA (low-rank adaptation module) is trained separately for each character in each comic. By locally reparameterizing the SD model through LoRA, AI can be trained to accurately identify and color specific characters (roles) with less training data and video memory usage, thereby ensuring the consistency of the color of specific characters, overcoming the problems of inconsistent coloring and unstable quality in traditional AI raw images. (2) The controllability of model training is further enhanced, and the stability, explainability, consistency and controllability of the model are improved. The quality of AI-generated images is significantly improved, and the effect of manual outlining and coloring can be directly achieved.

[0077] Embodiment 2 Figure 7 The block diagram of the image processing device according to the second embodiment of the present application is schematically shown. The device can be divided into one or more program modules, one or more program modules are stored in a storage medium, and are executed by one or more processors to complete the embodiment of the present application. The program module referred to in the embodiment of the present application refers to a series of computer program instruction segments that can complete specific functions. The following description will specifically introduce the functions of each program module in this embodiment. Figure 7 As shown, the apparatus 1000 may include: a first acquisition module 1100, a determination module 1200, an adjustment module 1300 and a second acquisition module 1400, wherein: A first acquisition module 1100 is used to acquire low-rank adaptation modules and initial images of multiple characters, wherein the initial image includes a target character; A determination module 1200 is used to determine the low-rank adaptation module of the target character from the low-rank adaptation modules of the multiple characters according to the initial image; An adjustment module 1300 is used to adjust the pre-trained diffusion model according to the low-rank adaptation module of the target role to obtain a target diffusion model; The second acquisition module 1400 is used to acquire a target image of the target character based on the initial image and the target diffusion model.

[0078] As an optional embodiment, the initial image includes: an uncolored first initial image or a second initial image, the first initial image includes the structure of the target character, and the second initial image is obtained by enhancing the structure of the target character in the first initial image; the target image includes a colored image.

[0079] As an optional embodiment, determining the low-rank adaptation module of the target character from the low-rank adaptation modules of the multiple characters according to the initial image includes: Calling a labeler to process the initial image to obtain character information; According to the role information, the low-rank adaptation module of the target role is obtained from the low-rank adaptation modules of the multiple roles.

[0080] As an optional embodiment, the image processing apparatus 1000 is further configured to: Inputting the character information and the initial image into a pre-trained control network to obtain control information through the control network; Correspondingly, based on the initial image and the target diffusion model, acquiring a target image of the target character includes: The initial image and the control information are input into the target diffusion model to acquire the target image through the target diffusion model.

[0081] As an optional embodiment, acquiring a target image of the target character based on the initial image and the target diffusion model includes: The initial image is input into the target diffusion model to obtain the target image through the target diffusion model.

[0082] As an optional embodiment, the low-rank adaptation module of each role is obtained by the following operations: Acquire a visual reference set of a character, the visual reference set comprising: a plurality of colored pictures of the character; Model training is performed using the visual reference set as training samples to obtain a low-rank adaptation module of the character.

[0083] As an optional embodiment, the image processing apparatus 1000 is further configured to: Iterating the target image multiple times through the target diffusion model until the target image obtained by the last iteration meets a preset standard; In each iteration, the target image output by the target diffusion model in the previous iteration is used as the input of the target diffusion model in this iteration.

[0084] Embodiment 3 Figure 8The schematic diagram of the hardware architecture of a computer device 10000 suitable for implementing the image processing method according to the third embodiment of the present application is schematically shown. In some embodiments, the computer device 10000 may be a terminal device such as a smart phone, a wearable device, a tablet computer, a personal computer, a vehicle terminal, a game console, a virtual device, a workbench, a digital assistant, a set-top box, a robot, etc. In other embodiments, the computer device 10000 may be a rack server, a blade server, a tower server, or a cabinet server (including an independent server, or a server cluster composed of multiple servers), etc. Figure 8 As shown, the computer device 10000 includes but is not limited to: a memory 10010, a processor 10020, and a network interface 10030 that can communicate with each other through a system bus. Among them: The memory 10010 includes at least one type of computer-readable storage medium, and the readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (such as SD or DX memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 10010 can be an internal storage module of the computer device 10000, such as a hard disk or memory of the computer device 10000. In other embodiments, the memory 10010 can also be an external storage device of the computer device 10000, such as a plug-in hard disk equipped on the computer device 10000, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. Of course, the memory 10010 can also include both the internal storage module of the computer device 10000 and its external storage device. In this embodiment, the memory 10010 is generally used to store an operating system and various application software installed in the computer device 10000, such as program codes of image processing methods, etc. In addition, the memory 10010 can also be used to temporarily store various data that have been output or are to be output.

[0085] In some embodiments, the processor 10020 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other chips. The processor 10020 is generally used to control the overall operation of the computer device 10000, such as performing control and processing related to data interaction or communication with the computer device 10000. In this embodiment, the processor 10020 is used to run the program code stored in the memory 10010 or process data.

[0086] The network interface 10030 may include a wireless network interface or a wired network interface, and the network interface 10030 is generally used to establish a communication link between the computer device 10000 and other computer devices. For example, the network interface 10030 is used to connect the computer device 10000 to an external terminal through a network, and to establish a data transmission channel and a communication link between the computer device 10000 and the external terminal. The network may be a wireless or wired network such as an intranet, the Internet, the Global System of Mobile communication (GSM), Wideband Code Division Multiple Access (WCDMA), 4G network, 5G network, Bluetooth, Wi-Fi, etc.

[0087] It should be pointed out that Figure 8 Only a computer device having components 10010 - 10030 is shown, but it should be understood that implementation of all of the components shown is not a requirement, and more or fewer components may alternatively be implemented.

[0088] In this embodiment, the image processing method stored in the memory 10010 can also be divided into one or more program modules and executed by one or more processors (such as processor 10020) to complete the embodiment of the present application.

[0089] Embodiment 4 An embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored, wherein when the computer program is executed by a processor, the steps of the image processing method in the embodiment are implemented.

[0090] In this embodiment, the computer-readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (for example, SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, disk, optical disk, etc. In some embodiments, the computer-readable storage medium can be an internal storage unit of a computer device, such as a hard disk or memory of the computer device. In other embodiments, the computer-readable storage medium can also be an external storage device of a computer device, such as a plug-in hard disk equipped on the computer device, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. Of course, the computer-readable storage medium can also include both the internal storage unit of the computer device and its external storage device. In this embodiment, the computer-readable storage medium is generally used to store an operating system and various application software installed on the computer device, such as the program code of the image processing method in the embodiment, etc. In addition, the computer-readable storage medium can also be used to temporarily store various types of data that have been output or are to be output.

[0091] Embodiment 5 An embodiment of the present application also provides a computer program product, including a computer program, which implements the method in the above embodiment when executed by a processor.

[0092] Obviously, those skilled in the art should understand that the modules or steps of the above-mentioned embodiments of the present application can be implemented by general-purpose computer devices, they can be concentrated on a single computer device, or distributed on a network composed of multiple computer devices, optionally, they can be implemented by executable program codes of computer devices, so that they can be stored in a storage device and executed by the computer device, and in some cases, the steps shown or described can be executed in a different order from that herein, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. In this way, the embodiments of the present application are not limited to any specific combination of hardware and software.

[0093] It should be noted that the above are only preferred embodiments of the present application, and the patent protection scope of the present application is not limited thereto. Any equivalent structure or equivalent process transformation made using the contents of the specification and drawings of the present application, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. An image processing method, characterized in that: The method comprises: Obtaining low-rank adaptation modules and initial images of multiple characters, wherein the initial images include a target character; Determining, according to the initial image, a low-rank adaptation module of the target character from the low-rank adaptation modules of the multiple characters; According to the low-rank adaptation module of the target role, the pre-trained diffusion model is adjusted to obtain a target diffusion model; A target image of the target character is acquired based on the initial image and the target diffusion model.

2. The method according to claim 1, characterized in that The initial image includes: an uncolored first initial image or a second initial image, the first initial image includes the structure of the target character, and the second initial image is obtained by enhancing the structure of the target character in the first initial image; the target image includes a colored image.

3. The method according to claim 1, characterized in that Determining, according to the initial image, a low-rank adaptation module of the target character from the low-rank adaptation modules of the multiple characters, comprising: Calling a labeler to process the initial image to obtain character information; According to the role information, the low-rank adaptation module of the target role is obtained from the low-rank adaptation modules of the multiple roles.

4. The method according to claim 1, characterized in that Also includes: Inputting the character information and the initial image into a pre-trained control network to obtain control information through the control network; Correspondingly, based on the initial image and the target diffusion model, acquiring a target image of the target character includes: The initial image and the control information are input into the target diffusion model to acquire the target image through the target diffusion model.

5. The method according to claim 1, characterized in that Acquiring a target image of the target character based on the initial image and the target diffusion model includes: The initial image is input into the target diffusion model to obtain the target image through the target diffusion model.

6. The method according to claim 1, characterized in that The low-rank adaptation module of each role is obtained by the following operations: Acquire a visual reference set of a character, the visual reference set comprising: a plurality of colored pictures of the character; Model training is performed using the visual reference set as training samples to obtain a low-rank adaptation module of the character.

7. The method according to any one of claims 1 to 6, characterized in that: Also includes: Iterating the target image multiple times through the target diffusion model until the target image obtained by the last iteration meets a preset standard; In each iteration, the target image output by the target diffusion model in the previous iteration is used as the input of the target diffusion model in this iteration.

8. An image processing device, characterized in that: The device comprises: A first acquisition module, used to acquire low-rank adaptation modules and initial images of multiple characters, wherein the initial image includes a target character; A determination module, configured to determine the low-rank adaptation module of the target character from the low-rank adaptation modules of the multiple characters according to the initial image; An adjustment module, used for adjusting the pre-trained diffusion model according to the low-rank adaptation module of the target role to obtain a target diffusion model; The second acquisition module is used to acquire a target image of the target character based on the initial image and the target diffusion model.

9. A computer device, characterized in that: include: at least one processor; and a memory communicatively connected to the at least one processor; wherein: The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and when the computer instructions are executed by a processor, the method according to any one of claims 1 to 7 is implemented.

11. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to claims 1 to 7 are implemented.

Images