Image processing method and device

Abstract

The embodiment of the invention provides an image processing method, and the method comprises the steps: obtaining an initial image and a preset mask, the initial image comprises an original image, and the preset mask comprises a first region and a second region; obtaining a target low-rank adaptation module according to the initial image; adjusting a diffusion model based on the target low-rank adaptation module to obtain a target diffusion model; based on the initial image, the preset mask and the target diffusion model, a target image is obtained, the size of the target image is the same as the size of the preset mask, the target image comprises the original image and an extended image, the original image corresponds to the first area, and the extended image corresponds to the second area. According to the technical scheme provided by the embodiment of the invention, natural and efficient expansion can be realized on the basis of keeping the original painting style, so that the expanded picture is consistent with the original picture in painting style, matched in content and smooth in transition, and the overall visual effect of image expansion is improved.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of artificial intelligence technology, and in particular, to an image processing method, apparatus, computer equipment, computer-readable storage medium, and computer program product. Background Art

[0002] With the rapid development of information technology, image creation has become an important way for people to express their creativity and has attracted more and more attention. Image creation includes multiple steps such as drawing, coloring, expansion, detail processing and post-editing. However, the current image expansion effect is still unstable. The expanded part does not match the original image in style and content, and the seam transition is unnatural, resulting in a poor browsing experience.

[0003] It should be noted that the above content is not necessarily prior art, nor is it intended to limit the scope of patent protection of this application. Summary of the invention

[0004] The embodiments of the present application provide an image processing method, apparatus, computer device, computer-readable storage medium, and computer program product to solve or alleviate one or more of the technical problems raised above.

[0005] One aspect of an embodiment of the present application provides an image processing method, the method comprising: Acquire an initial image and a preset mask, wherein the initial image includes an original picture, and the preset mask includes a first area and a second area; According to the initial image, obtaining a target low-rank adaptation module; Based on the target low-rank adaptation module, adjusting the pre-trained diffusion model to obtain a target diffusion model; Acquire a target image based on the initial image, the preset mask and the target diffusion model; The size of the target image is the same as the size of the preset mask, the target image includes the original picture and the extended picture, the original picture corresponds to the first area, and the extended picture corresponds to the second area.

[0006] Optionally, the original picture corresponds to the first area, including: the size of the original picture is the same as the size of the first area, and the position of the original picture in the target image is the same as the position of the first area in the preset mask; The extended picture corresponds to the second area, including: a size of the extended picture is the same as a size of the second area, and a position of the extended picture in the target image is the same as a position of the second area in the preset mask.

[0007] Optionally, obtaining a target low-rank adaptation module according to the initial image includes: Inputting the initial image into a pre-trained classification model, and obtaining the work corresponding to the initial image through the classification model; Determining the target low-rank adaptation module from low-rank adaptation modules of multiple works according to the work corresponding to the initial image; Among them, one work corresponds to a low-rank adaptation module.

[0008] Optionally, obtaining a target low-rank adaptation module according to the initial image includes: Processing the initial image by a labeler to obtain labeling information of the initial image; Matching the artwork corresponding to the initial image according to the annotation information of the initial image; Determining the target low-rank adaptation module from low-rank adaptation modules of multiple works according to the work corresponding to the initial image; Among them, one work corresponds to a low-rank adaptation module.

[0009] Optionally, the low-rank adaptation module of the work is obtained by the following operations: Acquire an image set of the work, wherein the image set includes a preset number of images of the work; The model is trained using the image set as training samples to obtain a low-rank adaptation module of the work.

[0010] Optionally, acquiring a target image based on the initial image, the preset mask and the target diffusion model includes: The initial image and the preset mask are input into the target diffusion model to obtain the target image through the target diffusion model.

[0011] Optionally, the image processing method further includes: Inputting the initial image into a pre-trained control network to obtain control information through the control network; Correspondingly, acquiring a target image based on the initial image, the preset mask and the target diffusion model includes: The initial image, the preset mask and the control information are input into the target diffusion model to obtain the target image through the target diffusion model.

[0012] Another aspect of an embodiment of the present application provides an image processing device, the device comprising: A first acquisition module, used to acquire an initial image and a preset mask, wherein the initial image includes an original picture, and the preset mask includes a first area and a second area; A second acquisition module, used for acquiring a target low-rank adaptation module according to the initial image; An adjustment module, used for adjusting the pre-trained diffusion model based on the target low-rank adaptation module to obtain a target diffusion model; A third acquisition module, configured to acquire a target image based on the initial image, the preset mask and the target diffusion model; The size of the target image is the same as the size of the preset mask, the target image includes the original picture and the extended picture, the original picture corresponds to the first area, and the extended picture corresponds to the second area.

[0013] Another aspect of an embodiment of the present application provides a computer device, including: at least one processor; and a memory communicatively coupled to the at least one processor; Wherein: the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method as described above.

[0014] Another aspect of an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer instructions, and when the computer instructions are executed by a processor, the method described above is implemented.

[0015] Another aspect of an embodiment of the present application provides a computer program product, including a computer program, which implements the method described above when executed by a processor.

[0016] The above technical solution adopted in the embodiment of the present application may have the following advantages: Obtain a preset mask and an initial image including the original picture. Based on the initial image, obtain a target low-rank adaptation module to adjust the pre-trained diffusion model to obtain a target diffusion model. Based on the initial image, the preset mask and the target diffusion model, a target image including the original picture and the extended picture and having the same size as the preset mask can be obtained. Among them, the original picture corresponds to the first area, and the extended picture corresponds to the second area. It can be seen that the embodiment of the present application fine-tunes the diffusion model through the target low-rank adaptation module to obtain a target diffusion model that adapts to the style of the original picture, thereby achieving a natural and efficient expansion on the basis of maintaining the original style, so that the extended picture is consistent with the style of the original picture, the content matches, and the transition is smooth, so as to achieve a seamless connection between the old and new pictures and enhance the overall visual effect of the dynamic comic. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The accompanying drawings exemplarily illustrate the embodiments and constitute a part of the specification, and together with the text description of the specification, are used to explain the exemplary implementation of the embodiments. The embodiments shown are for illustrative purposes only and do not limit the scope of the claims. In all drawings, the same reference numerals refer to similar but not necessarily identical elements.

[0018] Figure 1 The flowchart of the image processing method according to the first embodiment of the present application is schematically shown; Figure 2 The initial image and the target image according to the first embodiment of the present application are schematically shown; Figure 3 The newly added flow chart of the image processing method according to the first embodiment of the present application is schematically shown; Figure 4 Schematically shows Figure 1 Flow chart of sub-steps of step S102; Figure 5 Schematically shows Figure 1 Another sub-step flow chart of step S102; Figure 6 The overall flow chart of the image processing method according to the first embodiment of the present application is schematically shown; Figure 7 A block diagram schematically shows an image processing device according to the second embodiment of the present application; and Figure 8 The hardware architecture diagram of the computer device according to the third embodiment of the present application is schematically shown. DETAILED DESCRIPTION

[0019] In order to make the purpose, technical solutions and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application.

[0020] It should be noted that the descriptions involving "first", "second", etc. in the embodiments of the present application are only for descriptive purposes and cannot be understood as indicating or implying their relative importance or implicitly indicating the number of technical features indicated. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In addition, the technical solutions between the various embodiments can be combined with each other, but they must be based on the ability of ordinary technicians in the field to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such combination of technical solutions does not exist and is not within the scope of protection required by this application.

[0021] In the description of the present application, it should be understood that the numerical labels before the steps do not indicate the order in which the steps are executed, but are only used to facilitate the description of the present application and to distinguish each step, and therefore should not be understood as a limitation on the present application.

[0022] First, the following terms are explained: LoRA (Low-Rank Adaptation): Introduces a low-rank matrix to locally reparameterize the model to achieve efficient adaptive adjustment.

[0023] SD (Stable Diffusion) model: Diffusion model that can be used for AI image generation.

[0024] ControlNet: An extended model.

[0025] Mask: It is used to define the areas to be processed and the areas not to be processed in the image.

[0026] Image expansion: that is, picture expansion. In the embodiment of the present application, AI technology can be used to expand the known picture content and increase the content in the picture.

[0027] Dynamic comics: Convert comics into dynamic videos by adding character body movements, facial expressions and environmental dynamic effects to create comics with dynamic performance.

[0028] Secondly, in order to facilitate those skilled in the art to understand the technical solutions provided in the embodiments of the present application, the relevant technologies are described below: With the rapid development of information technology, image creation has become an important way for people to express their creativity and has attracted more and more attention. Image creation includes multiple steps such as drawing, coloring, expansion, detail processing and post-editing.

[0029] However, the applicant has learned that the relevant image expansion technology still has the following defects: (1) The expansion effect is unstable, the style and content of the expanded part do not match the original image, and there are logical contradictions and conflicts; (2) The placement of objects in certain scenes of the image needs to be consistent to ensure the coherence and consistency of the scene, and the relevant image expansion technology is still unable to maintain the stability of core elements and the consistency of scene objects; (3) Unnatural transitions are prone to occur at the edge seams, and the connection between the new and old images is poor, which affects the user experience.

[0030] To this end, the embodiment of the present application provides an image processing technology solution. In this technical solution: (1) Local reparameterization (LoRA) technology is used to train for a specific style of painting to ensure that the redrawn (expanded) style is consistent with the original image; (2) The image generation function of SD technology is used to generate new picture content, and ControlNet is applied for redrawing control to ensure seamless connection between new and old pictures; (3) The controllability of model training is further enhanced, and the stability, consistency, interpretability and controllability of the model are improved. While improving the model's ability to assist in image expansion, the stability of core elements and the consistency of scene objects are guaranteed; (4) Natural and efficient expansion is achieved, which improves the overall visual effect of dynamic comics. See below for details.

[0031] The technical solutions of the present application will be described below through a number of embodiments. It should be noted that these embodiments can be implemented in a variety of different forms and should not be construed as being limited to the embodiments described here.

[0032] Embodiment 1 Figure 1 The flowchart of the image processing method according to the first embodiment of the present application is schematically shown.

[0033] like Figure 1 As shown, the image processing method may include steps S100 to S106, wherein: Step S100: acquiring an initial image and a preset mask, wherein the initial image includes an original picture, and the preset mask includes a first area and a second area.

[0034] Step S102: acquiring a target low-rank adaptation module according to the initial image.

[0035] Step S104: Based on the target low-rank adaptation module, the pre-trained diffusion model is adjusted to obtain a target diffusion model.

[0036] Step S106, based on the initial image, the preset mask and the target diffusion model, a target image is acquired, wherein the size of the target image is the same as the size of the preset mask, the target image includes the original picture and an extended picture, the original picture corresponds to the first area, and the extended picture corresponds to the second area.

[0037] The image processing method provided in this embodiment obtains a preset mask and an initial image including an original picture. Based on the initial image, a target low-rank adaptation module is obtained to adjust the pre-trained diffusion model to obtain a target diffusion model. Based on the initial image, the preset mask and the target diffusion model, a target image including the original picture and the extended picture and having the same size as the preset mask can be obtained. Among them, the original picture corresponds to the first area, and the extended picture corresponds to the second area. It can be seen that the embodiment of the present application fine-tunes the diffusion model through the target low-rank adaptation module to obtain a target diffusion model that adapts to the original picture, thereby achieving a natural and efficient expansion on the basis of maintaining the original style, so that the extended picture is consistent with the style of the original picture, the content matches, and the transition is smooth, so as to achieve a seamless connection between the old and new pictures and enhance the overall visual effect of the dynamic comic.

[0038] The following combination Figure 1 , each step in steps S100~S106 and other optional steps are explained in detail.

[0039] Step S100 , obtaining an initial image and a preset mask, wherein the initial image includes an original picture, and the preset mask includes a first area and a second area.

[0040] The initial image can be various types of images, such as photos, paintings, screenshots, etc. Taking the comic scene as an example, the initial image can be any storyboard in the comic. Among them, the storyboard refers to each individual frame or frame in the comic. The initial image may include an unmodified storyboard picture (original picture) obtained directly from the comic. The preset mask can be a binary image, in which the black part (0 value) corresponds to the first area, and the first area does not need to be processed. The white part (1 value) can correspond to the second area, and the second area needs to be processed. The image processing method provided in the embodiment of the present application is used for expanding the picture, and accordingly, the first area can be exactly the same size as the initial image (i.e., the original picture) to shield the initial image during the expansion process to ensure the integrity and reliability of the original picture. The size of the preset mask can be set according to actual conditions, such as being configured to be the same as the size of the player window.

[0041] Step S102 , according to the initial image, obtain a target low-rank adaptation module.

[0042] Each comic has its own unique style. The objects corresponding to different scenes in the comics will also be different. For example, in ancient times, a room could have a bed, a desk, a screen, calligraphy and paintings, etc. (such as Figure 2As shown), modern rooms can have refrigerators, TVs, computers, desk lamps, etc. The AI ​​expansion of related technologies uses a large number of comics as samples for training, which may cause the model to be confused about the style of each comic and the objects corresponding to each scene, thereby generating an extended picture that is inconsistent with the style and content of the original picture, seriously affecting the user experience. If a dedicated expansion AI is trained for each comic separately, a large amount of training data and computing resources are required. To alleviate the above problems, the embodiment of the present application can train a low-rank adaptation module, namely LoRA, for each comic (each work). Exemplarily, the published comics can be extracted and analyzed, and the style of a specific comic can be learned and understood, thereby forming a low-rank adaptation module suitable for a specific comic. The low-rank adaptation module can be used as a plug-in for the SD model, and can be trained and updated independently of the SD model. Among them, the SD model has a picture-generating function and can be used for a variety of image processing tasks, such as coloring, repairing, and expanding pictures. The low-rank adaptation module can be used to ensure that the style of the redrawn part (expanded picture) is consistent with the original comic (original picture).

[0043] The low-rank adaptation module can be trained in a variety of ways, such as: simulated samples, synthetic samples, sample enhancement (image rotation, scaling), etc. An exemplary scheme is provided below.

[0044] In an optional embodiment, if Figure 3 As shown, the low-rank adaptation module of the work can be obtained by the following operations: Step S300: obtaining an image set of the work, wherein the image set includes a preset number of images of the work.

[0045] Step S302: Perform model training using the image set as training samples to obtain a low-rank adaptation module of the work.

[0046] For example, an image set of a work (such as a comic) can be obtained. The work can also be an illustration, a photo, etc. The image processing method of the embodiment of the present application is not limited to expanding comics, but can also expand other types of works. Taking comics as an example, the image set can include images obtained by intercepting published comics. Figure 2 As shown, the image set may include multiple images of different scenes, postures or sizes in the comics. The number of images can be determined according to the actual situation, for example: no less than 40-50, 200-300, etc. The low-rank adaptation module of the work can be obtained by training with the image set as training samples.

[0047] In this embodiment, a low-rank adaptation module of the work can be obtained through limited sample training, effectively reducing the demand for training data and the consumption of computing resources. The low-rank adaptation module can be used to fine-tune the diffusion model in the future, further saving computing resources by local reparameterization, and improving the diffusion model's ability to express specific comic styles, thereby optimizing the quality of the enlarged image.

[0048] The above embodiment exemplifies a method for training a LoRA for each work. The following further exemplifies how to quickly obtain a target low-rank adaptation module from the low-rank adaptation modules of multiple works.

[0049] In an optional embodiment, if Figure 4 As shown, step S102 may include: Step S400: input the initial image into a pre-trained classification model, and obtain the work corresponding to the initial image through the classification model.

[0050] Step S402: Determine the target low-rank adaptation module from the low-rank adaptation modules of multiple works according to the work corresponding to the initial image, wherein one work corresponds to one low-rank adaptation module.

[0051] Exemplarily, the initial image can be input into a pre-trained classification model. The classification result of the initial image is obtained through the classification model. The classification result can be the ID (identification) of the comic corresponding to the initial image, or the name of the comic, etc. Based on the classification result of the initial image, it can be determined which comic the initial image belongs to, that is, which work it corresponds to. Based on the work corresponding to the initial image, the target low-rank adaptation module can be quickly matched from the low-rank adaptation modules of multiple works.

[0052] In this embodiment, the classification model can be used to quickly identify the work to which the initial image belongs, so as to quickly determine the target low-rank adaptation module, effectively optimize the production efficiency of image expansion, and ensure the adaptability of the target diffusion model to the style of the initial image.

[0053] In an optional embodiment, if Figure 5 As shown, step S102 may include: Step S500: Process the initial image by a labeler to obtain labeling information of the initial image.

[0054] Step S502: matching the work corresponding to the initial image according to the annotation information of the initial image.

[0055] Step S504: Determine the target low-rank adaptation module from the low-rank adaptation modules of multiple works according to the work corresponding to the initial image, wherein one work corresponds to one low-rank adaptation module.

[0056] The annotator can be a pre-trained machine learning model or a tool with preset annotation rules, which can be used for image processing and analysis, such as automatically identifying and annotating specific objects in the initial image. Exemplarily, the annotator can be used to identify the content of the initial image and add one or more taggers, which can constitute the annotation information of the initial image. According to the annotation information of the initial image, the work corresponding to the initial image can be efficiently determined. Based on the work corresponding to the initial image, the target low-rank adaptation module can be quickly matched from the low-rank adaptation modules of multiple works.

[0057] In this embodiment, the initial image is processed by a tagger to obtain multiple taggers. The tagger can be used to infer the prompt words (annotation information), and the target low-rank adaptation module can be quickly obtained based on the prompt words, thereby ensuring the adaptability of the target diffusion model to the style of the initial image.

[0058] The above embodiments exemplarily introduce how to train and obtain a target low-rank adaptation module. The following will exemplarily introduce how to fine-tune a diffusion model based on a target low-rank adaptation module.

[0059] Step S104 , based on the target low-rank adaptation module, the pre-trained diffusion model is adjusted to obtain a target diffusion model.

[0060] Exemplarily, the target low-rank adaptation module may include adaptation parameters for the style of the initial image, which can be used to fine-tune the diffusion model. The diffusion model may be an SD model pre-trained on a large-scale dataset. Loading the pre-trained diffusion model, and loading the target low-rank adaptation module into a specific layer of the diffusion model, and fusing the parameters of LoRA with the existing parameters of the diffusion model, the target diffusion model can be obtained. The stability, interpretability, consistency and controllability of the target diffusion model after fusing LoRA are effectively optimized. The target diffusion model can ensure that the newly generated picture (extended picture) in the process of expanding the picture is consistent with the style and content of the original picture.

[0061] In this embodiment, the diffusion model is fine-tuned through the target low-rank adaptation module to obtain a target diffusion model that adapts to the style of the original picture. There is no need to completely retrain the diffusion model, which can reduce resource consumption and ensure the consistency and controllability of the enlarged picture style and content.

[0062] Step S106 , based on the initial image, the preset mask and the target diffusion model, a target image is acquired, wherein the size of the target image is the same as the size of the preset mask, the target image includes the original picture and an extended picture, the original picture corresponds to the first area, and the extended picture corresponds to the second area.

[0063] The target image can be obtained by applying the image generation function of the target diffusion model to expand the initial image. Figure 6 As shown, the input of AI is the initial image, and the final output is the target image. The target image may include an original picture and an extended picture, and the size of the target image may be the same as the size of a preset mask, wherein the original picture corresponds to the first area of ​​the preset mask, and the extended picture corresponds to the second area of ​​the preset mask. In an optional embodiment, the size of the original picture may be the same as the size of the first area, and the position of the original picture in the target image may be the same as the position of the first area in the preset mask. The extended picture may be the same size as the second area, and the position of the extended picture in the target image may be the same as the position of the second area in the preset mask. There are many ways to apply the target diffusion model, and different inputs may affect the quality of the output target image. Several exemplary schemes will be provided below.

[0064] In an optional embodiment, step S106 may include: inputting the initial image and the preset mask into the target diffusion model to obtain the target image through the target diffusion model.

[0065] Exemplarily, the initial image and the preset mask can be directly input into the target diffusion model. The target diffusion model can expand the canvas of the initial image based on the preset mask until the size of the canvas is the same as the preset mask, and the extended part of the canvas corresponds to the second area of ​​the preset mask. By using the Img2img Inpaint function of the SD model after the fusion of LoRA, the extended part of the canvas is redrawn to obtain an extended picture, and the extended picture is combined with the original picture to obtain the target image. In this embodiment, by fusing the SD model of the target LoRA, the extended picture can be quickly drawn, and the extended picture can be kept consistent with the style and content of the original picture, thereby improving the stability and performance of the target image and reducing the contradictions of the extended picture.

[0066] In an optional embodiment, the image processing method may further include: inputting the initial image into a pre-trained control network to obtain control information through the control network. Correspondingly, step S106 may include: inputting the initial image, the preset mask and the control information into the target diffusion model to obtain the target image through the target diffusion model.

[0067] The control network can be a pre-trained extended model for generating control information to adjust or guide the target diffusion model, thereby generating a target image with higher quality, more stability, and more natural transition between the old and new images. Exemplarily, the initial image can be input into the control network, and the control information can be generated through the control network. The control information can include style, detail enhancement, specific parameters, etc. The control information, the initial image, and the preset mask are input into the target diffusion model together, and the target diffusion model enhances the control capability of the extended image according to the control information, so that the transition between the extended image and the original image is natural.

[0068] In this embodiment, the control network is introduced to further enhance the image expansion capability of the target diffusion model, which can effectively improve the quality of AI image expansion, solve the problem of unnatural connection between new and old images, and improve the overall visual effect of dynamic animation.

[0069] In order to make this application easier to understand, the following Figure 2 and Figure 6 An exemplary application is provided.

[0070] S1: Get the initial image and the preset mask. The initial image is as follows: Figure 2 As shown on the left. Figure 6 As shown, the preset mask may be a binary image, the black portion may be the first region, and the white portion may be the second region.

[0071] S2: According to the initial image, obtain the target low-rank adaptation module.

[0072] S3: Adjust the SD model according to the target low-rank adaptation module to obtain the target diffusion model.

[0073] S4: Input the initial image into the pre-trained control network to obtain control information.

[0074] S5: Input the initial image, mask, and control information into the target diffusion model to obtain the target image through the target diffusion model. Figure 2 As shown on the right, it includes an original picture and an extended picture. The size and position of the original picture correspond to the first area, and the size and position of the extended picture correspond to the second area.

[0075] In this exemplary application: (1) Local reparameterization (LoRA) technology is used to train a specific comic style to ensure that the redrawn (expanded) style is consistent with the original comic; (2) The image generation function of SD technology is used to generate new picture content, and ControlNet is applied for redrawing control to ensure seamless connection between the new and old pictures; (3) The controllability of model training is further enhanced, and the stability, consistency, explainability and controllability of the model are improved. While improving the AI-assisted image expansion capability, the stability of core elements and the consistency of scene objects are guaranteed; (4) Natural and efficient expansion is achieved, improving the overall visual effect of dynamic comics.

[0076] Embodiment 2 Figure 7 The block diagram of the image processing device according to the second embodiment of the present application is schematically shown. The device can be divided into one or more program modules, one or more program modules are stored in a storage medium, and are executed by one or more processors to complete the embodiment of the present application. The program module referred to in the embodiment of the present application refers to a series of computer program instruction segments that can complete specific functions. The following description will specifically introduce the functions of each program module in this embodiment. Figure 7 As shown, the device 1000 may include: a first acquisition module 1100, a second acquisition module 1200, an adjustment module 1300 and a third acquisition module 1400, wherein: A first acquisition module 1100 is used to acquire an initial image and a preset mask, wherein the initial image includes an original picture, and the preset mask includes a first area and a second area; A second acquisition module 1200 is used to acquire a target low-rank adaptation module according to the initial image; An adjustment module 1300 is used to adjust the pre-trained diffusion model based on the target low-rank adaptation module to obtain a target diffusion model; A third acquisition module 1400 is used to acquire a target image based on the initial image, the preset mask and the target diffusion model; The size of the target image is the same as the size of the preset mask, the target image includes the original picture and the extended picture, the original picture corresponds to the first area, and the extended picture corresponds to the second area.

[0077] As an optional embodiment, the original picture corresponds to the first area, including: the size of the original picture is the same as the size of the first area, and the position of the original picture in the target image is the same as the position of the first area in the preset mask; The extended picture corresponds to the second area, including: a size of the extended picture is the same as a size of the second area, and a position of the extended picture in the target image is the same as a position of the second area in the preset mask.

[0078] As an optional embodiment, obtaining a target low-rank adaptation module according to the initial image includes: Inputting the initial image into a pre-trained classification model, and obtaining the work corresponding to the initial image through the classification model; Determining the target low-rank adaptation module from low-rank adaptation modules of multiple works according to the work corresponding to the initial image; Among them, one work corresponds to a low-rank adaptation module.

[0079] As an optional embodiment, obtaining a target low-rank adaptation module according to the initial image includes: Processing the initial image by a labeler to obtain labeling information of the initial image; Matching the artwork corresponding to the initial image according to the annotation information of the initial image; Determining the target low-rank adaptation module from low-rank adaptation modules of multiple works according to the work corresponding to the initial image; Among them, one work corresponds to a low-rank adaptation module.

[0080] As an optional embodiment, the low-rank adaptation module of the work is obtained by the following operations: Acquire an image set of the work, wherein the image set includes a preset number of images of the work; The model is trained using the image set as training samples to obtain a low-rank adaptation module of the work.

[0081] As an optional embodiment, acquiring a target image based on the initial image, the preset mask and the target diffusion model includes: The initial image and the preset mask are input into the target diffusion model to obtain the target image through the target diffusion model.

[0082] As an optional embodiment, the image processing apparatus 1000 is further configured to: Inputting the initial image into a pre-trained control network to obtain control information through the control network; Correspondingly, acquiring a target image based on the initial image, the preset mask and the target diffusion model includes: The initial image, the preset mask and the control information are input into the target diffusion model to obtain the target image through the target diffusion model.

[0083] Embodiment 3 Figure 8 The schematic diagram of the hardware architecture of a computer device 10000 suitable for implementing the image processing method according to the third embodiment of the present application is schematically shown. In some embodiments, the computer device 10000 may be a terminal device such as a smart phone, a wearable device, a tablet computer, a personal computer, a vehicle terminal, a game console, a virtual device, a workbench, a digital assistant, a set-top box, a robot, etc. In other embodiments, the computer device 10000 may be a rack server, a blade server, a tower server, or a cabinet server (including an independent server, or a server cluster composed of multiple servers), etc. Figure 8 As shown, the computer device 10000 includes but is not limited to: a memory 10010, a processor 10020, and a network interface 10030 that can communicate with each other through a system bus. Among them: The memory 10010 includes at least one type of computer-readable storage medium, and the readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (such as SD or DX memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 10010 can be an internal storage module of the computer device 10000, such as a hard disk or memory of the computer device 10000. In other embodiments, the memory 10010 can also be an external storage device of the computer device 10000, such as a plug-in hard disk equipped on the computer device 10000, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. Of course, the memory 10010 can also include both the internal storage module of the computer device 10000 and its external storage device. In this embodiment, the memory 10010 is generally used to store an operating system and various application software installed in the computer device 10000, such as program codes of image processing methods, etc. In addition, the memory 10010 can also be used to temporarily store various data that have been output or are to be output.

[0084] In some embodiments, the processor 10020 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other chips. The processor 10020 is generally used to control the overall operation of the computer device 10000, such as performing control and processing related to data interaction or communication with the computer device 10000. In this embodiment, the processor 10020 is used to run the program code stored in the memory 10010 or process data.

[0085] The network interface 10030 may include a wireless network interface or a wired network interface, and the network interface 10030 is generally used to establish a communication link between the computer device 10000 and other computer devices. For example, the network interface 10030 is used to connect the computer device 10000 to an external terminal through a network, and to establish a data transmission channel and a communication link between the computer device 10000 and the external terminal. The network may be a wireless or wired network such as an intranet, the Internet, the Global System of Mobile communication (GSM), Wideband Code Division Multiple Access (WCDMA), 4G network, 5G network, Bluetooth, Wi-Fi, etc.

[0086] It should be pointed out that Figure 8 Only a computer device having components 10010 - 10030 is shown, but it should be understood that implementation of all of the components shown is not a requirement, and more or fewer components may alternatively be implemented.

[0087] In this embodiment, the image processing method stored in the memory 10010 can also be divided into one or more program modules and executed by one or more processors (such as processor 10020) to complete the embodiment of the present application.

[0088] Embodiment 4 An embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored, wherein when the computer program is executed by a processor, the steps of the image processing method in the embodiment are implemented.

[0089] In this embodiment, the computer-readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (for example, SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, disk, optical disk, etc. In some embodiments, the computer-readable storage medium can be an internal storage unit of a computer device, such as a hard disk or memory of the computer device. In other embodiments, the computer-readable storage medium can also be an external storage device of a computer device, such as a plug-in hard disk equipped on the computer device, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. Of course, the computer-readable storage medium can also include both the internal storage unit of the computer device and its external storage device. In this embodiment, the computer-readable storage medium is generally used to store an operating system and various application software installed on the computer device, such as the program code of the image processing method in the embodiment, etc. In addition, the computer-readable storage medium can also be used to temporarily store various types of data that have been output or are to be output.

[0090] Embodiment 5 An embodiment of the present application also provides a computer program product, including a computer program, which implements the method in the above embodiment when executed by a processor.

[0091] Obviously, those skilled in the art should understand that the modules or steps of the above-mentioned embodiments of the present application can be implemented by general-purpose computer devices, they can be concentrated on a single computer device, or distributed on a network composed of multiple computer devices, optionally, they can be implemented by executable program codes of computer devices, so that they can be stored in a storage device and executed by the computer device, and in some cases, the steps shown or described can be executed in a different order from that herein, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. In this way, the embodiments of the present application are not limited to any specific combination of hardware and software.

[0092] It should be noted that the above are only preferred embodiments of the present application, and the patent protection scope of the present application is not limited thereto. Any equivalent structure or equivalent process transformation made using the contents of the specification and drawings of the present application, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. An image processing method, characterized in that: The method comprises: Acquire an initial image and a preset mask, wherein the initial image includes an original picture, and the preset mask includes a first area and a second area; According to the initial image, obtaining a target low-rank adaptation module; Based on the target low-rank adaptation module, adjusting the pre-trained diffusion model to obtain a target diffusion model; Acquire a target image based on the initial image, the preset mask and the target diffusion model; The size of the target image is the same as the size of the preset mask, the target image includes the original picture and the extended picture, the original picture corresponds to the first area, and the extended picture corresponds to the second area.

2. The method according to claim 1, characterized in that The original picture corresponds to the first area, including: the size of the original picture is the same as the size of the first area, and the position of the original picture in the target image is the same as the position of the first area in the preset mask; The extended picture corresponds to the second area, including: a size of the extended picture is the same as a size of the second area, and a position of the extended picture in the target image is the same as a position of the second area in the preset mask.

3. The method according to claim 1, characterized in that According to the initial image, a target low-rank adaptation module is obtained, including: Inputting the initial image into a pre-trained classification model, and obtaining the work corresponding to the initial image through the classification model; Determining the target low-rank adaptation module from low-rank adaptation modules of multiple works according to the work corresponding to the initial image; Among them, one work corresponds to a low-rank adaptation module.

4. The method according to claim 1, characterized in that According to the initial image, a target low-rank adaptation module is obtained, including: Processing the initial image by a labeler to obtain labeling information of the initial image; Matching the artwork corresponding to the initial image according to the annotation information of the initial image; Determining the target low-rank adaptation module from low-rank adaptation modules of multiple works according to the work corresponding to the initial image; Among them, one work corresponds to a low-rank adaptation module.

5. The method according to claim 3 or 4, characterized in that: The low-rank adaptation module of the work is obtained by the following operations: Acquire an image set of the work, wherein the image set includes a preset number of images of the work; The model is trained using the image set as training samples to obtain a low-rank adaptation module of the work.

6. The method according to claim 1, characterized in that Acquiring a target image based on the initial image, the preset mask and the target diffusion model includes: The initial image and the preset mask are input into the target diffusion model to obtain the target image through the target diffusion model.

7. The method according to claim 1, characterized in that Also includes: Inputting the initial image into a pre-trained control network to obtain control information through the control network; Correspondingly, acquiring a target image based on the initial image, the preset mask and the target diffusion model includes: The initial image, the preset mask and the control information are input into the target diffusion model to obtain the target image through the target diffusion model.

8. An image processing device, characterized in that: The device comprises: A first acquisition module, used to acquire an initial image and a preset mask, wherein the initial image includes an original picture, and the preset mask includes a first area and a second area; A second acquisition module, used for acquiring a target low-rank adaptation module according to the initial image; An adjustment module, used for adjusting the pre-trained diffusion model based on the target low-rank adaptation module to obtain a target diffusion model; A third acquisition module, configured to acquire a target image based on the initial image, the preset mask and the target diffusion model; The size of the target image is the same as the size of the preset mask, the target image includes the original picture and the extended picture, the original picture corresponds to the first area, and the extended picture corresponds to the second area.

9. A computer device, characterized in that: include: at least one processor; and a memory communicatively connected to the at least one processor; wherein: The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and when the computer instructions are executed by a processor, the method according to any one of claims 1 to 7 is implemented.

11. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to claims 1 to 7 are implemented.

Images