Image generation method and device and electronic equipment

Abstract

The invention relates to an image generation method and device and electronic equipment. The method comprises the following steps: acquiring a to-be-restored original image; segmenting the original image to obtain multiple frames of original sub-images; repairing each frame of original sub-image through a diffusion model with a target style to obtain a corresponding repaired sub-image; and splicing each frame of repaired sub-image to obtain a repaired target image. According to the scheme provided by the invention, the large-size image can be effectively and smoothly processed on the premise of minimizing the loss of details, and the target image with the target style can be efficiently and accurately repaired and obtained.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an image generation method, device and electronic device. Background Art

[0002] With the rapid development of artificial intelligence technology, AIGC (Artificial Intelligent Generative Content) has been widely used in various fields, including art creation. AIGC technology has brought new possibilities and opportunities to art creation, enabling artists and creators to generate various styles of art more efficiently with the power of artificial intelligence.

[0003] As a treasure of ancient Chinese art, mural art is famous for its unique style, exquisite craftsmanship and rich cultural connotations. Due to the age and the influence of various factors such as nature and man, most murals have varying degrees of damage. The murals are missing, blurred, and the color aging areas make the murals unable to be fully presented. The existing treatment method is manual restoration. However, before the operator starts the restoration, if the content of the restoration cannot be accurately grasped, the progress or accuracy of the restoration will be affected; repeated operations may cause further damage to the murals. Therefore, how to accurately restore mural art has become a crucial and challenging task. Summary of the invention

[0004] In order to solve or partially solve the problems existing in the related art, the present application provides an image generation method, device and electronic device, which can effectively and smoothly process large-size images while minimizing the loss of details, and efficiently and accurately repair and obtain a target image with a target style.

[0005] The first aspect of the present application provides an image generation method, which includes:

[0006] Obtain the original image to be restored;

[0007] Segmenting the original image to obtain multiple frames of original sub-images;

[0008] The original sub-images of each frame are repaired respectively by using a pre-trained diffusion model to obtain corresponding repaired sub-images;

[0009] The repaired sub-images of each frame are spliced ​​to obtain a repaired target image.

[0010] In some implementations, the pre-trained diffusion model is obtained by training according to the following method:

[0011] Acquire training data, the training data comprising sample images having the target style and corresponding text annotations;

[0012] According to the training data, a low-rank adaptation model based on the initial diffusion model is trained to obtain a low-rank matrix corresponding to the trained low-rank adaptation model;

[0013] The low-rank matrix is ​​added to the weight matrix corresponding to the key layer of the initial diffusion model to obtain a diffusion model with a target style.

[0014] In some implementations, the sample images include real-scene images and virtual images, and the number of the real-scene images is greater than the number of the virtual images; the types of text annotations include color style, paint texture, and picture content.

[0015] In some implementations, while or after the original sub-images of each frame are restored by using a diffusion model having a target style to obtain corresponding restored sub-images, the method includes:

[0016] The restored sub-image is optimized according to the prompt text, and / or the lines, colors and composition of the restored sub-image are optimized to obtain an optimized restored sub-image.

[0017] In some implementation modes, segmenting the original image to obtain multiple frames of original sub-images includes:

[0018] The original image is segmented according to a preset size to obtain multiple frames of original sub-images; wherein adjacent multiple frames of original sub-images have overlapping areas.

[0019] In some implementations, the step of stitching the restored sub-images to obtain a restored target image includes:

[0020] According to the initial positions of the original sub-images, the corresponding restored sub-images are spliced ​​to obtain a restored target image; wherein the overlapping areas of adjacent multiple frames of the restored sub-images display corresponding color intensity values ​​according to a preset weighting rule.

[0021] In some implementations, before or after segmenting the original image, the method further includes:

[0022] The resolution of the original image or the original sub-image is improved to obtain the optimized original image or the original sub-image.

[0023] A second aspect of the present application provides an image generating device, comprising:

[0024] An image acquisition module, used to acquire the original image to be restored;

[0025] An image segmentation module is used to segment the original image to obtain multiple frames of original sub-images;

[0026] An image restoration module is used to restore the original sub-images of each frame by using a diffusion model with a target style to obtain corresponding restored sub-images;

[0027] The image stitching module is used to stitch the repaired sub-images to obtain a repaired target image.

[0028] In some embodiments, the device further comprises a detail enhancement module and / or an image optimization module, wherein:

[0029] The detail enhancement module is used to enhance the resolution of the original image or the original sub-image to obtain an optimized original image or original sub-image;

[0030] The image optimization module is used to optimize the restored sub-image according to the prompt text, and / or optimize the lines, colors and composition of the restored sub-image to obtain an optimized restored sub-image.

[0031] In a third aspect, the present application provides a diffusion model training device, which is used to train and obtain the diffusion model with the target style described in the first aspect; the training device comprises:

[0032] A data acquisition module, used to acquire training data, wherein the training data includes sample images with the target style and corresponding text annotations;

[0033] A model training module, used to train a low-rank adaptation model based on the initial diffusion model according to the training data, and obtain a low-rank matrix corresponding to the trained low-rank adaptation model;

[0034] The parameter updating module is used to add the low-rank matrix to the weight matrix corresponding to the key layer of the initial diffusion model to obtain a diffusion model with a target style.

[0035] A fourth aspect of the present application provides an electronic device, including:

[0036] Processor; and

[0037] A memory having executable codes stored thereon, which, when executed by the processor, causes the processor to execute the image generating method as described in the first aspect above.

[0038] A fifth aspect of the present application provides a computer-readable storage medium having executable code stored thereon. When the executable code is executed by a processor of an electronic device, the processor executes the image generation method as described in the first aspect above.

[0039] The sixth aspect of the present application provides a computer program product, including a computer program, characterized in that the computer program is used to execute computer program code instructions corresponding to the image generation method described in the first aspect above.

[0040] The technical solution provided by this application may have the following beneficial effects:

[0041] The image generation method of the present application performs image generation through a diffusion model, and generates a repaired target image of a target style based on the original image, so that the target image can fully present the original appearance of the damaged area; in addition, by processing the original image in a manner of first segmenting and then splicing, it is ensured that a target image of a full size is obtained, and each image detail is restored to a greater extent, ensuring that damaged pixels in small areas are repaired without omission, so as to generate a complete and detailed target image for display.

[0042] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] The above and other objects, features and advantages of the present application will become more apparent by describing in more detail the exemplary embodiments of the present application in conjunction with the accompanying drawings, wherein the same reference numerals generally represent the same components in the exemplary embodiments of the present application.

[0044] Figure 1 It is a flowchart of an image generation method provided in an embodiment of the present application;

[0045] Figure 2 It is a flowchart of a method for training a diffusion model provided in an embodiment of the present application;

[0046] Figure 3 is another flowchart of the image generation method provided in an embodiment of the present application;

[0047] Figure 4 is a schematic diagram of segmentation of the original image shown in this application;

[0048] Figure 5 is a schematic diagram of splicing the repaired sub-images shown in this application;

[0049] Figure 6 is a schematic diagram of the structure of an image generating device provided in an embodiment of the present application;

[0050] Figure 7 is another structural schematic diagram of the image generating device provided in an embodiment of the present application;

[0051] Figure 8It is a structural schematic diagram of a diffusion model training device provided in an embodiment of the present application;

[0052] Fig. 9 It is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0053] The embodiments of the present application will be described in more detail below with reference to the accompanying drawings. Although the embodiments of the present application are shown in the accompanying drawings, it should be understood that the present application can be implemented in various forms and should not be limited by the embodiments described herein. On the contrary, these embodiments are provided to make the present application more thorough and complete, and to fully convey the scope of the present application to those skilled in the art.

[0054] The terms used in this application are for the purpose of describing specific embodiments only and are not intended to limit this application. The singular forms of "a", "said" and "the" used in this application and the appended claims are also intended to include plural forms unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items.

[0055] It should be understood that although the terms "first", "second", "third", etc. may be used in this application to describe various information, this information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of this application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Thus, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of this application, the meaning of "multiple" is two or more, unless otherwise clearly and specifically defined.

[0056] In the related technologies, most murals are no longer in their original state due to their age and the influence of various factors. They have various damages such as fading of pigments, peeling of murals, hollowing, cracks, and partial missing. Before artificial restoration, if the complete original appearance of the murals can be accurately predicted for guided restoration, it will be helpful to improve the restoration efficiency of the murals and avoid secondary damage. In addition, the restoration of mural art not only requires high professional talents and professional skills, but also has the characteristics of magnificent composition, numerous images, and large scale. It is very difficult to handle manually and the cycle is also very long.

[0057] In response to the above problems, the present application provides an image generation method that can effectively and smoothly process large-size images while minimizing detail loss, and efficiently and accurately repair and obtain a target image with a target style.

[0058] The technical solution of the present application is described in detail below with reference to the accompanying drawings.

[0059] See also Figure 1 , an image generation method shown in the present application includes:

[0060] S110, obtaining an original image to be restored.

[0061] In the present application, the original image may be an image with a target style, wherein the target style may be set or selected according to needs, for example, a mural style, a sculpture style, a stone carving style, etc. The original image is used for restoration processing based on the image, and may be obtained by photographing the real scene to be restored, or by drawing a sketch or a design drawing, etc.

[0062] S120, segmenting the original image to obtain multiple frames of original sub-images.

[0063] The size of mural art scenes is generally large, and the mural art in the corresponding original image is also large. In the same frame of the original image, there may be many different forms of damage, and the damaged areas are scattered in different positions in the original image, and the size of each damaged area is different. If the large-size original image is used directly for subsequent processing, serious detail loss and image generation errors will occur. This application divides a frame of the original image into multiple frames of original sub-images, so that each frame of the original sub-image can be processed separately in the subsequent steps, thereby retaining all the details on the original image to the greatest extent.

[0064] S130, repairing the original sub-images of each frame respectively by using the pre-trained diffusion model to obtain corresponding repaired sub-images.

[0065] The diffusion model is a generative model based on a deep learning algorithm. It generates high-quality content through a gradual denoising and denoising process. It performs well in generating images, videos, audio and other high-dimensional data, and has strong performance in generation tasks. In the present disclosure, the pre-trained diffusion model is a pre-trained model for image generation. The diffusion model can generate a latent space representation of a repaired sub-image based on the latent space representation of the original sub-image, and further map the latent space representation of the repaired sub-image to the pixel space through a decoder to obtain a repaired sub-image. Among them, the latent space representation of the original sub-image is a latent space expression obtained by encoding the original sub-image using an image encoder, such as a variational auto-encoder (VAE for short). The image encoder is responsible for mapping the image data from the original pixel space to the latent space (Latent Space) to obtain an implicit and continuous representation, which can make the model calculation more efficient; and the decoder can reconstruct the original image from this latent space representation, and map the generation result of the diffusion model from the latent space to the pixel space.

[0066] In the present application, the diffusion model can be used as a basic model and trained by a pre-training data set containing image samples of the target style, so that the trained diffusion model can directionally generate images with the target style. In the present application, the target style can be, for example, the style of a mural, which is only an example and not a limitation. For different types of target styles, training can be performed independently to obtain multiple diffusion models that can generate different styles.

[0067] It can be understood that the trained diffusion model can transfer the target style of each frame of the original sub-image respectively, by regenerating and adjusting the pixels of the original image, especially correcting the damaged area with the target style, so that each frame of the repaired sub-image generated can retain the approximate content of the original image, and the repaired damaged area presents the characteristics of the target style.

[0068] S140, stitching the restored sub-images of each frame to obtain a restored target image.

[0069] It can be understood that the placement position of each frame of the repaired sub-image is the same as the corresponding original sub-image. The repaired sub-images of each frame are placed correspondingly according to the position mark information of the original sub-image, and the repaired sub-images at each adjacent position are spliced ​​together to form a target image with the same size as the original image. At the same time, the original appearance of the damaged area of ​​the original image is restored through the diffusion model, and there is no damaged area in the target image, and then a target image with a complete picture is obtained.

[0070] The image generation method of the present application performs image generation through a diffusion model, and generates a repaired target image of a target style based on the original image, so that the target image can fully present the original appearance of the damaged area; in addition, by processing the original image in a manner of first segmenting and then splicing, it is ensured that a target image of a full size is obtained, and each image detail is restored to a greater extent, ensuring that damaged pixels in small areas are repaired without omission, so as to generate a complete and detailed target image for display.

[0071] See also Figure 2 An embodiment of the present application further provides a method for training a diffusion model with a target style of the present application. The diffusion model with a target style of the present application can be obtained by training according to the following method.

[0072] S210, obtaining training data, where the training data includes sample images with a target style and corresponding text annotations.

[0073] The text annotation includes at least the following information: the picture content of the sample image and the relevant information corresponding to the target style. The relevant information corresponding to the target style includes at least one of the following: category feature information and modeling expression techniques. Category feature information includes regional cultural symbols of art, style feature information (such as oil painting, pencil drawing, mural, sculpture painting), etc. Modeling expression techniques refer to the techniques and means of creating artistic images, such as lines and light and shade.

[0074] For example, the Yongle Palace murals use the traditional heavy color filling method, with azurite and malachite as the main colors of the whole painting, and added simple tones such as brown, hematite, and cinnabar for embellishment. The whole picture looks steady, thick and decorative. Therefore, the above-mentioned characteristics of the Yongle Palace murals can be fully applied to the training model through the relevant information corresponding to the target style, which can make the generation of images in this style more accurate.

[0075] In order to train a diffusion model that can generate target style images, the sample images must have the target style for subsequent model learning. In some implementations, the sample images include real images and virtual images, and the number of real images is greater than the number of virtual images; the types of text annotations include color style, paint texture, and picture content description.

[0076] In some embodiments, the collected original sample images are segmented to obtain corresponding real-life images; and the collected original sample images are repaired to obtain corresponding virtual images. Taking the target style as the style of murals as an example, after capturing and collecting large-sized murals, a frame of large-sized original sample images can be segmented to form multiple frames of small-sized real-life images. The sizes of each frame of real-life images can be the same or different, and different real-life images can include pictures with the same partial area or pictures with different full areas. Further, the real-life images can be divided into positive sample images and negative sample images, and the number of positive sample images is greater than the number of negative sample images. Among them, the positive sample images occupy the vast majority, which are real-life images without damaged areas. That is, by segmenting the original sample images, while enriching the number of samples, the damaged areas can also be deleted and the pictures with normal areas retained can be used as positive sample images, so that the model can learn more details of the target style. On the contrary, the negative sample images occupy a minority, which are real-life images with damaged areas.

[0077] Furthermore, the sample images also include a small amount of virtual images. The virtual images can be drawn images with the target style, or they can be artificial restoration images based on the digital tablet corresponding to the real scene images, that is, the real scene images with damaged areas are restored by using the digital tablet. By adding a small amount of virtual images, the data volume of the positive sample images can be increased.

[0078] After collecting the sample images, text annotation is performed on each frame of the sample image. The content of the text annotation includes the color style corresponding to each frame of the sample image, the texture of the mineral pigment, and the content presented in the mural. The text annotation corresponding to a real-life image of a mural includes, for example: man, solo, male focus, facial hair, crown, beard, portrait, black eyes, black hair, mustache, ancient fresco, mineral pigment, realistic. For negative sample images, the text annotation can further include a feature description of the damaged area, which serves as a negative description so that the model can learn the negative features and eliminate the damaged features through negative prompt words during generation. Through the fitting and detailed text annotation of the positive sample images and the negative sample images, the subsequent model can be further guided to learn the target style.

[0079] S220, training a low-rank adaptation model based on the initial diffusion model according to the training data, and obtaining a low-rank matrix corresponding to the trained low-rank adaptation model.

[0080] In some specific implementations, the initial diffusion model may be, for example, a Stable Diffusion model (hereinafter referred to as the SD model) in the related art. The SD model is a generative deep learning model in the field of AIGC, which is mainly used for text-to-image generation or image-to-image generation. The low-rank adaptation model (LoRA model) adds a trainable low-rank matrix to train and adjust the overall initial diffusion model on the basis of the initial diffusion model. Taking the SD model as an example, these low-rank matrices can be inserted near the key layer of the SD model to change the behavior of the SD model under specific tasks or styles. Among them, the SD model is usually a model based on the Transformer architecture, and accordingly, the key layer may be a hidden layer near the multi-head attention layer (Multi-Head Attention) or the feed-forward neural network layer (Feed-Forward Network). The training of the LoRA model in this embodiment realizes the ability to generate high-quality target style images through a small amount of training data, and does not affect the general capabilities of the basic diffusion model.

[0081] Before training begins, you can also customize the values ​​of hyperparameters, such as learning rate, rank, training steps, batch size, regularization parameters, etc. Here are just examples. During the training process, by inputting training data and performing back propagation based on the loss value calculated by the preset loss function, you can adjust the weight parameters in the low-rank matrix. When the training of the preset batch is completed and the loss value reaches convergence, the trained low-rank matrix is ​​obtained.

[0082] In some implementations, the loss function may be selected from L1 (absolute value loss function) and / or L2 (mean square error loss function). In some specific implementations, when the loss function includes both L1 and L2, the corresponding two loss values ​​calculated in each round may be weightedly added to obtain a unique total loss value for reverse parameter adjustment of the low-rank matrix. When the total loss value converges, a trained low-rank matrix is ​​obtained.

[0083] In other embodiments, the initial diffusion model may also be a Midjourney model, a DALL-E 2 model, a Flux model, etc., which are merely exemplified and not limiting.

[0084] S230, adding the low-rank matrix to the weight matrix corresponding to the key layer of the initial diffusion model to obtain a diffusion model with the target style.

[0085] After training to obtain a low-rank matrix, the weight parameters corresponding to the trained low-rank matrix can be added to the original weight parameters of the key layer. By fine-tuning the weight parameters of these key layers, the features related to the target style learned by the LoRA model can be incorporated, and the behavior of the initial diffusion model under specific tasks or target styles can be changed to obtain a diffusion model that can generate images in the target style.

[0086] In summary, the training method of the diffusion model of the present application, the trained diffusion model can generate a target image based on the input original sub-image. Compared with the method of retraining the weight parameters of all hidden layers of the entire diffusion model to form a directional generation of the target style, the present application introduces a low-rank adaptation model to train the target style. By training the LoRA model to extract the trained low-rank matrix, and then adding the low-rank matrix to the original weight matrix corresponding to the initial diffusion model, the initial diffusion model can be fine-tuned to form a new diffusion model with the target style. Such a design can quickly adapt to specific tasks or styles by adding a small amount of trained low-rank matrices to the initial diffusion model without the need for large-scale retraining of the entire diffusion model, thereby improving training efficiency and saving training costs.

[0087] See also Figure 3 Based on the trained diffusion model with the target style in the above embodiment, the image generation method of the present application will be further introduced in combination with the embodiment below, which includes:

[0088] S310: Acquire an original image to be restored.

[0089] In the present application, the original image to be restored may be a photographed image corresponding to the mural, and the size of the original image is not limited thereto.

[0090] S320, dividing the original image according to a preset size to obtain multiple frames of original sub-images; wherein the adjacent multiple frames of original sub-images have overlapping areas.

[0091] In some embodiments, the preset size can be selected from (64-1024)×(64-1024), and the preset size is the size of each original sub-image after segmentation. Optionally, the preset size can be a square size such as 64×64, 128×128, 256×256, 512×512, 768×768, 1024×1024, or other rectangular sizes. The corresponding preset size can be selected according to the size of the original image for reasonable segmentation. This is only an example and is not limited.

[0092] In some implementations, the original image is segmented according to the same preset size, and the size of each original sub-image after segmentation is the same. Preferably, the preset size may be 512×512. It is understood that an appropriate size may control the number of original sub-images obtained, thereby balancing image quality, generation speed, and consumption of computing resources.

[0093] In some implementations, the original image is segmented according to a plurality of preset sizes, and the sizes of the multiple frames of original sub-images after segmentation may be partially the same or partially different. For example, the original sub-image containing the damaged area may have a larger preset size, that is, larger than the size of other original sub-images not containing the damaged area. Such a design may allow for more detailed repair of the damaged area.

[0094] Furthermore, taking the nine-grid grid as an example, each frame of the original sub-image has 3 to 8 adjacent frames of the original sub-image. When segmenting, each frame of the current original sub-image and the edge area of ​​the corresponding adjacent original sub-image have an intersection, and the intersection area forms an overlapping area. For ease of understanding, as Figure 4 As shown, a frame of original image is divided into 4 frames of original sub-images A / B / C / D, and the size of each frame of original sub-image is, for example, 512×512. Among them, in each frame of original sub-image, an area of, for example, 64×512 is defined in the edge area adjacent to other images as an overlapping area. This design facilitates the subsequent splicing process of S360, which will not be elaborated here.

[0095] In some implementations, in a single-frame original sub-image, the cumulative area of ​​all overlapping regions is smaller than the total area of ​​the single-frame original sub-image. For example, the cumulative area of ​​all overlapping regions may be 10% to 50% of the total area.

[0096] S330, improving the resolution of the original image or the original sub-image to obtain an optimized original image or the original sub-image.

[0097] In this step, the details of the original image or the original sub-image can be further improved before the image is repaired. In some implementations, the super-resolution model in the relevant technology can be used to improve the resolution of the original image or each frame of the original sub-image after segmentation. In this way, the repair effect of the subsequent steps is more refined based on the optimized original image or the original sub-image.

[0098] Among them, the super-resolution model is a model that can upgrade low-resolution images to high-resolution images. It can learn the mapping relationship between low-resolution and high-resolution images, and use the prior knowledge in the image (such as texture, edges, etc.) to reconstruct a high-resolution image with richer details, thereby improving the resolution of the original image or the original sub-image.

[0099] It should be noted that this step S330 can be executed or not. When it is executed, it can be executed before S320, that is, the resolution of the unsegmented original image frame is increased. Or it can be executed after S320, that is, the resolution of the segmented multiple frames of original sub-images is increased respectively.

[0100] S340, repairing the original sub-images of each frame respectively by using the pre-trained diffusion model to obtain corresponding repaired sub-images.

[0101] In this step, the diffusion model obtained by training according to the above training method can generate the target style of the mural in a directional manner. It can be understood that the damaged area in the original sub-image does not conform to the color style and pigment texture of the mural. After being identified and adjusted by the diffusion model, the color style, pigment texture and picture content of the damaged area are transformed and supplemented to the target style, and the corresponding repaired sub-image is obtained.

[0102] S350, optimizing the restored sub-image according to the prompt text, and / or optimizing the lines, colors and composition of the restored sub-image to obtain an optimized restored sub-image.

[0103] It should be noted that this step can be performed synchronously with S340 or after S340. That is, when the diffusion model in step S340 is used to repair the original sub-image, this step can be combined with synchronous fine-tuning and optimization of the repaired sub-image so that the diffusion model directly outputs the optimized repaired sub-image; or after the preliminary repaired sub-image is output in step S340, the optimization can be performed in step S350.

[0104] Among them, the user can customize the prompt text, and the diffusion model fine-tunes the restored sub-image according to the input prompt text. Optionally, the prompt text can be flexibly adjusted with reference to the text annotations in the above training data, or it can be a preset trigger word, such as ancient fresco, mineral pigment, so as to fine-tune the color style, pigment texture and picture content in the restored sub-image. The diffusion model performs restoration based on the intervention of the prompt text to obtain a better restored sub-image.

[0105] Furthermore, plug-ins can be connected to the above diffusion model to optimize the lines, colors and composition in the repaired sub-image. For example, the tail plug-in in ControlNet can be selected to control the color and composition of the image, and the line 2 plug-in can be selected to control the lines in the image. By combining multiple plug-ins, the diffusion model can simultaneously limit the changes in the color, composition and lines in the image when generating the image, so that the generated new image is more in line with the requirements of the target style.

[0106] S360, according to the initial positions of the original sub-images, respectively splicing the corresponding optimized restored sub-images to obtain a restored target image; wherein the overlapping areas of adjacent multi-frame restored sub-images display corresponding color intensity values ​​according to a preset weighting rule.

[0107] In this step, each frame of the restored sub-image is placed according to the position of the corresponding original sub-image, and multiple adjacent frames of the restored sub-image are spliced ​​to form a complete target image, and the target image has the same size as the original image.

[0108] Furthermore, in order to make the image transition at the splicing point smoother, the overlapping area of ​​each frame of the repaired sub-image displays the corresponding color intensity value according to the preset weighting rule. In some embodiments, in the overlapping area of ​​adjacent multi-frame repaired sub-images, the sum of the weights of the pixels at the same position in the multi-frame repaired sub-images is equal to 1; the weight values ​​of the pixels at the same position in different repaired sub-images are set according to the distance. The closer the distance to the center point of the current repaired sub-image, the greater the weight value of the pixel; otherwise, the smaller the weight value.

[0109] For ease of understanding, Figure 5 As shown, the adjacent areas of the restored sub-images A and B are overlapping areas, and the pixel positions of the overlapping areas of the two are completely consistent. Among them, the pixels A1 and A2 of the restored sub-image A are the pixels B1 and B2 at the corresponding positions in the restored sub-image B. A1, A2, B1 and B2 all have corresponding color intensity values ​​(RGB values). Among them, the distance between the pixel A1 and the center point of the restored sub-image A is smaller than the distance between the pixel B1 and the center point of the restored sub-image B. Therefore, the weight value corresponding to the pixel A1 can be, for example, 0.9, and the weight value corresponding to the pixel B1 can be, for example, 0.1, 0.9+0.1=1. The distance between the pixel A2 and the center point of the restored sub-image A is greater than the distance between the pixel B2 and the center point of the restored sub-image B. Therefore, the weight value corresponding to the pixel A2 can be, for example, 0.1, and the weight value corresponding to the pixel B2 can be, for example, 0.9, 0.1+0.9=1. When the repaired sub-images A and B are spliced ​​to form the target image M, the pixel points A1 and B1 overlap to form the corresponding pixel point M1, and the pixel points A2 and B2 overlap to form the corresponding pixel point M2. Then the RGB M1 =0.9RGB A1 +0.1RGB B1 , RGB M2 =0.1RGB A2 +0.9RGB B2By analogy, the RGB value of each pixel in the overlapped area after stitching can be calculated respectively, and the stitched area displayed by the calculated RGB value has a smoother and more natural color transition. It can be understood that the above weight values ​​are only for illustration and are not limited thereto.

[0110] In summary, the image generation method of the present application can repair large-sized damaged original images by first segmenting and then splicing, and by using weighted fusion of pixel color intensity values, so that the repaired target image can be presented completely and smoothly while minimizing detail loss; at the same time, through a pre-trained diffusion model, the damaged area can be adjusted to a picture that conforms to the target style, so that the repaired target image can accurately present the target style with exquisite details, which is helpful for users to perform comparative repair and has a high reference value for repair.

[0111] Corresponding to the aforementioned application function realization method embodiment, the present application also provides an image generation device, a diffusion model training device and corresponding embodiments.

[0112] Figure 6 It is a schematic diagram of the structure of the image generating device shown in this application.

[0113] See also Figure 6 The image generation device shown in the present application includes an image acquisition module 610, an image segmentation module 620, an image restoration module 630 and an image stitching module 640. Among them:

[0114] The image acquisition module 610 is used to acquire the original image to be restored.

[0115] The image segmentation module 620 is used to segment the original image to obtain multiple frames of original sub-images.

[0116] The image restoration module 630 is used to restore the original sub-images of each frame respectively through the pre-trained diffusion model to obtain corresponding restored sub-images.

[0117] The image stitching module 640 is used to stitch the restored sub-images to obtain a restored target image.

[0118] In some specific implementations, the image segmentation module 620 is used to segment the original image according to a preset size to obtain multiple frames of original sub-images; wherein the adjacent multiple frames of original sub-images have overlapping areas.

[0119] In some specific implementations, the image stitching module 640 is used to stitch the corresponding restored sub-images according to the initial positions of the original sub-images to obtain a restored target image; wherein the overlapping areas of adjacent multi-frame restored sub-images display corresponding color intensity values ​​according to a preset weighting rule.

[0120] See also Figure 7 In some specific implementations, the image generation device of the present application further includes a detail enhancement module 650 for enhancing the resolution of the original image or the original sub-image to obtain an optimized original image or the original sub-image.

[0121] In some specific embodiments, the image generation device of the present application also includes an image optimization module 660, which is used to optimize the repaired sub-image according to the prompt text, and / or to optimize the lines, colors and composition of the repaired sub-image to obtain an optimized repaired sub-image.

[0122] From this example, it can be seen that the image generation device of the present application can perform segmentation, repair and splicing restoration based on the large-size original image. While minimizing the loss of details, it can effectively and smoothly process large-size images, and efficiently and accurately repair and obtain the target image of the large-size image to meet the requirements of the target style.

[0123] See also Figure 8 One embodiment of the present application further provides a diffusion model training device, which is used to train and obtain the diffusion model with the target style in the above embodiment; the training device includes:

[0124] The data acquisition module 810 is used to acquire training data, where the training data includes sample images with a target style and corresponding text annotations.

[0125] The model training module 820 is used to train the low-rank adaptation model based on the initial diffusion model according to the training data, and obtain the low-rank matrix corresponding to the trained low-rank adaptation model.

[0126] The parameter updating module 830 is used to add the low-rank matrix to the weight matrix corresponding to the key layer of the initial diffusion model to obtain a diffusion model with a target style.

[0127] It can be seen from this example that the diffusion model training device of the present application can directionally generate a diffusion model with a target style. By adopting a low-rank adaptation model, only a small amount of training data and a shorter training cycle are required to train a diffusion model capable of generating images with a target style, thereby improving the training efficiency of the model and reducing the training cost of the model.

[0128] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated again here.

[0129] Fig. 9 It is a schematic diagram of the structure of the electronic device shown in this application.

[0130] See also Fig. 9 , the electronic device 1000 includes a memory 1010 and a processor 1020 .

[0131] The processor 1020 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or the processor may be any conventional processor, etc.

[0132] The memory 1010 may include various types of storage units, such as system memory, read-only memory (ROM), and permanent storage devices. Among them, ROM can store static data or instructions required by the processor 1020 or other modules of the computer. The permanent storage device may be a readable and writable storage device. The permanent storage device may be a non-volatile storage device that does not lose the stored instructions and data even after the computer is powered off. In some embodiments, the permanent storage device uses a large-capacity storage device (such as a magnetic or optical disk, flash memory) as a permanent storage device. In some other embodiments, the permanent storage device may be a removable storage device (such as a floppy disk, optical drive). The system memory may be a readable and writable storage device or a volatile readable and writable storage device, such as a dynamic random access memory. The system memory may store some or all instructions and data required by the processor at run time. In addition, the memory 1010 may include any combination of computer-readable storage media, including various types of semiconductor storage chips (such as DRAM, SRAM, SDRAM, flash memory, programmable read-only memory), and disks and / or optical disks may also be used. In some embodiments, the memory 1010 may include a readable and / or writable removable storage device, such as a laser disc (CD), a read-only digital versatile disc (such as a DVD-ROM, a double-layer DVD-ROM), a read-only Blu-ray disc, an ultra-density optical disc, a flash memory card (such as an SD card, a mini SD card, a Micro-SD card, etc.), a magnetic floppy disk, etc. The computer-readable storage medium does not include carrier waves and transient electronic signals transmitted wirelessly or wired.

[0133] The memory 1010 stores executable codes, and when the executable codes are processed by the processor 1020 , the processor 1020 can execute part or all of the methods described above.

[0134] In addition, the method according to the present application may also be implemented as a computer program or a computer program product, which includes computer program code instructions for executing some or all of the steps in the above method of the present application.

[0135] Alternatively, the present application can also be implemented as a computer-readable storage medium (or non-transitory machine-readable storage medium or machine-readable storage medium) on which executable code (or computer program or computer instruction code) is stored. When the executable code (or computer program or computer instruction code) is executed by a processor of an electronic device (or server, etc.), the processor executes part or all of the steps of the above-mentioned method according to the present application.

[0136] The embodiments of the present application have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and changes will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or improvements to the technology in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. An image generation method, characterized in that: include: Obtain the original image to be restored; Segmenting the original image to obtain multiple frames of original sub-images; The original sub-images of each frame are repaired respectively by using a pre-trained diffusion model to obtain corresponding repaired sub-images; The repaired sub-images of each frame are spliced ​​to obtain a repaired target image.

2. The method according to claim 1, characterized in that The pre-trained diffusion model is obtained by training in the following manner: Acquire training data, the training data including sample images having the target style and corresponding text annotations; According to the training data, a low-rank adaptation model based on the initial diffusion model is trained to obtain a low-rank matrix corresponding to the trained low-rank adaptation model; The low-rank matrix is ​​added to the weight matrix corresponding to the key layer of the initial diffusion model to obtain a diffusion model with a target style.

3. The method according to claim 2, characterized in that The sample images include real scene images and virtual images, and the number of the real scene images is greater than the number of the virtual images; The types of text annotations include color style, paint texture, and picture content.

4. The method according to claim 1, characterized in that: While or after the original sub-images of each frame are restored by using the diffusion model with the target style to obtain the corresponding restored sub-images, the method includes: The restored sub-image is optimized according to the prompt text, and / or the lines, colors and composition of the restored sub-image are optimized to obtain an optimized restored sub-image.

5. The method according to claim 1, characterized in that The step of segmenting the original image to obtain multiple frames of original sub-images includes: The original image is divided according to a preset size to obtain multiple frames of original sub-images; wherein adjacent multiple frames of original sub-images have overlapping areas.

6. The method according to claim 5, characterized in that The step of stitching the restored sub-images to obtain a restored target image includes: According to the initial positions of the original sub-images, the corresponding restored sub-images are spliced ​​to obtain a restored target image; wherein the overlapping areas of adjacent multiple frames of the restored sub-images display corresponding color intensity values ​​according to a preset weighting rule.

7. The method according to claim 1, characterized in that Before or after segmenting the original image, the method further includes: The resolution of the original image or the original sub-image is improved to obtain the optimized original image or the original sub-image.

8. An image generating device, characterized in that: include: An image acquisition module, used to acquire the original image to be restored; An image segmentation module is used to segment the original image to obtain multiple frames of original sub-images; An image restoration module is used to restore the original sub-images of each frame by using a diffusion model with a target style to obtain corresponding restored sub-images; The image stitching module is used to stitch the repaired sub-images to obtain a repaired target image.

9. A diffusion model training device, characterized in that: Used for training to obtain the diffusion model with the target style as described in any one of claims 1 to 7; the training device comprises: A data acquisition module, used to acquire training data, wherein the training data includes sample images with the target style and corresponding text annotations; A model training module, used to train a low-rank adaptation model based on the initial diffusion model according to the training data, and obtain a low-rank matrix corresponding to the trained low-rank adaptation model; The parameter updating module is used to add the low-rank matrix to the weight matrix corresponding to the key layer of the initial diffusion model to obtain a diffusion model with a target style.

10. An electronic device, characterized in that: include: processor; as well as A memory having executable codes stored thereon, which, when executed by the processor, causes the processor to execute the image generating method according to any one of claims 1 to 7.

11. A computer-readable storage medium having executable codes stored thereon, which, when executed by a processor of an electronic device, causes the processor to execute the image generating method according to any one of claims 1 to 7.

Images