Image processing method and device

Abstract

The invention discloses an image processing method. The method comprises the following steps: acquiring a to-be-processed image and a target specification parameter; edge region expansion is carried out on the to-be-processed image, a first image conforming to the target specification parameter is obtained, and the first image comprises the to-be-processed image and an expanded edge region; the first image and image generation prompt information are input into an image expansion model, the image expansion model generates image content for the edge area based on the to-be-processed image, the image generation prompt information comprises positive prompt information and negative prompt information, and the process of generating the image content for the edge area by the image expansion model comprises N rounds of iterative computation, in the latter N-p rounds of iterative computation of the N rounds of iterative computation, the image expansion model predicts an image generation result based on the forward prompt information; and obtaining a target image matched with the target specification parameter according to the to-be-processed image and target image content generated by the image expansion model. By using the scheme, the efficiency of converting the to-be-processed image into the target image can be improved.

Description

Technical Field

[0001] The present application relates to the field of image processing, and in particular to an image processing method and device. Background Art

[0002] In some scenarios, it is necessary to adjust an image to a certain fixed specification, such as a certain fixed aspect ratio or a certain fixed size. Currently, the image can be adjusted to the aforementioned fixed specification by directly scaling the image, or by cropping the image first and then scaling it. However, in this way, the image will be distorted after adjustment, and accordingly, the visual experience brought to the user is poor.

[0003] The inventor of the present application has found through research that, at present, there are some artificial intelligence drawing schemes based on diffusion models, which can redraw (inpainting) a part of an existing image. Specifically, it can redraw a part of an existing image in combination with the content of the existing image. When the redrawing area is the outer edge of the image, the outward expansion (outpainting) of the existing image can be achieved. In this way, the image can be adjusted to a certain fixed specification.

[0004] However, when the diffusion model is currently used to perform the outpainting task, the execution efficiency of the diffusion model is low.

[0005] Therefore, a solution is urgently needed to solve the above problems. Summary of the invention

[0006] In order to solve or at least partially solve the above technical problems, the present application provides an image processing method and device.

[0007] In a first aspect, the present application provides an image processing method, the method comprising:

[0008] Acquire an image to be processed and a target specification parameter, wherein the target specification parameter is used to indicate a specification that the image to be processed needs to meet after processing the image to be processed;

[0009] Expanding the edge region of the image to be processed to obtain a first image that meets the target specification parameters, wherein the first image includes the image to be processed and the expanded edge region;

[0010] Inputting the first image and the image generation prompt information into an image expansion model, the image expansion model generating image content for the edge area based on the image to be processed, wherein the image generation prompt information includes positive prompt information and negative prompt information, and the process of the image expansion model generating image content for the edge area includes N rounds of iterative calculations, in the first p rounds of iterative calculations of the N rounds of iterative calculations, the image expansion model predicts image generation results based on the positive prompt information and the negative prompt information, and in the last Np rounds of iterative calculations of the N rounds of iterative calculations, the image expansion model predicts image generation results based on the positive prompt information;

[0011] A target image matching the target specification parameters is obtained according to the image to be processed and the target image content generated by the image expansion model.

[0012] Optionally, the method further includes:

[0013] Obtaining word embeddings of preset image generation prompt words, wherein the image generation prompt words include positive prompt words and negative prompt words;

[0014] The word embedding of the image generation prompt word is used as the image generation prompt information, wherein the word embedding of the positive prompt word is used as the positive prompt information, and the word embedding of the negative prompt word is used as the negative prompt information.

[0015] Optionally, the image expansion model supports generating new image content based on an input image of any image resolution, and the image expansion model is obtained by training with training sample images of a preset resolution, and inputting the first image into the image expansion model comprises:

[0016] Converting the resolution of the first image to the preset resolution to obtain a first image with the preset resolution;

[0017] The first image having the preset resolution is input into the image expansion model.

[0018] Optionally, the step of expanding the edge region of the image to be processed to obtain a first image that meets the target specification parameters includes:

[0019] The edge region of the image to be processed is expanded according to the edge pixels of the image to be processed to obtain a first image that meets the target specification parameters, wherein the pixels of the edge region are obtained according to the edge pixels of the image to be processed.

[0020] Optionally, the image expansion model is a stable diffusion model to which at least one model speed-up technology is applied, and the at least one model speed-up technology includes: DeepCache technology and / or OneDiff technology.

[0021] Optionally, in the i-th round of iterative calculation in the first p rounds of iterative calculation in the N rounds of iterative calculation, the image expansion model is specifically used to:

[0022] Obtaining a first noise according to the first predicted image obtained by the iterative calculation of the i-1th round and the positive prompt information, and obtaining a second noise according to the first predicted image and the negative prompt information; i is an integer greater than or equal to 2;

[0023] Denoising the first predicted image according to the difference between the first noise and the second noise to obtain the second predicted image obtained by the i-th round of iterative calculation;

[0024] In the first round of calculation in the N rounds of iterations, the image expansion model is specifically used to:

[0025] Obtaining third noise according to the first image superimposed with input noise and the positive prompt information, and obtaining fourth noise according to the first image superimposed with input noise and the negative prompt information;

[0026] The first image superimposed with the input noise is denoised according to the difference between the third noise and the fourth noise to obtain the predicted image obtained by the first round of iterative calculation.

[0027] Optionally, in the j-th round of iterative calculation in the Np-th round of iterative calculation, the image expansion model is specifically used to:

[0028] A fifth noise is obtained according to the third predicted image obtained by the j-1th round of iterative calculation and the positive prompt information; j is an integer greater than or equal to 2;

[0029] Denoising the third predicted image according to the fifth noise to obtain a fourth predicted image obtained by the j-th round of iterative calculation;

[0030] In the first round of calculation in the subsequent Np rounds of iterations, the image expansion model is specifically used to:

[0031] Obtaining sixth noise according to the second predicted image obtained by the p-th round of iterative calculation and the positive prompt information;

[0032] The second predicted image obtained by the p-th round of iterative calculation is denoised according to the sixth noise to obtain the predicted image obtained by the first round of iterative calculation in the subsequent Np rounds of iteration.

[0033] In a second aspect, the present application provides an image processing device, the device comprising:

[0034] A first acquisition unit is used to acquire an image to be processed and a target specification parameter, wherein the target specification parameter is used to indicate a specification that the image to be processed needs to meet after processing the image to be processed;

[0035] An edge expansion unit, configured to expand the edge region of the image to be processed to obtain a first image that meets the target specification parameters, wherein the first image includes the image to be processed and the expanded edge region;

[0036] An input unit, used for inputting the first image and image generation prompt information into an image expansion model, wherein the image expansion model generates image content for the edge area based on the image to be processed, wherein the image generation prompt information includes positive prompt information and negative prompt information, and the process of the image expansion model generating image content for the edge area includes N rounds of iterative calculations, in which the image expansion model predicts image generation results based on the positive prompt information and the negative prompt information in the first p rounds of iterative calculations of the N rounds of iterative calculations, and in which the image expansion model predicts image generation results based on the positive prompt information and the negative prompt information in the last Np rounds of iterative calculations of the N rounds of iterative calculations,

[0037] The first determining unit is used to obtain a target image matching the target specification parameters according to the image to be processed and the target image content generated by the image expansion model.

[0038] Optionally, the device further comprises:

[0039] A second acquisition unit is used to acquire word embeddings of preset image generation prompt words, wherein the image generation prompt words include positive prompt words and negative prompt words;

[0040] The second determining unit is used to use the word embedding of the image generation prompt word as the image generation prompt information, wherein the word embedding of the positive prompt word is used as the positive prompt information, and the word embedding of the negative prompt word is used as the negative prompt information.

[0041] Optionally, the image expansion model supports generating new image content based on an input image of any image resolution, the image expansion model is obtained by training with training sample images of a preset resolution, and the input unit is used to:

[0042] Converting the resolution of the first image to the preset resolution to obtain a first image with the preset resolution;

[0043] The first image having the preset resolution is input into the image expansion model.

[0044] Optionally, the edge expansion unit is used to:

[0045] The edge region of the image to be processed is expanded according to the edge pixels of the image to be processed to obtain a first image that meets the target specification parameters, wherein the pixels of the edge region are obtained according to the edge pixels of the image to be processed.

[0046] Optionally, the image expansion model is a stable diffusion model to which at least one model speed-up technology is applied, and the at least one model speed-up technology includes: DeepCache technology and / or OneDiff technology.

[0047] Optionally, in the i-th round of iterative calculation in the first p rounds of iterative calculation in the N rounds of iterative calculation, the image expansion model is specifically used to:

[0048] Obtaining a first noise according to the first predicted image obtained by the iterative calculation of the i-1th round and the positive prompt information, and obtaining a second noise according to the first predicted image and the negative prompt information; i is an integer greater than or equal to 2;

[0049] Denoising the first predicted image according to the difference between the first noise and the second noise to obtain the second predicted image obtained by the i-th round of iterative calculation;

[0050] In the first round of calculation in the N rounds of iterations, the image expansion model is specifically used to:

[0051] Obtaining third noise according to the first image superimposed with input noise and the positive prompt information, and obtaining fourth noise according to the first image superimposed with input noise and the negative prompt information;

[0052] The first image superimposed with the input noise is denoised according to the difference between the third noise and the fourth noise to obtain the predicted image obtained by the first round of iterative calculation.

[0053] Optionally, in the j-th round of iterative calculation in the Np-th round of iterative calculation, the image expansion model is specifically used to:

[0054] A fifth noise is obtained according to the third predicted image obtained by the j-1th round of iterative calculation and the positive prompt information; j is an integer greater than or equal to 2;

[0055] Denoising the third predicted image according to the fifth noise to obtain a fourth predicted image obtained by the j-th round of iterative calculation;

[0056] In the first round of calculation in the subsequent Np rounds of iterations, the image expansion model is specifically used to:

[0057] Obtaining sixth noise according to the second predicted image obtained by the p-th round of iterative calculation and the positive prompt information;

[0058] The second predicted image obtained by the p-th round of iterative calculation is denoised according to the sixth noise to obtain the predicted image obtained by the first round of iterative calculation in the subsequent Np rounds of iteration.

[0059] In a third aspect, an embodiment of the present application provides an electronic device, the electronic device comprising a processor and a memory;

[0060] The processor is used to execute instructions stored in the memory so that the electronic device performs the method as described in any one of the first aspects above.

[0061] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, comprising instructions, wherein the instructions instruct a device to execute a method as described in any one of the above first aspects.

[0062] In a fifth aspect, an embodiment of the present application provides a computer program product, which, when executed on a computer, enables the computer to execute any of the methods described in the first aspect above.

[0063] Compared with the prior art, the embodiments of the present application have the following advantages:

[0064] The present application provides an image processing method, which includes: obtaining an image to be processed and a target specification parameter, wherein the target specification parameter is used to indicate the specification that the image to be processed needs to meet after the image to be processed is processed. Then, the edge region of the image to be processed is expanded to obtain a first image that meets the target specification parameter. The first image includes the image to be processed and the expanded edge region. Further, the first image and the image generation prompt information can be input into an image expansion model, and the image expansion model generates image content for the edge region based on the image to be processed. Further, according to the image to be processed and the target image content generated by the image expansion model, a target image matching the target specification parameter is obtained. In the present application, the aforementioned image generation prompt information includes positive prompt information and negative prompt information, and the process of the image expansion model generating image content for the edge region includes N rounds of iterative calculations, and in the first p rounds of iterative calculations of the N rounds of iterative calculations, the image expansion model predicts the image generation result based on the positive prompt information and the negative prompt information, and in the last Np rounds of iterative calculations of the N rounds of iterative calculations, the image expansion model predicts the image generation result based on the positive prompt information. Compared with the traditional image expansion model, each round of iterative calculation requires the combination of positive prompt information and the negative prompt information. In the present application, in the subsequent Np rounds of iterative calculation, the image expansion model does not need to combine negative prompt information to predict the image generation result, thereby reducing the amount of calculation of the image expansion model, saving computing resources, and improving the image generation efficiency of the image expansion model. That is, using this solution, the efficiency of converting the image to be processed into the target image can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0066] Figure 1 A flowchart of an image processing method provided in an embodiment of the present application;

[0067] Figure 2 A schematic diagram of the structure of an image processing device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0068] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0069] Before introducing the solution provided by this application, the technology that may be involved in this application is first introduced.

[0070] Diffusion model: The reasoning of the diffusion model is a step-by-step denoising process. The noise will be predicted by U-Net and gradually denoised through multiple steps of iterative calculation to form the final output image.

[0071] Hint words: After being encoded into corresponding word embeddings, the hint words can be used as control conditions for the cross-attention mechanism in U-Net to control the distribution of noise predictions to move toward the text hint words. Hint words include positive hint words and negative hint words. Positive hint words are used to indicate the conditions that the generated image must meet, while negative hint words are used to indicate the content that the generated image should avoid.

[0072] U-Net: It uses an encoder-decoder structure, and its reasoning process goes through the process of latent variable downsampling and re-upsampling.

[0073] Stable diffusion extra large (SDXL) model: is an image generation model that can generate images based on text. It can excellently generate high-definition and realistic images with an image resolution of 1024*1024.

[0074] Classifier-free guidance: is a technique used in computer vision and machine learning that can guide the learning of the model by generating adversarial examples without a classifier.

[0075] DeepCache: A model speed-up technology that can be applied to diffusion models (such as SDXL models). It is observed that different denoising stages can appropriately cache some downsampled intermediate results for direct use in subsequent stages, thereby avoiding repeated calculations to improve the inference speed of the diffusion model. However, this solution will have a certain impact on the image quality of the image content generated by the diffusion model.

[0076] OneDiff: An open source acceleration library for diffusion models, which uses code compilation tools for diffusion models and powerful optimized graphics processing unit (GPU) kernels to accelerate inference without affecting the inference results.

[0077] Various non-limiting implementations of the present application are described in detail below in conjunction with the accompanying drawings.

[0078] Exemplary Methods

[0079] See also Figure 1 , which is a flow chart of an image processing method provided in an embodiment of the present application. Figure 1 The method shown can be applied to the client or the server. In the following examples, the method is applied to the client as an example for explanation.

[0080] Figure 1 The method shown may, for example, include the following steps: S101 - S104 .

[0081] S101: Acquire an image to be processed and target specification parameters, where the target specification parameters are used to indicate specifications that the image to be processed needs to meet after processing the image to be processed.

[0082] In one example, the target specification parameter may be a ratio of image length to image width, for example, 4:3. In another example, the target specification parameter may be a specific image size, in which case the target specification parameter may include image length and image width.

[0083] In one example, the target specification parameter may be input by a user. For example, the page displayed by the client may include an image specification parameter input area, and the user may input the target specification parameter in the image specification parameter input area.

[0084] In one example, the image to be processed may be an image input (i.e., uploaded) by a user, or an image obtained by processing an image input by a user. As a specific example, in addition to the aforementioned image specification parameter input area, the page displayed by the client may also include an image upload control, through which the user may trigger an image upload operation, thereby uploading the corresponding image.

[0085] In one implementation of the embodiment of the present application, the aforementioned image to be processed may be an image that does not include a border of a specific color (e.g., black). In one example, for an image input by a user, the client may perform border detection on the image. If it is detected that the image input by the user includes a border of a specific color, the client may perform a border cropping operation on the image input by the user, thereby obtaining an image to be processed that does not include a border of the specific color. If it is detected that the image input by the user does not include a border of the specific color, the image input by the user may be directly used as the image to be processed.

[0086] The embodiments of the present application do not specifically limit the specific implementation of border detection of an image. In one example, for example, a border detection model can be used to detect the border of the image. In addition, after detecting that the image includes a border, for example, the color of the border can be determined. If the border is a solid color and the color is the aforementioned specific color, it can be determined that the image includes a border of the aforementioned specific color.

[0087] S102: Expanding the edge region of the image to be processed to obtain a first image that meets the target specification parameters, wherein the first image includes the image to be processed and the expanded edge region.

[0088] After obtaining the image to be processed, the image to be processed is further processed to obtain a target image that meets the target specification parameters. Specifically, the edge area of ​​the image to be processed can be expanded so that the expanded image meets the target specification parameters, and in order to enable the expanded image to bring a better visual experience, new image content can also be generated for the expanded edge area based on the image to be processed.

[0089] In one example, based on the size of the image to be processed and the target specification parameters, the size of the edge area that needs to be expanded by the image to be processed to meet the target specification parameters can be determined, and then, based on the determined size of the edge area, the edge area of ​​the image to be processed is expanded, so as to obtain a first image that meets the target specification parameters. It is not difficult to understand that the first image includes the image to be processed and the expanded edge area. S103: Input the first image and the image generation prompt information into the image expansion model, and the image expansion model generates image content for the edge area based on the image to be processed, wherein the image generation prompt information includes positive prompt information and negative prompt information, and the process of the image expansion model generating image content for the edge area includes N rounds of iterative calculations, in the first p rounds of iterative calculations of the N rounds of iterative calculations, the image expansion model predicts the image generation result based on the positive prompt information and the negative prompt information, and in the last Np rounds of iterative calculations of the N rounds of iterative calculations, the image expansion model predicts the image generation result based on the positive prompt information.

[0090] After obtaining the first image, in order to make the image content of the expanded edge area included in the first image and the image content of the image to be processed more coordinated, thereby improving the visual experience brought to the user by the entire image, the first image and the image generation prompt information can be input into the image expansion model.

[0091] The image expansion model mentioned here may be a stable diffusion model, for example, an SDXL model. The image expansion model may be used to generate new image content based on existing image content and image generation prompt information. Therefore, after the first image and the image generation prompt information are input into the image expansion model, the image expansion model may generate new image content for the aforementioned edge region included in the first image based on the image content of the image to be processed and the image generation prompt information.

[0092] The process of the image expansion model generating image content for the edge area includes N rounds of iterative calculations. Wherein N is an integer greater than or equal to 2, for example, N is 100. In a specific example, the image expansion model may include U-Net, which is used to predict the image generation result, wherein the image generation result predicted by U-Net may include N rounds of iterative calculations, and the image generation result predicted by the last round of iterative calculations can be used as the output result of the image expansion model. In traditional technology, for each round of iterative calculations in the N rounds of iterative calculations, U-Net needs to perform calculations based on positive prompt information and negative prompt information respectively, which results in a large amount of calculation for the image expansion model to generate images for the edge area, and correspondingly, the efficiency is low, and more computing resources are consumed.

[0093] In an embodiment of the present application, considering that the image generation result predicted by U-Net already includes part of the image content when several rounds of the aforementioned N rounds of iteration have been executed, in this case, the aforementioned negative prompt information has limited contribution to the generation of new image content. Therefore, in the present application, for the N rounds of iteration, the image expansion model (specifically, it can be the U-Net in the image expansion model) predicts the image generation result based on the positive prompt information and the negative prompt information in the first p rounds of iterative calculations of the N rounds of iterative calculations. Wherein, p is an integer greater than or equal to 1 and less than N. In a specific example, in the first p rounds of iterative calculations, the traditional classifier-free guidance technology can be applied, specifically:

[0094] For the first round of iterative calculations in the first p rounds of iterative calculations, the image expansion model can calculate the third noise based on the first image with the input noise superimposed and the positive prompt information, and calculate the fourth noise based on the first image with the input noise superimposed and the negative positive prompt information. Further, the difference between the third noise and the fourth noise can be calculated to obtain the first difference. After obtaining the first difference, the first difference can be used to denoise the first image with the input noise superimposed to obtain the predicted image obtained by the first round of iterative calculation. Regarding the specific implementation of the image expansion model calculating the third noise, the fourth noise, and denoising the first image with the input noise superimposed based on the first difference, it can be similar to the principle of the traditional classifier-free guidance technology, and will not be repeated here. Among them, the first image with the input noise superimposed can be, for example, the input noise superimposed on the edge area of ​​the first image, and the area where the image to be processed in the first image is located is not superimposed with noise. The input noise mentioned here can be a preset noise generated by the image expansion model, such as Gaussian noise. The input noise is used as one of the inputs of U-Net.

[0095] Correspondingly, for the i-th (i is an integer greater than or equal to 2) round of iterative calculation in the first p rounds of iterative calculation, the image expansion model can obtain the first noise based on the first predicted image obtained by the i-1th round of iterative calculation and the positive prompt information, and obtain the second noise based on the first predicted image and the negative prompt information. Furthermore, the difference between the first noise and the second noise can be calculated to obtain the second difference. After obtaining the second difference, the first predicted image can be denoised using the second difference to obtain the second predicted image obtained by the i-th round of iterative calculation. Regarding the specific implementation of the image expansion model calculating the first noise, the second noise, and denoising the first predicted image based on the second difference, it can be similar to the principle of the traditional classifier-free guidance technology, and will not be repeated here.

[0096] In the Np rounds of iterative calculations after the N rounds of iterative calculations, the image expansion model predicts the image generation result based on the positive prompt information. That is to say, in the Np rounds of iterative calculations after the traditional classifier-free guidance technology is no longer applied, and the U-Net no longer needs to perform corresponding calculations based on negative prompt information, thereby reducing the amount of calculation of the U-Net and correspondingly improving the efficiency of the image expansion model in generating image content. Specifically:

[0097] For the first round of iterative calculation in the latter Np rounds of iteration, the image expansion model can calculate the sixth noise based on the second predicted image obtained by the p-th round of iterative calculation and the positive prompt information. After obtaining the sixth noise, the second predicted image obtained by the p-th round of iterative calculation can be denoised based on the sixth noise to obtain the predicted image obtained by the first round of iterative calculation in the latter Np rounds of iteration. In this process, the image expansion model no longer needs to predict the noise based on the negative prompt information, nor does it need to calculate the difference between the sixth noise and the noise predicted based on the negative prompt information.

[0098] For the jth (j is an integer greater than or equal to 2) round of calculation in the latter Np rounds of iteration, the image expansion model can calculate the fifth noise based on the third predicted image obtained by the j-1th round of iterative calculation and the positive prompt information. After obtaining the fifth noise, the third predicted image can be denoised based on the fifth noise to obtain the fourth predicted image obtained by the jth round of iterative calculation. In this process, the image expansion model does not need to predict the noise based on the negative prompt information, nor does it need to calculate the difference between the fifth noise and the noise predicted based on the negative prompt information.

[0099] Regarding the image generation prompt information, it should be noted that:

[0100] In one example, the image generation prompt information may be an image generation prompt word. The positive prompt information may be a positive prompt word, and the negative prompt information may be a negative prompt word. For this case, the aforementioned image expansion model may also include a text encoder for converting the positive prompt word and the negative prompt word into corresponding word embeddings, respectively, so that U-Net generates image content based on the word embeddings. In one example, the positive prompt word and the negative prompt word may be input by a user through a prompt word input entrance.

[0101] In another example, the positive prompt information may be a word embedding of a positive prompt word, and the negative prompt information may be a word embedding of a negative prompt word. Furthermore, the word embedding of the positive prompt word and the word embedding of the negative prompt word may be preset. In other words, the client may obtain the word embedding of the preset image generation prompt word, and the image generation prompt word includes a positive prompt word and a negative prompt word. Further, the word embedding of the image generation prompt word is used as the image generation prompt information, wherein the word embedding of the positive prompt word is used as the positive prompt information, and the word embedding of the negative prompt word is used as the negative prompt information. In this way, since the word embedding of the positive prompt word and the word embedding of the negative prompt word can be directly obtained, the image expansion model may not need to include a text encoder, and accordingly, the computing resources consumed by the text encoder to perform corresponding operations can be saved. Moreover, since there is no need for the text encoder to perform corresponding operations, the efficiency of the image expansion model in generating image content is also improved.

[0102] In one example, the aforementioned image expansion model may be obtained by pre-training. Specifically, the image expansion model is obtained by training with training sample images of a preset resolution. The embodiment of the present application does not specifically limit the specific training method of the image expansion model. The embodiment of the present application also does not specifically limit the preset resolution. The preset resolution may be, for example, 1024*1024. For example, when the image expansion model is an SDXL model, the preset resolution is 1024*1024.

[0103] The aforementioned image expansion model supports generating new image content based on an input image of any image resolution. In other words, no matter what resolution the first image has, the image expansion model can generate image content for the edge region of the first image. Therefore, in one example, after obtaining the first image based on the image to be processed, the first image can be directly input into the image expansion model.

[0104] In addition, considering that the image expansion model is trained based on training sample images of a preset resolution, if the resolution of the image input to the image expansion model is the preset resolution, the higher the quality of the new image content generated by the image expansion model for the input image. In view of this, in order to improve the quality of the image content generated for the aforementioned edge area, after obtaining the first image, the resolution of the first image can be converted to the preset resolution to obtain a first image with a resolution of the preset resolution. Further, the first image with a resolution of the preset resolution is input to the image expansion model.

[0105] In addition, in order to further improve the efficiency of the image expansion model in generating new image content for the aforementioned edge area, at least one model speed-up technology can also be applied to the image expansion model. The embodiment of the present application does not specifically limit the at least one model speed-up technology, and the at least one model speed-up technology may include, for example, DeepCache technology and / or OneDiff technology. For example, DeepCache technology and OneDiff technology can both be applied to the image expansion model. By adopting this solution, the model speed-up technology originally applied to "generating new images based on text" can be applied to the image expansion model that can "generate new images based on images", thereby effectively improving the efficiency of the image expansion model in generating image content for the aforementioned edge area. Regarding DeepCache technology and OneDiff technology, its overall technical framework can refer to the relevant description of DeepCache technology and OneDiff technology traditionally applied to "generating new images based on text", and no repeated description will be made here.

[0106] S104: Obtaining a target image matching the target specification parameters according to the image to be processed and the target image content generated by the image expansion model.

[0107] For the convenience of description, the image content generated by the image expansion model for the aforementioned edge region included in the first image is referred to as "target image content". Then:

[0108] After obtaining the target image content, the target image can be obtained based on the image to be processed and the target image content. The specifications of the target image are the same as those of the first image, and the difference between the two is that the content of the edge area in the target image is different from the content of the edge area in the first image. Therefore, the target image is also an image that meets the aforementioned target specification parameters.

[0109] In one example, in order to ensure that the obtained target image can bring a better visual experience to the user, after obtaining the target image content, the target image content can be checked to obtain the check result, and further, based on the check result, it is determined how to obtain the target image based on the aforementioned target image content and the image to be processed. In one example, multiple conditions can be pre-set, and after obtaining the check result, it is determined how to obtain the target image based on the aforementioned target image content and the image to be processed according to the conditions matched by the check result.

[0110] In one example, the aforementioned multiple conditions may include a first condition and a second condition, wherein the strictness of the first condition is higher than the strictness of the second condition. For this case:

[0111] If the aforementioned inspection result matches the first condition, it means that the target image content meets the requirements, and the new image composed of the target image and the image to be processed will not bring visual impact to the user. In this case, the target image including the image to be processed and the target image content can be directly obtained based on the target image content and the image to be processed. Among them, the aforementioned inspection result matches the first condition, which can be understood as the inspection result indicating that the target image content meets the first condition.

[0112] If the aforementioned inspection result matches the second condition, it means that the target image content may not meet the requirements, and the new image composed of the target image and the image to be processed may bring visual impact to the user. In this case, the target image content can be first blurred, for example, Gaussian blurring the target image content. Further, based on the image to be processed and the target image content after blurring, a target image including the image to be processed and the target image content after blurring is obtained. Among them, the aforementioned inspection result matches the second condition, which can be understood as the inspection result indicating that the target image content meets the second condition.

[0113] The embodiments of the present application do not specifically limit the aforementioned multiple conditions. In one example, considering that if the newly generated image content (i.e., the target image content) includes a new subject, it may bring a bad visual experience to the user. Therefore, the aforementioned multiple conditions may be related to whether the target image content includes a subject. In this case, the aforementioned content check of the target image content may be to detect the target image content using at least one subject detection model to determine whether the target image content includes a subject. In one example, the image composed of the image to be processed and the target image content can be directly input into at least one subject detection model, respectively, to detect whether the target image content includes a subject.

[0114] When the multiple conditions are related to whether the target image content includes a subject, in one example, the aforementioned first condition, for example, may be that the target image content does not include a subject. Correspondingly, the second condition may be that the target image content includes a small part of a complete subject. Specifically, the second condition may be that the content of the first area in the target image content and the content of the second area in the image to be processed constitute a subject, and the ratio of the first area to the second area is less than or equal to the first ratio threshold, that is, the first area is smaller than the second area. Among them, the first ratio threshold may be a value less than 1, that is, in the present application: in the case where the target image content does not include a subject, the target image can be obtained directly based on the target image content and the image to be processed. In the case where the target image content includes only a small part of the complete subject, the target image content can be blurred, and the target image can be obtained based on the image to be processed and the target image content after the blurred processing, so as to minimize the visual impact on the user.

[0115] In one example, the at least one subject detection model may include, for example, one or more of a face detection model, a body detection model, and a text detection model.

[0116] The face detection model is used to detect whether an image includes a face. It can output the coordinates of the vertices (such as the vertices of a rectangular box) of the area where the detected face is located, and the probability that the area is a face.

[0117] The human body detection model is used to detect whether an image includes a human body. It can output the coordinates of multiple points (such as vertices of a rectangular box) in the area where the detected human body is located, as well as the probability that the area is a human body.

[0118] The text detection model is used to detect whether an image includes text. It can output the coordinates of the vertices (such as the vertices of a rectangular box) of the area where the detected text is located, and the probability that the area contains text.

[0119] From the above description, it can be seen that, using the solution of the embodiment of the present application, compared with the traditional image expansion model, each round of iterative calculation requires the combination of positive prompt information and the negative prompt information. In the present application, in the subsequent Np rounds of iterative calculation, the image expansion model does not need to combine negative prompt information to predict the image generation result, thereby reducing the amount of calculation of the image expansion model, saving computing resources, and improving the image generation efficiency of the image expansion model. That is, using this solution, the efficiency of converting the image to be processed into the target image can be improved.

[0120] In one example, when expanding the edge region of the image to be processed, the edge region may be pre-filled with certain content so that the image expansion model can generate new image content for the edge region based on the image content of the image to be processed and the content of the edge region. In this way, the efficiency of generating new image content for the edge region can be improved compared to the case where the edge region only includes noise.

[0121] In one example, the edge region may be filled with a solid color, or filled with a lama model. In another example, the edge region may be filled with a mirror image based on the content of the image to be processed.

[0122] The inventor of the present application found that using the lama model to fill the edge area requires additional computing resources. The use of mirror filling will cause the image content ultimately generated by the image expansion model for the edge area to include some strange content (such as strange text, strange faces, human bodies, etc.). If the edge pixels of the image to be processed are used to fill the edge area, the image content ultimately generated by the image expansion model for the edge area will have a higher quality. Moreover, the aforementioned edge area is filled based on the edge pixels of the image to be processed, and the computing resources consumed are relatively small. Therefore, in another example, when S102 is specifically implemented, the edge area of ​​the image to be processed can be expanded according to the edge pixels of the image to be processed. Specifically, the edge pixels of the image to be processed can be used to fill the edge area, thereby obtaining a first image.

[0123] Exemplary Devices

[0124] See also Figure 2 , Figure 2 A schematic diagram of the structure of an image processing device provided in an embodiment of the present application. Figure 2 The image processing device 200 shown can be used to execute the image processing method provided by the above method embodiment. The device 200 can specifically include: a first acquisition unit 201, an edge expansion unit 202, an input unit 203 and a first determination unit 204.

[0125] A first acquisition unit 201 is used to acquire an image to be processed and a target specification parameter, wherein the target specification parameter is used to indicate a specification that the image to be processed needs to meet after processing the image to be processed;

[0126] An edge expansion unit 202 is used to expand the edge area of ​​the image to be processed to obtain a first image that meets the target specification parameters, wherein the first image includes the image to be processed and the expanded edge area;

[0127] An input unit 203 is used to input the first image and the image generation prompt information into an image expansion model, and the image expansion model generates image content for the edge area based on the image to be processed, wherein the image generation prompt information includes positive prompt information and negative prompt information, and the process of the image expansion model generating image content for the edge area includes N rounds of iterative calculations, in the first p rounds of iterative calculations of the N rounds of iterative calculations, the image expansion model predicts the image generation result based on the positive prompt information and the negative prompt information, and in the last Np rounds of iterative calculations of the N rounds of iterative calculations, the image expansion model predicts the image generation result based on the positive prompt information;

[0128] The first determining unit 204 is configured to obtain a target image matching the target specification parameters according to the image to be processed and the target image content generated by the image expansion model.

[0129] Optionally, the device further comprises:

[0130] A second acquisition unit is used to acquire word embeddings of preset image generation prompt words, wherein the image generation prompt words include positive prompt words and negative prompt words;

[0131] The second determining unit is used to use the word embedding of the image generation prompt word as the image generation prompt information, wherein the word embedding of the positive prompt word is used as the positive prompt information, and the word embedding of the negative prompt word is used as the negative prompt information.

[0132] Optionally, the image expansion model supports generating new image content based on an input image of any image resolution, the image expansion model is obtained by training with training sample images of a preset resolution, and the input unit 203 is used to:

[0133] Converting the resolution of the first image to the preset resolution to obtain a first image with the preset resolution;

[0134] The first image having the preset resolution is input into the image expansion model.

[0135] Optionally, the edge expansion unit 202 is used to:

[0136] The edge region of the image to be processed is expanded according to the edge pixels of the image to be processed to obtain a first image that meets the target specification parameters, wherein the pixels of the edge region are obtained according to the edge pixels of the image to be processed.

[0137] Optionally, the image expansion model is a stable diffusion model to which at least one model speed-up technology is applied, and the at least one model speed-up technology includes: DeepCache technology and / or OneDiff technology.

[0138] Optionally, in the i-th round of iterative calculation in the first p rounds of iterative calculation in the N rounds of iterative calculation, the image expansion model is specifically used to:

[0139] Obtaining a first noise according to the first predicted image obtained by the iterative calculation of the i-1th round and the positive prompt information, and obtaining a second noise according to the first predicted image and the negative prompt information; i is an integer greater than or equal to 2;

[0140] Denoising the first predicted image according to the difference between the first noise and the second noise to obtain the second predicted image obtained by the i-th round of iterative calculation;

[0141] In the first round of calculation in the N rounds of iterations, the image expansion model is specifically used to:

[0142] Obtaining third noise according to the first image superimposed with input noise and the positive prompt information, and obtaining fourth noise according to the first image superimposed with input noise and the negative prompt information;

[0143] The first image superimposed with the input noise is denoised according to the difference between the third noise and the fourth noise to obtain the predicted image obtained by the first round of iterative calculation.

[0144] Optionally, in the j-th round of iterative calculation in the Np-th round of iterative calculation, the image expansion model is specifically used to:

[0145] A fifth noise is obtained according to the third predicted image obtained by the j-1th round of iterative calculation and the positive prompt information; j is an integer greater than or equal to 2;

[0146] Denoising the third predicted image according to the fifth noise to obtain a fourth predicted image obtained by the j-th round of iterative calculation;

[0147] In the first round of calculation in the subsequent Np rounds of iterations, the image expansion model is specifically used to:

[0148] Obtaining sixth noise according to the second predicted image obtained by the p-th round of iterative calculation and the positive prompt information;

[0149] The second predicted image obtained by the p-th round of iterative calculation is denoised according to the sixth noise to obtain the predicted image obtained by the first round of iterative calculation in the subsequent Np rounds of iteration.

[0150] Since the device 200 is a device corresponding to the image processing method provided by the above method embodiment, the specific implementation of each unit of the device 200 is based on the same concept as the above method embodiment. Therefore, regarding the specific implementation of each unit of the device 200, please refer to the relevant description part of the above method embodiment, which will not be repeated here.

[0151] The embodiment of the present application also provides an electronic device, the electronic device comprising a processor and a memory;

[0152] The processor is used to execute the instructions stored in the memory, so that the electronic device executes the image processing method provided by the above method embodiment.

[0153] An embodiment of the present application provides a computer-readable storage medium, including instructions, wherein the instructions instruct a device to execute the image processing method provided by the above method embodiment.

[0154] The embodiment of the present application also provides a computer program product. When the computer program product is run on a computer, the computer is enabled to execute the image processing method provided by the above method embodiment.

[0155] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the invention disclosed herein. The present application is intended to cover any modification, use or adaptation of the present application, which follows the general principles of the present application and includes common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present application are indicated by the following claims.

[0156] It should be understood that the present application is not limited to the precise structures described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

[0157] The above description is only a preferred embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application should be included in the protection scope of the present application.

Claims

1. An image processing method, characterized in that: The method comprises: Acquire an image to be processed and a target specification parameter, wherein the target specification parameter is used to indicate a specification that the image to be processed needs to meet after processing the image to be processed; Expanding the edge region of the image to be processed to obtain a first image that meets the target specification parameters, wherein the first image includes the image to be processed and the expanded edge region; Inputting the first image and the image generation prompt information into an image expansion model, the image expansion model generating image content for the edge area based on the image to be processed, wherein the image generation prompt information includes positive prompt information and negative prompt information, and the process of the image expansion model generating image content for the edge area includes N rounds of iterative calculations, in the first p rounds of iterative calculations of the N rounds of iterative calculations, the image expansion model predicts image generation results based on the positive prompt information and the negative prompt information, and in the last Np rounds of iterative calculations of the N rounds of iterative calculations, the image expansion model predicts image generation results based on the positive prompt information; A target image matching the target specification parameters is obtained according to the image to be processed and the target image content generated by the image expansion model.

2. The method according to claim 1, characterized in that The method further comprises: Obtaining word embeddings of preset image generation prompt words, wherein the image generation prompt words include positive prompt words and negative prompt words; The word embedding of the image generation prompt word is used as the image generation prompt information, wherein the word embedding of the positive prompt word is used as the positive prompt information, and the word embedding of the negative prompt word is used as the negative prompt information.

3. The method according to claim 1, characterized in that The image expansion model supports generating new image content based on an input image of any image resolution, the image expansion model is obtained by training with a training sample image of a preset resolution, and the first image is input into the image expansion model, comprising: Converting the resolution of the first image to the preset resolution to obtain a first image with the preset resolution; The first image having the preset resolution is input into the image expansion model.

4. The method according to claim 1, characterized in that The step of expanding the edge region of the image to be processed to obtain a first image that meets the target specification parameters includes: The edge region of the image to be processed is expanded according to the edge pixels of the image to be processed to obtain a first image that meets the target specification parameters, wherein the pixels of the edge region are obtained according to the edge pixels of the image to be processed.

5. The method according to any one of claims 1 to 4, characterized in that: The image expansion model is a stable diffusion model to which at least one model speed-up technology is applied. The at least one model speed-up technology includes: DeepCache technology and / or OneDiff technology.

6. The method according to any one of claims 1 to 4, characterized in that: In the i-th round of iterative calculation in the first p rounds of iterative calculation in the N rounds of iterative calculation, the image expansion model is specifically used to: Obtaining a first noise according to the first predicted image obtained by the iterative calculation of the i-1th round and the positive prompt information, and obtaining a second noise according to the first predicted image and the negative prompt information; i is an integer greater than or equal to 2; Denoising the first predicted image according to the difference between the first noise and the second noise to obtain the second predicted image obtained by the i-th round of iterative calculation; In the first round of calculation in the N rounds of iterations, the image expansion model is specifically used to: Obtaining third noise according to the first image superimposed with input noise and the positive prompt information, and obtaining fourth noise according to the first image superimposed with input noise and the negative prompt information; The first image superimposed with the input noise is denoised according to the difference between the third noise and the fourth noise to obtain the predicted image obtained by the first round of iterative calculation.

7. The method according to claim 6, characterized in that In the j-th round of iterative calculation in the Np-th round of iterative calculation, the image expansion model is specifically used for: A fifth noise is obtained according to the third predicted image obtained by the j-1th round of iterative calculation and the positive prompt information; j is an integer greater than or equal to 2; Denoising the third predicted image according to the fifth noise to obtain a fourth predicted image obtained by the j-th round of iterative calculation; In the first round of calculation in the subsequent Np rounds of iterations, the image expansion model is specifically used to: Obtaining sixth noise according to the second predicted image obtained by the p-th round of iterative calculation and the positive prompt information; The second predicted image obtained by the p-th round of iterative calculation is denoised according to the sixth noise to obtain the predicted image obtained by the first round of iterative calculation in the subsequent Np rounds of iteration.

8. An image processing device, characterized in that: The device comprises: A first acquisition unit, used for acquiring an image to be processed and a target specification parameter, wherein the target specification parameter is used for indicating a specification that the image to be processed needs to meet after processing the image to be processed; An edge expansion unit, configured to expand the edge region of the image to be processed to obtain a first image that meets the target specification parameters, wherein the first image includes the image to be processed and the expanded edge region; An input unit, used for inputting the first image and image generation prompt information into an image expansion model, wherein the image expansion model generates image content for the edge area based on the image to be processed, wherein the image generation prompt information includes positive prompt information and negative prompt information, and the process of the image expansion model generating image content for the edge area includes N rounds of iterative calculations, in which the image expansion model predicts image generation results based on the positive prompt information and the negative prompt information in the first p rounds of iterative calculations of the N rounds of iterative calculations, and in which the image expansion model predicts image generation results based on the positive prompt information and the negative prompt information in the last Np rounds of iterative calculations of the N rounds of iterative calculations, The first determining unit is used to obtain a target image matching the target specification parameters according to the image to be processed and the target image content generated by the image expansion model.

9. An electronic device, characterized in that: The electronic device comprises a processor and a memory; The processor is used to execute instructions stored in the memory so that the electronic device executes the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The method comprises instructions, wherein the instructions instruct a device to execute the method according to any one of claims 1 to 7.

Images