Image processing method and device

Abstract

The invention discloses an image processing method and device, and belongs to the technical field of electronic equipment.The method comprises the steps that a first image and a second image are obtained, the first image is an image obtained after down-sampling is conducted on an original image, and the second image is an image obtained after the first image is processed through a first network model; the first network model is used for adjusting the brightness and color of the first image and outputting the adjusted second image; the first image and the second image are input into a second network model to be processed, a third image is obtained, the second network model is used for correcting the texture of the second image to be consistent with the texture of the first image, and the corrected third image is output; and generating a target image corresponding to the original image based on the third image.

Description

Technical Field

[0001] The present application belongs to the technical field of electronic equipment, and specifically relates to an image processing method and a device thereof. Background Art

[0002] The Stable Diffusion Model is a multimodal text-generated image (abbreviated as "text-generated image") model. It introduces text (text prompt) to guide and control the diffusion process based on the conventional diffusion model. At the same time, with the introduction of the variational autoencoder (VAE), the number of pixels involved in the diffusion process of the stable diffusion model has been greatly reduced, thereby accelerating the training and reasoning process of the network, and the application of text-generated images has achieved a breakthrough. Although the original stable diffusion model is a text-generated image application, a large number of image-to-image (abbreviated as "image-to-image") conversion methods based on the stable diffusion model have been proposed in subsequent studies. The automatic brightness and color adjustment of the image-to-image application is an important application of the image-to-image conversion method. The user provides an input image, and the brightness and color adjustment network based on the stable diffusion model can automatically output an image that meets the user's specific brightness and color conditions.

[0003] However, since the entire diffusion process in the stable diffusion model is carried out in the VAE domain, which is an image compression domain, there will be a random sampling process in the process of restoring the VAE domain to the image domain. The brightness and color adjustment network based on the stable diffusion model will inevitably face the problem of image texture change loss, resulting in the final output image having poor texture although the brightness and color meet the conditions. Therefore, it is necessary to provide an image processing method to correct the texture of the image output by the brightness and color adjustment network, thereby improving the scope of application of image-to-image applications based on the stable diffusion model. Summary of the invention

[0004] The purpose of the embodiments of the present application is to provide an image processing method and device thereof, which can correct the texture of an image output by a brightness and color adjustment network, thereby improving the scope of use of image-to-image applications based on a stable diffusion model.

[0005] In a first aspect, an embodiment of the present application provides an image processing method, the method comprising:

[0006] Acquire a first image and a second image, wherein the first image is an image obtained by downsampling the original image, and the second image is an image obtained by processing the first image through a first network model, and the first network model is used to adjust the brightness and color of the first image and output the adjusted second image;

[0007] Inputting the first image and the second image into a second network model for processing to obtain a third image, wherein the second network model is used to correct the texture of the second image to be consistent with the texture of the first image, and output the corrected third image;

[0008] Based on the third image, a target image corresponding to the original image is generated.

[0009] In a second aspect, an embodiment of the present application provides an image processing device, the device comprising:

[0010] an acquisition module, configured to acquire a first image and a second image, wherein the first image is an image obtained by downsampling the original image, and the second image is an image obtained by processing the first image through a first network model, and the first network model is configured to adjust the brightness and color of the first image and output the adjusted second image;

[0011] a processing module, configured to input the first image and the second image into a second network model for processing to obtain a third image, wherein the second network model is configured to correct the texture of the second image to be consistent with the texture of the first image, and output the corrected third image;

[0012] A generating module is used to generate a target image corresponding to the original image based on the third image.

[0013] In a third aspect, an embodiment of the present application provides an electronic device, comprising a processor, a memory, and a program or instruction stored in the memory and executable on the processor, wherein the program or instruction, when executed by the processor, implements the steps of the image processing method described in the first aspect.

[0014] In a fourth aspect, an embodiment of the present application provides a readable storage medium, on which a program or instruction is stored, and when the program or instruction is executed by a processor, the steps of the image processing method described in the first aspect are implemented.

[0015] In a fifth aspect, an embodiment of the present application provides a chip, comprising a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run a program or instruction to implement the steps of the image processing method as described in the first aspect.

[0016] In an embodiment of the present application, a first image and a second image are obtained, wherein the first image is an image obtained by downsampling the original image, and the second image is an image obtained by processing the first image through a first network model, and the first network model is used to adjust the brightness and color of the first image and output the adjusted second image; the first image and the second image are input into the second network model for processing to obtain a third image, wherein the second network model is used to correct the texture of the second image to be consistent with the texture of the first image, and output the corrected third image; based on the third image, a target image corresponding to the original image is generated.

[0017] It can be seen that in the embodiment of the present application, in order to address the problem that the brightness and color of the image output by the first network model meet the requirements but the texture is poor, the input image and output image of the first network model can be simultaneously input into the second network model for processing. The second network model uses the image texture of the input image of the first network model as a guide to correct the image texture of the output image of the first network model. Thus, a target image that meets the requirements in terms of brightness, color and texture can be obtained, thereby achieving the purpose of correcting the texture of the output image of the first network model and improving the scope of use of image-to-image applications based on the stable diffusion model. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 is a flow chart of an image processing method provided by an embodiment of the present application;

[0019] Figure 2 This is an example diagram of the network architecture of the stable diffusion model provided in the embodiment of the present application;

[0020] Figure 3 is an example diagram of an image processing method provided by an embodiment of the present application;

[0021] Figure 4 is a structural block diagram of an image processing device provided in an embodiment of the present application;

[0022] Figure 5 It is a structural schematic diagram of an electronic device provided in an embodiment of the present application;

[0023] Figure 6 It is a schematic diagram of the hardware structure of an electronic device implementing an embodiment of the present application. DETAILED DESCRIPTION

[0024] The following will be combined with the drawings in the embodiments of the present application to clearly describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the present application belong to the scope of protection of this application.

[0025] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the terms used in this way are interchangeable where appropriate, so that the embodiments of the present application can be implemented in an order other than those illustrated or described herein, and the objects distinguished by "first", "second", etc. are generally of one type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0026] The original stable diffusion model is a natural image application. In subsequent studies, a large number of image-to-image conversion methods based on the stable diffusion model have been proposed, which has led to the emergence of image-to-image applications based on the stable diffusion model. Since the entire diffusion process in the stable diffusion model is carried out in the VAE domain, which is an image compression domain, there will be a random sampling process in the process of restoring the VAE domain to the image domain. Therefore, image-to-image applications based on the stable diffusion model will inevitably face the problem of image texture change loss. Taking the brightness and color adjustment network based on the stable diffusion model as an example, the image output by the brightness and color adjustment network meets the conditions, but the texture is poor.

[0027] In the related art, in image quality tasks, the main method used is to fuse the features in the VAE encoder of the input image, so as to make some trade-offs between the generated intensity and image fidelity. Since the stable diffusion model has a strong generation capability, it is an imperfect but acceptable method to appropriately regress the generated intensity from the perspective of fidelity. However, in brightness and color tasks, if this method is also used, since the information of the original image will be integrated, an undesirable brightness or color will be obtained, and the methods applicable to image quality tasks will no longer be applicable. At the same time, due to the powerful pre-training capability of the stable diffusion model, the brightness and color conversion based on the stable diffusion model can achieve many functions that cannot be achieved by traditional CV methods or traditional convolutional neural networks. This problem is a bottleneck problem that limits the development of brightness and color tasks in the AIGC era and needs to be solved urgently. In order to solve the above technical problems, the embodiments of the present application provide an image processing method and a device thereof.

[0028] The image processing method provided in the embodiment of the present application is described in detail below through specific embodiments and their application scenarios in conjunction with the accompanying drawings.

[0029] It should be noted that the image processing method provided in the embodiment of the present application is applicable to electronic devices. In practical applications, the electronic devices may include: mobile terminals such as smart phones, tablet computers, personal digital assistants, or computer devices such as laptop computers, desktops, servers, etc., and the embodiment of the present application is not limited to this.

[0030] Figure 1 is a flow chart of an image processing method provided in an embodiment of the present application, such as Figure 1 As shown, the method may include the following steps: S101, S102 and S103;

[0031] In S101, a first image and a second image are obtained, wherein the first image is an image obtained by downsampling the original image, and the second image is an image obtained by processing the first image through a first network model, and the first network model is used to adjust the brightness and color of the first image and output the adjusted second image.

[0032] In the embodiment of the present application, the first network model may also be referred to as a brightness and color adjustment network, which is a model obtained by training based on a stable diffusion model and is a graph-to-graph application based on a stable diffusion model.

[0033] In some embodiments, the above S101 may include the following steps: S1011, S1012 and S1013;

[0034] In S1011, an original image is acquired.

[0035] In the embodiment of the present application, the original image refers to the image that needs to be processed and is input by the user. The original image is usually an image whose brightness and color do not meet the requirements. The resolution of the original image is recorded as the original resolution.

[0036] In S1012, the original image is downsampled to obtain a first image.

[0037] In the embodiment of the present application, the resolution of the first image is smaller than the resolution of the original image.

[0038] In the embodiment of the present application, due to factors such as processing speed and computer video memory, the images that can be processed by the stable diffusion algorithm are often small, and the most common processing resolution is 512*512 pixels. However, the images in people's daily lives, that is, the resolution of the original images to be processed in the embodiment of the present application, usually ranges from several million to tens of millions of pixels. For example, the more common image resolution is about 4000*3000 pixels, which is much larger than the resolution that the stable diffusion model can process. Therefore, before inputting into the first network model, the original image needs to be downsampled to a resolution that can be processed by the stable diffusion algorithm, such as 512*512 pixels, to obtain the first image.

[0039] In the embodiment of the present application, image downsampling is a commonly used image processing method, and there are many ways, for example, using an area-based downsampling method, which can better represent the information of pixels within the original image range. In practical applications, the INTER_AREA mode of the resize function in the open source image processing library OPENCV can be used for downsampling.

[0040] In S1013, the first image is input into the first network model for processing to obtain a second image, wherein the first network model is a model trained based on a stable diffusion model, and the first network model does not include a feature fusion module.

[0041] In the embodiment of the present application, the resolution of the second image is the same as the resolution of the first image.

[0042] In the embodiment of the present application, compared with the first image, the brightness and color of the second image meet the requirements (relatively ideal) but the texture is poor.

[0043] For ease of understanding, the training process of the first network model is described in detail. The training process of the first network model may include the following steps: S20, S21 and S22;

[0044] S20, constructing a training set for the first network model.

[0045] In an embodiment of the present application, in order to improve the performance of the first network model on a single task / application, a data set of a single task / application may be used as a training set, for example, a PPR10K data set or a FiveK data set.

[0046] Taking the FiveK dataset as an example, the image I before adjustment in the FiveK dataset is used as the input data in the model training process, and the adjusted image G of an expert (such as Expert C) is used as the output target in the model training process.

[0047] In the embodiment of the present application, considering that the data rules between different tasks / applications are the same, it is possible to consider using various data to train a universal first network model, which can be used in new tasks / applications without retraining. Accordingly, a collection of many brightness and color adjustment data sets can be used as a training set, for example, the ImageNet data set or the LAION-5B data set.

[0048] S21, constructing the first backbone network based on the stable diffusion model.

[0049] For example, Figure 2As shown, the network structure of the original stable diffusion model can include the following modules: VAE encoder, VAE decoder, denoising diffusion network, image injection module and feature fusion module (TrainableControllable Feature Wrapping, CFW);

[0050] The VAE encoder is used to convert the input image from the image domain to the VAE domain; the image injection module is used to inject the information of the input image into the denoising diffusion network; the denoising diffusion network is used to obtain the final VAE domain image Z through a certain number of steps (e.g., 1000) of loop iterations. 0 ; The feature fusion module is used to fuse the VAE features of the input image and the image Z 0 , the fusion ratio is controlled by the weight parameter. The larger the weight parameter, the greater the contribution of the VAE feature of the input image, the more similar the output image is to the input image, and the better the image fidelity, but at the same time the brightness and color generation effect will deteriorate; the VAE decoder is used to decode the image in the VAE domain into a normal color image in the image domain, that is, the output image.

[0051] The original stable diffusion model was first used for image quality enhancement tasks to obtain a high-definition image from a low-definition image. This architecture can also be used for the brightness and color adjustment algorithm involved in the embodiments of the present application. An image whose brightness and color do not meet the conditions is input, and a result image whose brightness and color meet the conditions (close to the ideal state) is obtained.

[0052] Considering the VAE problem mentioned above, the output image of the stable diffusion model will have texture change loss, and unlike the image quality task, in the brightness and color task, if Figure 2 If the feature fusion module in the image is used to improve the fidelity performance, the undesirable brightness and color in the input image will be brought to the output image. Therefore, in the embodiment of the present application, the feature fusion module is removed in the brightness and color task, that is, the backbone network based on the stable diffusion model does not include the feature fusion module.

[0053] S22: Based on the training set, the first backbone network is trained until the model converges to obtain a first network model.

[0054] Exemplarily, the training set of the first network model includes multiple groups of training samples, each group of training samples includes an input image I and an output image G, wherein the input image I is an image whose brightness and color do not meet the requirements, and the output image G is an image whose brightness and color meet the requirements after the input image I is adjusted by professional image processing personnel.

[0055] The input image I in a batch of training samples is used as the input data of the first backbone network, and the output image G is used as the output target of the first backbone network. The first backbone network is trained so that the network output is close to the output target G. When the deviation between the actual output image of the network and the image G is less than a certain range, the training target is considered to be achieved and the training task is terminated. The trained network can learn the change law of the data set used for training and can be used for reasoning. For example, an input image similar to the input in the FiveK data set is sent to the network, and an output image close to that of Expert C can be obtained.

[0056] It can be seen that in the embodiment of the present application, since the first network model does not include a feature fusion module, the brightness and color that do not meet the conditions in the original image will not be brought to the output image, so that the brightness and color of the image output by the first network model meet the conditions.

[0057] It should be noted that the brightness and color mentioned above meet the conditions, which can also be called relatively ideal brightness and color, which means that the brightness satisfies a certain brightness threshold range, and the color satisfies a certain color threshold range; the brightness and color mentioned above do not meet the conditions, which can also be called unideal brightness and color, which means that the brightness does not meet a certain brightness threshold range, and the color does not meet a certain color threshold range.

[0058] In S102, the first image and the second image are input into the second network model for processing to obtain a third image, wherein the second network model is used to correct the texture of the second image to be consistent with the texture of the first image, and output the corrected third image.

[0059] In an embodiment of the present application, the second network model may also be referred to as a texture correction network, which is a model obtained by training an image-to-image transformation network based on a training set. Since the training set is used to provide sample images containing information such as brightness, color, and texture, the second network model can learn rich prior knowledge of brightness, color, and texture information.

[0060] In the embodiment of the present application, the resolution of the third image is the same as the resolution of the first image.

[0061] In the embodiment of the present application, compared with the first image and the second image, the brightness, color and texture of the third image all meet the requirements.

[0062] It should be noted that the brightness, color and texture mentioned above meet the conditions, which can also be called ideal brightness and color and correct texture, which means that the brightness satisfies a certain brightness threshold range, the color satisfies a certain color threshold range, and the texture of the output image is consistent with the texture of the original image; the brightness, color and texture mentioned above do not meet the conditions, which can also be called unsatisfactory brightness and color and poor texture, which means that the brightness does not meet a certain brightness threshold range, the color does not meet a certain color threshold range, and the texture of the output image is inconsistent with the texture of the original image.

[0063] In some embodiments, considering that in the image-to-image brightness and color adjustment task, adjacent pixels often use the same or similar adjustments, this adjustment can be regarded as a relatively low-frequency signal. At the same time, the texture loss caused by VAE is mainly reflected in the high-frequency information. Based on the above differences, a training set for the texture correction task can be constructed.

[0064] Accordingly, the training set includes multiple groups of different training samples, wherein each group of training samples includes: a sample initial image, a sample reference image and a sample texture loss image; the sample reference image is an image in which the numerical value of the target parameter item of the sample initial image is adjusted to meet the preset conditions, and the target parameter item includes but is not limited to at least one of brightness, color and texture; the sample texture loss image is an image obtained after the sample reference image is processed by the variational autoencoder and decoder; the sample initial image and the sample texture loss image in each group of training samples are used as output data in the model training process; the sample reference image in each group of training samples is used as the output target in the model training process.

[0065] For ease of understanding, the training process of the second network model is described in detail. The training process of the second network model may include the following steps: S30, S31 and S32;

[0066] S30, constructing a training set for the second network model.

[0067] In an embodiment of the present application, in order to improve the performance of the second network model on a single task / application, a data set of a single task / application may be used as a training set, for example, a PPR10K data set or a FiveK data set.

[0068] In the embodiment of the present application, considering that the data rules between different tasks / applications are the same, it is possible to consider using a variety of data to train a universal second network model, which can be used in new tasks / applications without retraining. Accordingly, a collection of many texture restoration datasets can be used as a training set, such as the ImageNet dataset or the LAION-5B dataset.

[0069] Exemplarily, taking the FiveK dataset as an example, the unadjusted image I in the FiveK dataset is used as one of the input data in the model training process, and the adjusted image G of an expert (such as Expert C) is used as the output target in the model training process. The adjusted image G is input into the encoder and decoder of VAE to obtain image G'. The brightness and color of image G' are very close to image G, but there are some changes in texture, and this change is the loss caused by the VAE network; wherein the above image I is the sample initial image, image G is the sample reference image, and image G' is the sample texture loss image.

[0070] S31, constructing a second backbone network based on an image-to-image transformation network.

[0071] In the embodiment of the present application, the purpose of the second network model is to map the image I whose brightness and color do not meet the requirements and the image G' whose brightness and color meet the requirements but has texture loss to the image G that meets the brightness and color requirements and has correct texture.

[0072] Commonly used image-to-image transformation networks can meet this requirement, among which U-Net is the most representative. The slight difference is that the conventional image-to-image is 3 channels (color images have 3 channels in total, R, G and B) to 3 channels. In the embodiment of the present application, it is necessary to change to 6-channel input and 3-channel output. Therefore, the channels of the input image need to be changed to 6.

[0073] In the embodiment of the present application, during the training process of the second backbone network, the input 6-channel data is obtained by stitching two 3-channel images, where one image is image I and the other image is image G', and image G' is obtained by processing image G through the complete VAE process, that is, after the encoder and decoder, image G' is an image whose brightness and color meet the requirements but has texture loss, which is used to simulate the output of the stable diffusion model in the actual reasoning process. Image G is used as the output target of the second backbone network.

[0074] S32: Based on the training set, the second backbone network is trained until the model converges to obtain a second network model.

[0075] Exemplarily, the training set includes multiple groups of training samples, each group of training samples includes a sample initial image I, a sample reference image G and a sample texture loss image G', wherein image I is an image whose brightness and color do not meet the requirements, image G is an image whose brightness and color meet the requirements after image I is adjusted by professional image processing personnel, and image G' is an image that meets the brightness and color requirements but has texture loss.

[0076] The image I and the image G' in a batch of training samples are used as the input data of the second backbone network, and the image G is used as the output target of the second backbone network. The second backbone network is trained so that the network output approaches the output target G. When the deviation between the network output image and the image G is less than a certain range, it is considered that the training target is achieved and the training task is terminated. The trained network can learn the variation law of the data set used for training, and can fit a second network model that has the ability to integrate the texture of the input image and the brightness and color of the image with texture loss but correct brightness and color. The second network model can be used for reasoning. The first image and the second image in the aforementioned step 101 are input into the model to obtain a texture-corrected third image.

[0077] It can be seen that in the embodiment of the present application, the input of the second network model includes the input image and output image of the first network model. Since the texture of the input image is what is ultimately desired, and the brightness and color of the output image are also what is ultimately desired, this network has the ability to combine the information of the two, and uses the texture of the input image to correct the texture changes of the output image, and automatically integrates the brightness, color and texture of the two images, so as to obtain a final result in which the texture is consistent with the input, and the brightness and color are consistent with the output of the stable diffusion model.

[0078] In S103, based on the third image, a target image corresponding to the original image is generated.

[0079] In the embodiment of the present application, the resolution of the third image is smaller than the resolution of the original image.

[0080] In the embodiment of the present application, the resolution of the target image is the same as the resolution of the original image.

[0081] In the embodiment of the present application, compared with the original image, the brightness, color and texture of the target image all meet the requirements.

[0082] Considering that the resolution of the obtained image is the same as that of the original image when the third image is directly upsampled, the upsampling process may cause the loss of high-frequency information of pixels, resulting in image distortion. In order to avoid the above problem, in some embodiments, the above S103 may include the following steps: S1031, S1032 and S1033;

[0083] In S1031, a pixel value difference operation is performed on the same pixel position of the first image and the third image to obtain a fourth image.

[0084] In the embodiment of the present application, the pixel value of each pixel position in the first image is subtracted from the pixel value of the corresponding same pixel position in the third image to obtain the fourth image.

[0085] In the embodiment of the present application, the pixel value of each pixel position in the fourth image represents the image adjustment amount between the first image and the third image at the pixel position.

[0086] In S1032, the fourth image is upsampled to obtain a fifth image, wherein the resolution of the fifth image is the same as the resolution of the original image.

[0087] In the embodiment of the present application, the image adjustment amount is obtained from the above step S1031, but because the resolution of the fourth image corresponding to the image adjustment amount is lower than the resolution of the original image, in order to apply this image adjustment amount to the original image of the original resolution, the fourth image needs to be upsampled.

[0088] In the embodiment of the present application, the pixel value of each pixel position in the fifth image represents the adjustment amount of the original image from the expected target image.

[0089] Considering that conventional upsampling cannot meet the requirements, because it will cause the image and the adjustment amount to not match at the boundaries of some objects or detail-rich areas, which will lead to the appearance of artifacts such as halo and heterochromatic. In order to solve this problem, in an embodiment of the present application, by introducing a guide image, the edges of the image are guided during the upsampling process. Accordingly, the above S1032 may include the following steps: the original image may be determined as the guide image in the upsampling process of the fourth image; the fourth image is upsampled based on the guide image to obtain the fifth image.

[0090] In S1033, a pixel value difference operation is performed on the same pixel position of the original image and the fifth image to obtain a target image corresponding to the original image.

[0091] In the embodiment of the present application, the pixel value of each pixel position in the original image is subtracted from the pixel value of the corresponding same pixel position in the fifth image to obtain the target image.

[0092] Exemplarily, the first image is recorded as In_down, the second image is recorded as Out_down, and the third image is recorded as Diff_down, wherein Diff_down=In_down-Out_down.

[0093] The fourth image is recorded as Diff, where Diff=F_upsampling(In_down, Diff_down), and F_upsampling is recorded as a guided upsampling algorithm. Different from the conventional method, In_down participates in the upsampling process, and various edge-preserving smoothing algorithms are often used to implement it in practical applications.

[0094] The target image is denoted as M, where M=In_down-Diff.

[0095] It can be seen that in the embodiments of the present application, a target image with brightness, color and texture that meet the requirements at the original resolution can be generated based on a guided upsampling method and two image difference calculations.

[0096] In order to facilitate the overall understanding of the embodiments of the present application, Figure 3 The example diagram is shown in the following figure. Figure 3 As shown, firstly, the original image is obtained, and the original image is downsampled to obtain the first image; then the first image is input into the first network model for processing to obtain the second image; then the first image and the second image are simultaneously input into the second network model for processing to obtain the third image; then the pixel value difference operation is performed on the same pixel position of the first image and the third image to obtain the fourth image, and the fourth image is upsampled based on the original image to obtain the fifth image; finally, the pixel value difference operation is performed on the same pixel position of the original image and the fifth image to obtain the target image, thereby correcting the texture change problem in the result of the stable diffusion algorithm in the brightness and color task.

[0097] It can be seen from the above embodiment that in this embodiment, a first image and a second image are obtained, wherein the first image is an image obtained by downsampling the original image, and the second image is an image obtained by processing the first image by a first network model, and the first network model is used to adjust the brightness and color of the first image and output the adjusted second image; the first image and the second image are input into the second network model for processing to obtain a third image, wherein the second network model is used to correct the texture of the second image to be consistent with the texture of the first image, and output the corrected third image; based on the third image, a target image corresponding to the original image is generated.

[0098] It can be seen that in the embodiment of the present application, in order to address the problem that the brightness and color of the image output by the first network model meet the requirements but the texture is poor, the input image and output image of the first network model can be simultaneously input into the second network model for processing. The second network model uses the image texture of the input image of the first network model as a guide to correct the image texture of the output image of the first network model. Thus, a target image that meets the requirements in terms of brightness, color and texture can be obtained, thereby achieving the purpose of correcting the texture of the output image of the first network model and improving the scope of use of image-to-image applications based on the stable diffusion model.

[0099] Figure 4 is a structural block diagram of an image processing device provided in an embodiment of the present application, such as Figure 4 As shown, the image processing device 400 may include: an acquisition module 401, a processing module 402 and a generation module 403;

[0100] An acquisition module 401 is used to acquire a first image and a second image, wherein the first image is an image obtained by downsampling the original image, and the second image is an image obtained by processing the first image through a first network model, and the first network model is used to adjust the brightness and color of the first image and output the adjusted second image;

[0101] A processing module 402 is used to input the first image and the second image into a second network model for processing to obtain a third image, wherein the second network model is used to correct the texture of the second image to be consistent with the texture of the first image, and output the corrected third image;

[0102] The generating module 403 is configured to generate a target image corresponding to the original image based on the third image.

[0103] It can be seen from the above embodiment that in this embodiment, a first image and a second image are obtained, wherein the first image is an image obtained by downsampling the original image, and the second image is an image obtained by processing the first image by a first network model, and the first network model is used to adjust the brightness and color of the first image and output the adjusted second image; the first image and the second image are input into the second network model for processing to obtain a third image, wherein the second network model is used to correct the texture of the second image to be consistent with the texture of the first image, and output the corrected third image; based on the third image, a target image corresponding to the original image is generated.

[0104] It can be seen that in the embodiment of the present application, in order to address the problem that the brightness and color of the image output by the first network model meet the requirements but the texture is poor, the input image and output image of the first network model can be simultaneously input into the second network model for processing. The second network model uses the image texture of the input image of the first network model as a guide to correct the image texture of the output image of the first network model. Thus, a target image that meets the requirements in terms of brightness, color and texture can be obtained, thereby achieving the purpose of correcting the texture of the output image of the first network model and improving the scope of use of image-to-image applications based on the stable diffusion model.

[0105] Optionally, as an embodiment, the second network model is a model obtained by training an image-to-image transformation network based on a training set;

[0106] The training set includes multiple groups of different training samples, wherein each group of training samples includes: a sample initial image, a sample reference image and a sample texture loss image;

[0107] The sample reference image is an image in which the value of the target parameter item of the sample initial image after adjustment meets the preset conditions; the sample texture loss image is an image obtained after the sample reference image is processed by the variational autoencoder and the decoder;

[0108] The sample initial image and the sample texture loss image in each group of training samples are used as output data in the model training process; and the sample reference image in each group of training samples is used as an output target in the model training process.

[0109] Optionally, as an embodiment, the generating module 403 may include:

[0110] A first processing submodule, configured to perform a pixel value difference operation on the same pixel position of the first image and the third image to obtain a fourth image;

[0111] a second processing submodule, configured to upsample the fourth image to obtain a fifth image, wherein a resolution of the fifth image is the same as a resolution of the original image;

[0112] The third processing submodule is used to perform pixel value difference operation on the same pixel position between the original image and the fifth image to obtain a target image corresponding to the original image.

[0113] Optionally, as an embodiment, the second processing submodule may include:

[0114] a determining unit, configured to determine the original image as a guide image in an upsampling process of the fourth image;

[0115] An upsampling unit is used to upsample the fourth image based on the guide image to obtain a fifth image.

[0116] Optionally, as an embodiment, the acquisition module 401 may include:

[0117] An acquisition submodule, used for acquiring the original image;

[0118] A fourth processing submodule, configured to downsample the original image to obtain the first image;

[0119] The fifth processing submodule is used to input the first image into the first network model for processing to obtain the second image, wherein the first network model is a model trained based on a stable diffusion model, and the first network model does not include a feature fusion module.

[0120] The image processing device in the embodiment of the present application can be a device, or a component, integrated circuit, or chip in a terminal. The device can be a mobile electronic device or a non-mobile electronic device. Exemplarily, the mobile electronic device can be a mobile phone, a tablet computer, a laptop computer, a PDA, an in-vehicle electronic device, a wearable device, an ultra-mobile personal computer (Ultra-Mobile Personal Computer, UMPC), a netbook or a personal digital assistant (Personal Digital Assistant, PDA), etc., and the non-mobile electronic device can be a server, a network attached storage (Network Attached Storage, NAS), a personal computer (Personal Computer, PC), a television (Television, TV), a teller machine or a self-service machine, etc., which is not specifically limited in the embodiment of the present application.

[0121] The image processing device in the embodiment of the present application may be a device having an operating system. The operating system may be an Android operating system, an IOS operating system, or other possible operating systems, which are not specifically limited in the embodiment of the present application.

[0122] The image processing device provided in the embodiment of the present application can achieve Figure 1 To avoid repetition, the various processes implemented in the illustrated method embodiment will not be described again here.

[0123] Alternatively, if Figure 5 As shown, an embodiment of the present application also provides an electronic device 500, including a processor 501, a memory 502, and a program or instruction stored in the memory 502 and executable on the processor 501. When the program or instruction is executed by the processor 501, each process of the above-mentioned image processing method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be described here.

[0124] It should be noted that the electronic devices in the embodiments of the present application include the mobile electronic devices and non-mobile electronic devices mentioned above.

[0125] Figure 6 It is a schematic diagram of the hardware structure of an electronic device implementing an embodiment of the present application.

[0126] The electronic device 600 includes but is not limited to: a radio frequency unit 601, a network module 602, an audio output unit 603, an input unit 604, a sensor 605, a display unit 606, a user input unit 607, an interface unit 608, a memory 609, and a processor 610.

[0127] Those skilled in the art will appreciate that the electronic device 600 may also include a power source (such as a battery) for supplying power to each component, and the power source may be logically connected to the processor 610 through a power management system, thereby implementing functions such as managing charging, discharging, and power consumption management through the power management system. Figure 6 The electronic device structure shown in the figure does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently, which will not be described in detail here.

[0128] Among them, the processor 610 is used to obtain a first image and a second image, wherein the first image is an image obtained by downsampling the original image, and the second image is an image obtained by processing the first image through a first network model, and the first network model is used to adjust the brightness and color of the first image and output the adjusted second image; the first image and the second image are input into the second network model for processing to obtain a third image, wherein the second network model is used to correct the texture of the second image to be consistent with the texture of the first image, and output the corrected third image; based on the third image, a target image corresponding to the original image is generated.

[0129] It can be seen that in the embodiment of the present application, in order to address the problem that the brightness and color of the image output by the first network model meet the requirements but the texture is poor, the input image and output image of the first network model can be simultaneously input into the second network model for processing. The second network model uses the image texture of the input image of the first network model as a guide to correct the image texture of the output image of the first network model. Thus, a target image that meets the requirements in terms of brightness, color and texture can be obtained, thereby achieving the purpose of correcting the texture of the output image of the first network model and improving the scope of use of image-to-image applications based on the stable diffusion model.

[0130] Optionally, as an embodiment, the second network model is a model obtained by training an image-to-image transformation network based on a training set;

[0131] The training set includes multiple groups of different training samples, wherein each group of training samples includes: a sample initial image, a sample reference image and a sample texture loss image;

[0132] The sample reference image is an image in which the value of the target parameter item of the sample initial image after adjustment meets the preset conditions; the sample texture loss image is an image obtained after the sample reference image is processed by the variational autoencoder and the decoder;

[0133] The sample initial image and the sample texture loss image in each group of training samples are used as output data in the model training process; and the sample reference image in each group of training samples is used as an output target in the model training process.

[0134] Optionally, as an embodiment, the processor 610 is specifically used to perform pixel value difference operation on the same pixel position between the first image and the third image to obtain a fourth image; upsample the fourth image to obtain a fifth image, wherein the resolution of the fifth image is the same as the resolution of the original image; and perform pixel value difference operation on the same pixel position between the original image and the fifth image to obtain a target image corresponding to the original image.

[0135] Optionally, as an embodiment, the processor 610 is specifically configured to determine the original image as a guide image in an upsampling process of the fourth image; and upsample the fourth image based on the guide image to obtain a fifth image.

[0136] Optionally, as an embodiment, the processor 610 is specifically used to obtain the original image; downsample the original image to obtain the first image; input the first image into the first network model for processing to obtain the second image, wherein the first network model is a model trained based on a stable diffusion model, and the first network model does not include a feature fusion module.

[0137] It should be understood that in the embodiment of the present application, the input unit 604 may include a graphics processor (GPU) 6041 and a microphone 6042, and the graphics processor 6041 processes the image data of the static picture or video obtained by the image capture device (such as a camera) in the video capture mode or the image capture mode. The display unit 606 may include a display panel 6061, and the display panel 6061 may be configured in the form of a liquid crystal display, an organic light emitting diode, etc. The user input unit 607 includes a touch panel 6071 and other input devices 6072. The touch panel 6071 is also called a touch screen. The touch panel 6071 may include two parts: a touch detection device and a touch controller. Other input devices 6072 may include but are not limited to a physical keyboard, a function key (such as a volume control button, a switch button, etc.), a trackball, a mouse, and a joystick, which will not be repeated here. The memory 609 can be used to store software programs and various data, including but not limited to applications and operating systems. The processor 610 can integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, the user interface, and the application program, and the modem processor mainly processes wireless communication. It is understandable that the above-mentioned modem processor may not be integrated into the processor 610.

[0138] An embodiment of the present application also provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, each process of the above-mentioned image processing method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0139] The processor is a processor in the electronic device described in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0140] An embodiment of the present application further provides a chip, which includes a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned image processing method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0141] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

[0142] It should be noted that, in this article, the terms "comprise", "include" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise one..." do not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0143] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a computer software product, which is stored in a storage medium (such as ROM / RAM, a disk, or an optical disk), including instructions for enabling a terminal (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0144] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

Claims

1. An image processing method, It is characterized in that The method comprises: Acquire a first image and a second image, wherein the first image is an image obtained by downsampling the original image, and the second image is an image obtained by processing the first image through a first network model, and the first network model is used to adjust the brightness and color of the first image and output the adjusted second image; Inputting the first image and the second image into a second network model for processing to obtain a third image, wherein the second network model is used to correct the texture of the second image to be consistent with the texture of the first image, and output the corrected third image; Based on the third image, a target image corresponding to the original image is generated.

2. The method according to claim 1, It is characterized in that The second network model is a model obtained by training an image-to-image transformation network based on a training set; The training set includes multiple groups of different training samples, wherein each group of training samples includes: a sample initial image, a sample reference image and a sample texture loss image; The sample reference image is an image in which the value of the target parameter item of the sample initial image after adjustment meets the preset conditions; the sample texture loss image is an image obtained after the sample reference image is processed by the variational autoencoder and the decoder; The sample initial image and the sample texture loss image in each group of training samples are used as output data in the model training process; and the sample reference image in each group of training samples is used as an output target in the model training process.

3. The method according to claim 1 or 2, It is characterized in that The step of generating a target image corresponding to the original image based on the third image includes: Performing a pixel value difference operation on the same pixel position of the first image and the third image to obtain a fourth image; Upsampling the fourth image to obtain a fifth image, wherein a resolution of the fifth image is the same as a resolution of the original image; A pixel value difference operation is performed on the same pixel position of the original image and the fifth image to obtain a target image corresponding to the original image.

4. The method according to claim 3, It is characterized in that The step of upsampling the fourth image to obtain a fifth image includes: Determining the original image as a guide image in an upsampling process of the fourth image; The fourth image is upsampled based on the guide image to obtain a fifth image.

5. The method according to claim 1, It is characterized in that The acquiring of the first image and the second image comprises: Acquire the original image; Downsampling the original image to obtain the first image; The first image is input into the first network model for processing to obtain the second image, wherein the first network model is a model trained based on a stable diffusion model, and the first network model does not include a feature fusion module.

6. An image processing device, It is characterized in that The device comprises: an acquisition module, configured to acquire a first image and a second image, wherein the first image is an image obtained by downsampling the original image, and the second image is an image obtained by processing the first image through a first network model, and the first network model is configured to adjust the brightness and color of the first image and output the adjusted second image; a processing module, configured to input the first image and the second image into a second network model for processing to obtain a third image, wherein the second network model is configured to correct the texture of the second image to be consistent with the texture of the first image, and output the corrected third image; A generating module is used to generate a target image corresponding to the original image based on the third image.

7. The device according to claim 6, It is characterized in that The second network model is a model obtained by training an image-to-image transformation network based on a training set; The training set includes multiple groups of different training samples, wherein each group of training samples includes: a sample initial image, a sample reference image and a sample texture loss image; The sample reference image is an image in which the value of the target parameter item of the sample initial image after adjustment meets the preset conditions; the sample texture loss image is an image obtained after the sample reference image is processed by the variational autoencoder and the decoder; The sample initial image and the sample texture loss image in each group of training samples are used as output data in the model training process; and the sample reference image in each group of training samples is used as an output target in the model training process.

8. The device according to claim 6 or 7, It is characterized in that The generation module comprises: A first processing submodule, configured to perform a pixel value difference operation on the same pixel position of the first image and the third image to obtain a fourth image; a second processing submodule, configured to upsample the fourth image to obtain a fifth image, wherein a resolution of the fifth image is the same as a resolution of the original image; The third processing submodule is used to perform pixel value difference operation on the same pixel position between the original image and the fifth image to obtain a target image corresponding to the original image.

9. The device according to claim 8, It is characterized in that The second processing submodule includes: a determining unit, configured to determine the original image as a guide image in an upsampling process of the fourth image; An upsampling unit is used to upsample the fourth image based on the guide image to obtain a fifth image.

10. The device according to claim 6, It is characterized in that The acquisition module comprises: An acquisition submodule, used for acquiring the original image; A fourth processing submodule, configured to downsample the original image to obtain the first image; The fifth processing submodule is used to input the first image into the first network model for processing to obtain the second image, wherein the first network model is a model trained based on a stable diffusion model, and the first network model does not include a feature fusion module.

Images