Image processing method and device and electronic equipment

Abstract

The invention provides an image processing method and device and electronic equipment. A specific implementation mode of the method comprises the steps of obtaining a first image, a first noise image and a second noise image; fusing the first image and the first noise image to obtain a second image; and based on the second image, performing multiple rounds of denoising operation on the second noise image to obtain a target image which has the same style as the first image and has preset features. According to the embodiment, more details of the first image can be fused into the generated target image, so that the generated first image is more vivid, the quality of the generated image is improved, and the user experience is improved.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to an image processing method, device and electronic equipment. Background Art

[0002] With the application and development of artificial intelligence technology in the field of images, image generation technology has emerged. In related technologies, new images related to known images can be generated based on technologies such as GAN and VAE, but the generated images are relatively rough in details and have poor realism. There is still a big gap in image quality compared to real images. Therefore, there is a need for a solution that can generate high-quality images based on known images. Summary of the invention

[0003] The present disclosure provides an image processing method, an image processing device and an electronic device.

[0004] According to a first aspect, there is provided an image processing method, the method comprising:

[0005] Acquire a first image, a first noise image, and a second noise image;

[0006] fusing the first image with the first noise image to obtain a second image;

[0007] Based on the second image, multiple rounds of denoising operations are performed on the second noisy image to obtain a target image with the same style as the first image and preset features.

[0008] According to a second aspect, there is provided an image processing apparatus, the apparatus comprising:

[0009] An acquisition module, used for acquiring a first image, a first noise image and a second noise image;

[0010] a fusion module, used for fusing the first image and the first noise image to obtain a second image;

[0011] A denoising module is used to perform multiple rounds of denoising operations on the second noisy image based on the second image to obtain a target image with the same style as the first image and preset features.

[0012] According to a third aspect, a computer-readable storage medium is provided, wherein the storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of the above-mentioned first aspects is implemented.

[0013] According to a fourth aspect, an electronic device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the method described in any one of the first aspects is implemented.

[0014] The technical solution provided by the embodiments of the present disclosure may have the following beneficial effects:

[0015] The embodiments of the present disclosure provide an image processing method and device, which obtains a second image by fusing a first image and a first noise image, and performs multiple rounds of denoising operations on the second noise image based on the fused second image to obtain a target image. The multiple rounds of denoising operations can control the target image to have preset features, and the second image fused with the first image can make the generated target image have the same style as the first image. In this way, more details of the first image can be integrated into the generated target image, making the generated first image more realistic, improving the quality of the generated image, and enhancing the user experience.

[0016] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions of the embodiments of this specification, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this specification. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0018] Figure 1 It is a schematic diagram of a scene of image processing in the related art;

[0019] Figure 2 is a schematic diagram of an image processing scene according to an exemplary embodiment of the present disclosure;

[0020] Figure 3 is a flowchart of an image processing method according to an exemplary embodiment of the present disclosure;

[0021] Figure 4 is a flowchart of another image processing method according to an exemplary embodiment of the present disclosure;

[0022] Figure 5 is a block diagram of an image processing device according to an exemplary embodiment of the present disclosure;

[0023] Figure 6 is a schematic block diagram of an electronic device provided by some embodiments of the present disclosure;

[0024] Figure 7 is a schematic block diagram of another electronic device provided by some embodiments of the present disclosure;

[0025] Figure 8is a schematic diagram of a storage medium provided in some embodiments of the present disclosure. DETAILED DESCRIPTION

[0026] In order to enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below in conjunction with the drawings in the embodiments of this specification. Obviously, the described embodiments are only part of the embodiments of this specification, not all of the embodiments. Based on the embodiments in this specification, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of this specification.

[0027] When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are only examples of devices and methods consistent with some aspects of the present disclosure as detailed in the attached claims.

[0028] The terms used in this disclosure are only for the purpose of describing specific embodiments and are not intended to limit the disclosure. The singular forms of "a", "the" and "the" used in this disclosure are also intended to include plural forms, unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items.

[0029] It should be understood that although the terms first, second, third, etc. may be used in the present disclosure to describe various information, such information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of the present disclosure, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0030] With the application and development of artificial intelligence technology in the field of images, image generation technology has emerged. In related technologies, new images related to known images can be generated based on technologies such as GAN and VAE, but the generated images are relatively rough in details and less realistic, and there is still a big gap in image quality compared to real images.

[0031] At present, the emerging technology diffusion model can improve the quality of image generation. It can generate new images by performing multiple rounds of denoising operations on random noise through a pre-trained diffusion model. Specifically, Figure 1As shown, during the first round of denoising operation, the random noise image S01 and the conditional vector R01 can be input into the pre-trained diffusion model M0, wherein the conditional vector R01 includes a time series vector obtained based on the round sequence of the denoising operation. Under the guidance of the conditional vector R01, the diffusion model M0 can process the random noise image S01, predict the noise to be removed corresponding to the first round of denoising operation, and denoise the random noise image S01 based on the noise to be removed to obtain the intermediate image S02.

[0032] During the second round of denoising, the intermediate image S02 and the conditional vector R02 can be input into the diffusion model M0. Under the guidance of the conditional vector R02, the diffusion model M0 can process the intermediate image S02, predict the noise to be removed corresponding to the second round of denoising, and perform denoising on the intermediate image S02 based on the noise to be removed to obtain the intermediate image S03. The following similar processes are analogous and will not be repeated here. Finally, after n rounds of denoising, the target image A can be obtained.

[0033] In order to generate a new image based on a known image, in some schemes, some semantic feature vectors in the known image can be extracted, and the extracted semantic feature vectors can be added to the time series vector and input into the diffusion model as conditional vectors. However, since only some semantic feature vectors of the known image are involved in the generation of the image in this scheme, the generated new image only has some of the same semantic features as the known image, and it is impossible to incorporate more details of the known image into the generated new image, which has certain limitations.

[0034] The present disclosure provides an image processing method, which can fuse a known first image and a random first noise image to obtain a second image, and based on the fused second image, perform multiple rounds of denoising operations on the random second noise image to obtain a target image. Among them, the multiple rounds of denoising operations can control the target image to have preset features, and the second image fused with the known first image can make the generated target image have the same style as the first image. In this way, more details of the known image can be integrated into the generated new image, making the generated image more realistic, improving the quality of the generated image, and enhancing the user experience.

[0035] See also Figure 2 , is a schematic diagram of an image processing scene according to an exemplary embodiment. Figure 2 , the scheme of the present disclosure is schematically illustrated.

[0036] like Figure 2As shown, first, a first image P10 can be obtained as a reference image for generating a new image, and a random noise image S10 and a random noise image S21 can be obtained. The random noise image S10 and the random noise image S21 can be the same or different. Then, based on the first image P10 and the random noise image S10, multiple rounds of denoising operations are performed on the random noise image S21.

[0037] Specifically, before performing multiple rounds of denoising operations, for different rounds of denoising operations, the first image P10 and the random noise image S10 can be fused at different ratios to obtain the second image corresponding to each round of denoising operations. For example, for the first round of denoising operation, the first image P10 and the random noise image S10 are fused at a ratio of P1 to obtain the second image P11. For the second round of denoising operation, the first image P10 and the random noise image S10 are fused at a ratio of P2 to obtain the second image P12. ...For the nth round of denoising operation, the first image P10 and the random noise image S10 are fused at a ratio of Pn to obtain the second image P1n. Similar processes are analogous and will not be repeated here.

[0038] Then, the second image P11, the second image P12, ... and the second image P1n can be respectively input into the pre-trained model M1 to obtain the noise data to be removed corresponding to each round of denoising operation. For example, for the first round of denoising operation, the model M1 can process the second image P11 under the guidance of the conditional vector corresponding to the first round of denoising operation, and predict the noise S11 to be removed corresponding to the first round of denoising operation. For the second round of denoising operation, the model M1 can process the second image P12 under the guidance of the conditional vector corresponding to the second round of denoising operation, and predict the noise S12 to be removed corresponding to the second round of denoising operation. ...For the nth round of denoising operation, the model M1 can process the second image P1n under the guidance of the conditional vector corresponding to the nth round of denoising operation, and predict the noise S1n to be removed corresponding to the nth round of denoising operation. Similar processes are analogous, and are not repeated here. Among them, the conditional vector corresponding to any round of denoising operation includes a time series vector obtained based on the round sequence number of the round of denoising operation.

[0039] Finally, based on the noise to be removed S11, the noise to be removed S12, ... and the noise to be removed S1n, the random noise image S21 can be subjected to n rounds of denoising operations to obtain the target image B. Specifically, in the first round of denoising operations, the random noise image S21 is denoised based on the noise to be removed S11 to obtain the intermediate image S22. In the second round of denoising operations, the intermediate image S22 is denoised based on the noise to be removed S12 to obtain the intermediate image S23. ... In the nth round of denoising operations, the intermediate image S2n is denoised based on the noise to be removed S1n to obtain the target image B.

[0040] It should be noted that the model M1 can be trained using paired sample images, and the paired sample images can be a pair of related images, for example, the paired sample images can include sample image 1 and sample image 2, wherein sample image 1 can be an image under any condition, and sample image 2 is an image related to sample image 1 and including specified features. For example, sample image 1 can be an image of a cartoon character C in a standing posture, and sample image 2 can be an image of a cartoon character C in a flying posture.

[0041] A known image may be obtained as sample image 1, and a specified modification may be made to sample image 1, so that some features in sample image 1 are modified into specified features, thereby obtaining sample image 2 paired with sample image 1. Sample image 1 may also be directly drawn, and sample image 2 paired with sample image 1 may also be directly drawn. It is to be understood that this embodiment does not limit the specific method of obtaining paired sample images.

[0042] After obtaining the paired sample image 1 and sample image 2, the sample image 2 can be subjected to multiple steps of noise processing, adding a small amount of Gaussian white noise in each step of noise processing to obtain multiple intermediate results and the final noise image. Then, the sample image 1 and the randomly generated noise image are processed according to Figure 2 The information flows shown are respectively input into the corresponding models to be trained, the noise to be removed is predicted, and the prediction loss is calculated based on the intermediate results of the above-mentioned noise addition processing and the final noise image. The model parameters of the model to be trained are adjusted according to the prediction loss, thereby completing the training of the model.

[0043] The present disclosure will be described in detail below in conjunction with specific embodiments.

[0044] Figure 3 The flowchart of an image generation method according to an exemplary embodiment is shown. The execution subject of the method can be implemented as any device, platform, server or device cluster with computing and processing capabilities. The method includes the following steps:

[0045] like Figure 3As shown, in step 301, a first image, a first noise image and a second noise image are acquired.

[0046] In this embodiment, the first image is a known image used as a reference when generating a new image. The first image can be a real physical image, an artificially drawn image, or an image generated by any generation method. It can be understood that this embodiment does not limit the specific type of the first image. The first noise image and the second noise image can be randomly generated white noise images. The first noise image and the second noise image can be the same noise image or different noise images.

[0047] In step 302, the first image and the first noise image are fused to obtain a second image, and in step 303, based on the second image, multiple rounds of denoising operations are performed on the second noise image to obtain a target image with the same style as the first image and preset features.

[0048] In this embodiment, multiple rounds of denoising operations can be performed on the second noise image based on the first image and the first noise image to obtain a target image. The target image can be an image with the same style as the first image and with preset features. For example, the first image can be an image of a cartoon character with a standing posture, and the target image can be an image of the same cartoon character with a flying posture. The same cartoon character can be the same style as the first image in the target image, and the flying posture can be a preset feature of the target image.

[0049] Specifically, before performing multiple rounds of denoising operations, the first image and the first noise image may be fused to obtain a second image, and then multiple rounds of denoising operations may be performed on the second noise image based on the second image. In one implementation, the first image and the first noise image may be fused according to a preset ratio to obtain a frame of the second image. Then, a first round of denoising operations is performed based on the frame of the second image, and subsequent denoising operations are performed based on the results of the first round of denoising operations.

[0050] In another implementation, the first image and the first noise image may be fused at different ratios for different rounds of denoising operations to obtain a second image corresponding to each round of denoising operations, wherein the second images corresponding to different rounds of denoising operations are also different. Then, based on the second images corresponding to each round of denoising operations, each round of denoising operations is performed on the second noise image.

[0051] The present disclosure provides an image processing method, which obtains a second image by fusing a first image and a first noise image, and performs multiple rounds of denoising operations on the second noise image based on the fused second image to obtain a target image. The multiple rounds of denoising operations can control the target image to have preset features, and the second image fused with the first image can make the generated target image have the same style as the first image. In this way, more details of the first image can be integrated into the generated target image, making the generated first image more realistic, improving the quality of the generated image, and enhancing the user experience.

[0052] Figure 4 is a flow chart of another image processing method according to an exemplary embodiment, which describes a process of fusing a first image with a first noise image and performing multiple rounds of denoising operations on the second noise image, including the following steps:

[0053] like Figure 4 As shown, in step 401, a first image, a first noise image and a second noise image are obtained, and in step 402, the first image and the first noise image are fused according to different proportions to obtain the second image corresponding to each round of denoising operation.

[0054] In this embodiment, first, the fusion ratio information of the first image and the first noise image in each round of denoising operation can be determined. Specifically, in each round of denoising operation, the first image and the first noise image have a fusion ratio corresponding to the round sequence number, and the fusion ratio is different in different rounds of denoising operation. For example, the larger the round sequence number corresponding to the denoising operation, the larger the first weight occupied by the first image in the round of denoising operation, and the smaller the second weight occupied by the first noise image in the round of denoising operation. That is, as the round sequence number corresponding to the denoising operation increases, the first weight occupied by the first image becomes larger and larger, and the second weight occupied by the first noise image becomes smaller and smaller. According to the round sequence number corresponding to each round of denoising operation, the first weight occupied by the first image and the second weight occupied by the first noise image in each round of denoising operation can be determined as the fusion ratio information.

[0055] In some embodiments, the fusion ratio of the first image and the first noise image in each round of denoising operation can be determined based on the hyperparameters of the target model corresponding to each round of denoising operation. For example, for the tth round of denoising operation, the first weight and the second weight can be determined by the following formula:

[0056] in, ω2=1-ω1

[0057] Among them, α sRepresents the hyperparameter of the target model corresponding to the s-th round of denoising operation. ω1 can be used as the first weight of the first image in the t-th round of denoising operation, or a value close to ω1 can be selected as the first weight of the first image in the t-th round of denoising operation, and ω2 can be used as the second weight of the first noise image in the t-th round of denoising operation.

[0058] In this embodiment, the round sequence number corresponding to each round of denoising operation and the fusion ratio information of the first image and the first noise image in each round of denoising operation can be stored in association in advance. When obtaining the fusion ratio information corresponding to any round of denoising operation, the fusion ratio information associated with the round sequence number can be directly obtained from the pre-stored data based on the round sequence number corresponding to the round of denoising operation.

[0059] Next, based on the fusion ratio information of each round of denoising operation, the pixel values ​​of the first image and the pixel values ​​of the first noise image may be weightedly fused to obtain a second image corresponding to each round of denoising operation. Specifically, for any round of denoising operation, the pixel values ​​of the pixel points at the same position in the first image and the first noise image may be weightedly summed according to the first weight corresponding to the first image in the round of denoising operation and the second weight corresponding to the first noise image, thereby obtaining the second image corresponding to the round of denoising operation.

[0060] Since this embodiment performs pixel fusion on the first image and the first noise image to obtain the second image for denoising operation, more known details can be integrated into the target image, which helps to improve the quality of the generated image.

[0061] In step 403, based on the second image corresponding to each round of denoising operation, the noise data to be removed corresponding to each round of denoising operation is determined.

[0062] In this embodiment, the condition vector corresponding to each round of denoising operation can be obtained first. The condition vector corresponding to any round of denoising operation can be determined based on the round sequence number corresponding to the round denoising operation, or based on the round sequence number corresponding to the round denoising operation and other condition information. For example, for any round of denoising operation, the round sequence number corresponding to the round denoising operation can be obtained, and the timing vector can be calculated based on the round sequence number, and the timing vector can be used as the condition vector corresponding to the round denoising operation. Other condition information can also be obtained, and an additional vector can be calculated based on the other condition information, and the timing vector and the additional vector are merged and added, and the vector after the merging and addition is used as the condition vector corresponding to the round of denoising operation.

[0063] Then, based on the conditional vector corresponding to each round of denoising operation, the target model can be used to process the second image corresponding to each round of denoising operation to obtain the noise data to be removed corresponding to each round of denoising operation, wherein the target model can be a diffusion model pre-trained for preset features, so the target model corresponds to the preset features, and the target image finally generated can have the preset features. Specifically, for any round of denoising operation, the conditional vector corresponding to the round of denoising operation and the second image corresponding to the round of denoising operation can be input into the target model, and the target model can, under the guidance of the conditional vector, use the model parameters corresponding to the round of denoising operation to process the second image to obtain the noise data to be removed corresponding to the round of denoising operation.

[0064] Since this implementation processes the second image fused with the known first image through the target model under the guidance of the conditional vector and predicts the noise data to be removed, the effect of image generation is further improved.

[0065] In step 404, multiple rounds of denoising operations are performed on the second noisy image based on the above-mentioned noise data to be removed.

[0066] In this embodiment, performing multiple rounds of denoising operations on the second noise image may include iteratively performing each of the following denoising operations: first, determining the image to be processed corresponding to the current round of denoising operations, wherein the image to be processed corresponding to the first round of denoising operations is the second noise image, and the image to be processed corresponding to the non-first round of denoising operations is the processing result of the previous round of denoising operations. For example, the image to be processed corresponding to the second round of denoising operations is the processing result of the first round of denoising operations, and the image to be processed corresponding to the third round of denoising operations is the processing result of the second round of denoising operations, etc. Then, based on the noise data to be removed corresponding to the current round of denoising operations, denoising is performed on the image to be processed.

[0067] It should be noted that Figure 4 The process described in the embodiment may be an inference process of generating a target image based on a known first image, or may be a training process of training a target model (ie, an image editing model).

[0068] Specifically, when training the target model, step 401 and step 402 can be performed first, and the paired image corresponding to the first image can be obtained. Then, multiple rounds of denoising operations are performed on the paired image to obtain the denoising results corresponding to each round of denoising operations. Next, the following steps are performed iteratively for multiple times until the conditions for stopping the iteration are met: step 403, step 404, based on the denoising results corresponding to each round of denoising operations, the noise data added by each round of denoising operations, and the noise data to be removed corresponding to each round of denoising operations, the target loss is determined, and the parameters of the target model are adjusted based on the target loss. Among them, the number of rounds of denoising operations is the same as the number of rounds of denoising operations. When the target model training is completed, the image obtained by performing multiple rounds of denoising operations on the second noise image is the target image, and the similarity between the target image and the paired image is higher than the preset threshold.

[0069] The present disclosure provides an image processing method, which fuses a first image and a first noise image according to different proportions to obtain a second image corresponding to each round of denoising operation, determines the noise data to be removed corresponding to each round of denoising operation based on the second image corresponding to each round of denoising operation, and performs multiple rounds of denoising operations on the second noise image based on the noise data to be removed. Since the first image and the first noise image are fused according to different proportions, the probability distribution of the pixel values ​​of the second image to be denoised is closer to the image to be denoised, thereby improving the accuracy of the prediction, and further improving the effect of image generation, which helps to improve the user experience.

[0070] It should be noted that although the operations of the method of the embodiment of the present disclosure are described in a specific order in the above embodiment, this does not require or imply that these operations must be performed in this specific order, or that all the operations shown must be performed to achieve the desired result. On the contrary, the steps depicted in the flowchart can change the order of execution. For example, step 403 can be performed first, and then step 404 can be performed, or step 403 and step 404 can be performed simultaneously. Additionally or alternatively, some steps can be omitted, multiple steps can be combined into one step for execution, and / or one step can be decomposed into multiple steps for execution.

[0071] Corresponding to the aforementioned image processing method embodiment, the present disclosure also provides an embodiment of an image processing device.

[0072] like Figure 5 As shown, Figure 5 is a block diagram of an image processing device according to an exemplary embodiment of the present disclosure. The device may include: an acquisition module 501, a fusion module 502 and a denoising module 503.

[0073] The acquisition module 501 is used to acquire a first image, a first noise image and a second noise image.

[0074] The fusion module 502 is used to fuse the first image and the first noise image to obtain a second image.

[0075] The denoising module 503 is used to perform multiple rounds of denoising operations on the second noisy image based on the second image to obtain a target image with the same style as the first image and preset features.

[0076] In some implementations, the fusion module 502 may include: a fusion submodule (not shown in the figure).

[0077] The fusion submodule is used to fuse the first image and the first noise image according to different proportions for different rounds of denoising operations to obtain a second image corresponding to each round of denoising operations.

[0078] In some other embodiments, the fusion submodule includes: a first determination submodule and a processing submodule (not shown in the figure).

[0079] The first determination submodule is used to determine the fusion ratio information of the first image and the first noise image in each round of denoising operation.

[0080] The processing submodule is used to perform weighted fusion processing on the pixel values ​​of the first image and the pixel values ​​of the first noise image based on the fusion ratio information of each round of denoising operation to obtain the second image corresponding to each round of denoising operation.

[0081] In some other embodiments, the determination submodule is configured to: determine, according to the round sequence number corresponding to each round of denoising operation, a first weight occupied by the first image and a second weight occupied by the first noise image in each round of denoising operation as fusion ratio information, wherein the first weight is positively correlated with the round sequence number, and the second weight is negatively correlated with the round sequence number.

[0082] In some other implementations, the denoising module 503 includes: a second determining submodule and a denoising submodule (not shown in the figure).

[0083] The second determining submodule is used to determine the noise data to be removed corresponding to each round of denoising operation based on the second image corresponding to each round of denoising operation.

[0084] The denoising submodule is used to perform the multiple rounds of denoising operations on the second noisy image based on the above-mentioned noise data to be removed.

[0085] In some other embodiments, the second determination submodule is configured to: obtain a condition vector corresponding to each round of denoising operation, the condition vector corresponding to any round of denoising operation being determined based on at least the round sequence number corresponding to each round of denoising operation. Based on the condition vector corresponding to each round of denoising operation, the second image corresponding to each round of denoising operation is processed using the target model to obtain the noise data to be removed corresponding to each round of denoising operation.

[0086] In some other embodiments, the denoising submodule iteratively performs the following denoising operations in each round: determining the image to be processed corresponding to the current round of denoising operation. The image to be processed corresponding to the first round of denoising operation is the second noise image, and the image to be processed corresponding to the non-first round of denoising operation is the processing result of the previous round of denoising operation. Based on the noise data to be removed corresponding to the current round of denoising operation, the image to be processed is denoised.

[0087] For the device embodiment, since it basically corresponds to the method embodiment, the relevant parts can refer to the partial description of the method embodiment. The device embodiment described above is only schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the embodiment scheme of the present disclosure. A person of ordinary skill in the art can understand and implement it without paying creative labor.

[0088] Figure 6 A schematic block diagram of an electronic device provided in some embodiments of the present disclosure. Figure 6 As shown, the electronic device 910 includes a processor 911 and a memory 912, which can be used to implement a client or a server. The memory 912 is used to store computer executable instructions (e.g., one or more computer program modules) non-transiently. The processor 911 is used to run the computer executable instructions, and when the computer executable instructions are run by the processor 911, one or more steps in the image processing method described above can be executed, thereby implementing the image processing method described above. The memory 912 and the processor 911 can be interconnected through a bus system and / or other forms of connection mechanisms (not shown).

[0089] For example, the processor 911 may be a central processing unit (CPU), a graphics processing unit (GPU), or other forms of processing units having data processing capabilities and / or program execution capabilities. For example, the central processing unit (CPU) may be an X86 or ARM architecture, etc. The processor 911 may be a general-purpose processor or a dedicated processor, and may control other components in the electronic device 910 to perform desired functions.

[0090] For example, the memory 912 may include any combination of one or more computer program products, and the computer program product may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) and / or cache memory (cache), etc. Non-volatile memory may include, for example, read-only memory (ROM), hard disk, erasable programmable read-only memory (EPROM), portable compact disk read-only memory (CD-ROM), USB memory, flash memory, etc. One or more computer program modules may be stored on the computer-readable storage medium, and the processor 911 may run one or more computer program modules to implement various functions of the electronic device 910. Various applications and various data, as well as various data used and / or generated by the application, etc. may also be stored in the computer-readable storage medium.

[0091] It should be noted that, in the embodiments of the present disclosure, the specific functions and technical effects of the electronic device 910 can refer to the above description of the image processing method, which will not be repeated here.

[0092] Figure 7 A schematic block diagram of another electronic device provided for some embodiments of the present disclosure. The electronic device 920 is, for example, suitable for implementing the image processing method provided in the embodiments of the present disclosure. The electronic device 920 may be a terminal device, etc., and may be used to implement a client or a server. The electronic device 920 may include, but is not limited to, mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), wearable electronic devices, etc., and fixed terminals such as digital TVs, desktop computers, smart home devices, etc. It should be noted that Figure 7 The electronic device 920 shown is only an example and does not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0093] like Figure 7 As shown, the electronic device 920 may include a processing device (e.g., a central processing unit, a graphics processor, etc.) 921, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 922 or a program loaded from a storage device 928 into a random access memory (RAM) 923. In the RAM 923, various programs and data required for the operation of the electronic device 920 are also stored. The processing device 921, the ROM 922, and the RAM 923 are connected to each other via a bus 924. An input / output (I / O) interface 925 is also connected to the bus 924.

[0094] Typically, the following devices may be connected to the I / O interface 925: an input device 926 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 927 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 928 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 929. The communication device 929 may allow the electronic device 920 to communicate with other electronic devices wirelessly or by wire to exchange data. Although Figure 7 The electronic device 920 is shown to have various devices, but it should be understood that it is not required to implement or possess all of the devices shown, and the electronic device 920 may alternatively implement or possess more or fewer devices.

[0095] For example, according to an embodiment of the present disclosure, the above-mentioned image processing method can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program includes a program code for executing the above-mentioned image processing method. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device 929, or installed from a storage device 928, or installed from a ROM 922. When the computer program is executed by the processing device 921, the functions defined in the image processing method provided in the embodiment of the present disclosure can be implemented.

[0096] Figure 8 A schematic diagram of a storage medium provided in some embodiments of the present disclosure. Figure 8 As shown, the storage medium 930 may be a non-transitory computer-readable storage medium for storing non-transitory computer-executable instructions 931. When the non-transitory computer-executable instructions 931 are executed by a processor, the image processing method described in the embodiment of the present disclosure may be implemented. For example, when the non-transitory computer-executable instructions 931 are executed by a processor, one or more steps in the image processing method described above may be executed.

[0097] For example, the storage medium 930 may be applied to the above-mentioned electronic device. For example, the storage medium 930 may include a memory in the electronic device.

[0098] For example, the storage medium may include a memory card of a smart phone, a storage component of a tablet computer, a hard disk of a personal computer, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a portable compact disk read-only memory (CD-ROM), flash memory, or any combination of the above storage media, or other applicable storage media.

[0099] For example, the description of the storage medium 930 can refer to the description of the memory in the embodiment of the electronic device, and the repeated parts are not repeated. The specific functions and technical effects of the storage medium 930 can refer to the description of the image processing method above, and are not repeated here.

[0100] It should be noted that in the context of the present disclosure, a computer-readable medium may be a tangible medium that may contain or store a program for use by an instruction execution system, device or equipment or used in combination with an instruction execution system, device or equipment. A computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. A computer-readable storage medium may be, for example, but not limited to: an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium that contains or stores a program that may be used by an instruction execution system, device or device or used in combination with it. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, device, or device. The program code contained on the computer-readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0101] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. The present disclosure is intended to cover any variations, uses or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The description and examples are to be considered exemplary only, and the true scope and spirit of the present disclosure are indicated by the claims.

[0102] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. An image processing method, the method comprising: Acquire a first image, a first noise image, and a second noise image; fusing the first image with the first noise image to obtain a second image; Based on the second image, multiple rounds of denoising operations are performed on the second noisy image to obtain a target image with the same style as the first image and preset features.

2. The method according to claim 1, wherein: The fusing the first image and the first noise image to obtain a second image includes: For different rounds of denoising operations, the first image and the first noise image are fused according to different proportions to obtain a second image corresponding to each round of denoising operation.

3. The method according to claim 2, wherein: The denoising operation for different rounds, fusing the first image and the first noise image according to different proportions to obtain a second image corresponding to each round of denoising operation, includes: Determining fusion ratio information of the first image and the first noise image in each round of denoising operation; Based on the fusion ratio information of each round of denoising operation, weighted fusion processing is performed on the pixel values ​​of the first image and the pixel values ​​of the first noise image to obtain a second image corresponding to each round of denoising operation.

4. The method according to claim 3, wherein: The determining the fusion ratio information of the first image and the first noise image in each round of denoising operation includes: According to the round sequence number corresponding to each round of denoising operation, a first weight of the first image and a second weight of the first noise image in each round of denoising operation are determined as the fusion ratio information; wherein the first weight is positively correlated with the round sequence number, and the second weight is negatively correlated with the round sequence number.

5. The method according to claim 2, wherein: The performing multiple rounds of denoising operations on the second noisy image based on the second image includes: Determining the noise data to be removed corresponding to each round of denoising operation based on the second image corresponding to each round of denoising operation; Based on the to-be-removed noise data, the multiple rounds of denoising operations are performed on the second noisy image.

6. The method according to claim 5, wherein: The determining, based on the second image corresponding to each round of denoising operation, the noise data to be removed corresponding to each round of denoising operation comprises: Acquire the condition vector corresponding to each round of denoising operation; the condition vector corresponding to any round of denoising operation is determined based at least on the round sequence number corresponding to the round of denoising operation; Based on the conditional vector corresponding to each round of denoising operation, the second image corresponding to each round of denoising operation is processed using the target model to obtain the noise data to be removed corresponding to each round of denoising operation.

7. The method according to claim 5, wherein the performing the multiple rounds of denoising operations on the second noisy image based on the to-be-denoised data comprises iteratively performing each of the following rounds of denoising operations: Determine the image to be processed corresponding to the current round of denoising operation; where, The image to be processed corresponding to the first round of denoising operation is the second noise image, and the image to be processed corresponding to the non-first round of denoising operation is the processing result of the previous round of denoising operation; Based on the noise data to be removed corresponding to the current round of denoising operation, denoising is performed on the image to be processed.

8. The method according to claim 1, wherein: The method is used in a training process of an image editing model, and the method further comprises: Acquire a paired image of the first image; the paired image is related to the target image; Performing multiple rounds of denoising operations on the paired images to obtain denoising results corresponding to each round of denoising operations; wherein the number of rounds of the denoising operation is the same as the number of rounds of the denoising operation; Determine a target loss based on the noise addition result corresponding to each round of noise addition operation, the noise data added by each round of noise addition operation, and the noise data to be removed corresponding to each round of noise removal operation; The parameters of the image editing model are adjusted based on the target loss so that the similarity between the target image and the paired image is higher than a preset threshold.

9. An image processing device, comprising: An acquisition module, used for acquiring a first image, a first noise image and a second noise image; a fusion module, used for fusing the first image and the first noise image to obtain a second image; A denoising module is used to perform multiple rounds of denoising operations on the second noisy image based on the second image to obtain a target image with the same style as the first image and preset features.

10. A computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to execute the method according to any one of claims 1 to 8.

11. An electronic device comprising a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, the method according to any one of claims 1 to 8 is implemented.

Images