Method executed by electronic device, electronic device, storage medium and program product

Abstract

The invention provides a method executed by electronic equipment, the electronic equipment, a storage medium and a program product, and relates to the fields of computer vision, image processing, artificial intelligence and the like. The method comprises the steps that a first image is obtained, the first image is an image obtained after noise is added to a second image, and the second image comprises a target repairing area; using a first artificial intelligence AI network to perform at least one first denoising processing on the first image to obtain a first denoising processing result; and using a second AI network to repair the target repair area based on the first denoising processing result to obtain a repaired image. Optionally, the method performed by the electronic device may be performed using an artificial intelligence model.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a method executed by an electronic device, an electronic device, a storage medium, and a program product. Background Art

[0002] Image restoration is an important and challenging research direction in the field of image processing. Its purpose is to generate new content in the missing areas of images with missing content, thereby restoring a natural and complete image. However, the existing image restoration models generally have very high computational overhead, resulting in low efficiency of image restoration. Summary of the invention

[0003] The purpose of the embodiments of the present application is to solve the technical problem of low efficiency in image restoration.

[0004] According to one aspect of an embodiment of the present application, a method performed by an electronic device is provided, the method comprising:

[0005] Acquire a first image, where the first image is an image obtained by adding noise to the second image, and the second image includes a target repair area;

[0006] Using a first AI (Artificial Intelligence) network, performing at least one first denoising process on the first image to obtain a first denoising process result;

[0007] Using the second AI network, based on the first denoising result, the target repair area is repaired to obtain a repaired image.

[0008] According to another aspect of the embodiments of the present application, a method performed by an electronic device is also provided, the method comprising:

[0009] Get the target repair area selected by the user in the image;

[0010] Providing the user with category information of the repaired content, and obtaining the target category information selected by the user;

[0011] Using the first AI network, based on the target category information selected by the user, the target repair area in the image is repaired;

[0012] Using the second AI network, repairing the repair result of the first AI network;

[0013] Provide the restored image to the user.

[0014] According to another aspect of the embodiments of the present application, a method performed by an electronic device is also provided, the method comprising:

[0015] Acquire a first image, where the first image is an image obtained by adding noise to the second image, and the second image includes a target repair area;

[0016] Using the attention module, by Fourier transform, the attention corresponding to the first image is calculated, where the attention represents the degree of denoising of the first image;

[0017] Perform at least one third denoising process based on the attention calculation result on the first image to obtain an image after the target repair area is repaired.

[0018] According to another aspect of an embodiment of the present application, an electronic device is provided, which includes: a memory, a processor, and a computer program stored in the memory, and the processor executes the computer program to implement the method provided in the embodiment of the present application.

[0019] According to another aspect of an embodiment of the present application, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the method provided in the embodiment of the present application is implemented.

[0020] According to another aspect of an embodiment of the present application, a computer program product is provided. The computer program product includes a computer program. When the computer program is executed by a processor, the method provided in the embodiment of the present application is implemented.

[0021] The embodiments of the present application provide a method, an electronic device, a storage medium, and a program product executed by an electronic device. The method comprises acquiring a first image, wherein the first image is an image after adding noise to a second image, and the second image includes a target repair area; performing a first denoising process on the first image at least once using a first artificial intelligence (AI) network to obtain a first denoising process result; and performing a repair process on the target repair area based on the first denoising process result using a second AI network to obtain a repaired image. That is, the embodiments of the present application use the first AI network to perform a first denoising process to obtain a first denoising process result, and then use the second AI network to process the first denoising process result, thereby significantly improving the image repair efficiency while also maintaining the image repair quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in describing the embodiments of the present application are briefly introduced below.

[0023] Figure 1 A schematic diagram of a method flow chart executed by an electronic device provided in an embodiment of the present application;

[0024] Figure 2a A schematic diagram of a first AI network for removing noise provided in an embodiment of the present application;

[0025] Figure 2b A schematic diagram of a first AI network training method provided in an embodiment of the present application;

[0026] Figure 3a A schematic diagram of a second AI network provided in an embodiment of the present application;

[0027] Figure 3b A schematic diagram of a second AI network training method provided in an embodiment of the present application;

[0028] Figure 4a A schematic diagram of a denoising process based on a content feature map provided in an embodiment of the present application;

[0029] Figure 4b A schematic diagram of performing denoising based on a text feature map selected by a user provided in an embodiment of the present application;

[0030] Figure 4c A schematic diagram of performing denoising based on target restoration content-related information provided in an embodiment of the present application;

[0031] Figure 5 A schematic diagram of an attention module processing provided in an embodiment of the present application;

[0032] Figure 6 A schematic diagram of a first convolution operation process provided in an embodiment of the present application;

[0033] Figure 7 A schematic diagram of a second convolution operation process provided in an embodiment of the present application;

[0034] Figure 8 A schematic diagram of a first AI network architecture provided in an embodiment of the present application;

[0035] Fig. 9 A schematic diagram of a dynamic denoising process exit mechanism provided in an embodiment of the present application;

[0036] Fig.10 A schematic diagram of using a second AI network for processing provided in an embodiment of the present application;

[0037] Fig.11 A schematic diagram of another method of using a second AI network for processing provided in an embodiment of the present application;

[0038] Fig.12 A schematic diagram of using an RT module for processing provided in an embodiment of the present application;

[0039] Fig.13 A schematic diagram of a third convolution operation process provided in an embodiment of the present application;

[0040] Fig.14 A schematic diagram of a user selecting an area to remove an object provided in an embodiment of the present application;

[0041] Fig.15 A schematic diagram of an adaptive image cropping process provided in an embodiment of the present application;

[0042] Fig.16 A schematic diagram of a complete image restoration process provided by an embodiment of the present application;

[0043] Fig.17 A schematic diagram of a user editing phase processing flow provided in an embodiment of the present application;

[0044] Fig.18a A flowchart of another method performed by an electronic device provided in an embodiment of the present application;

[0045] Fig.18b A flowchart of another method performed by an electronic device provided in an embodiment of the present application;

[0046] Fig.19 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0047] The following description with reference to the accompanying drawings is provided to facilitate a comprehensive understanding of the various embodiments of the present application as defined by the claims and their equivalents. This description includes various specific details to facilitate understanding but should be considered as exemplary only. Therefore, one of ordinary skill in the art will recognize that various changes and modifications can be made to the various embodiments described herein without departing from the scope and spirit of the present application. In addition, for the sake of clarity and conciseness, descriptions of well-known functions and structures may be omitted.

[0048] The terms and expressions used in the following specification and claims are not limited to their dictionary meanings, but are merely used by the inventor to enable a clear and consistent understanding of the present application. Therefore, it should be apparent to those skilled in the art that the following description of various embodiments of the present application is provided for illustration purposes only and not to limit the purpose of the present application as defined in the appended claims and their equivalents.

[0049] It should be understood that the singular forms "a", "an", and "the" may also include plural references unless the context clearly indicates otherwise. Thus, for example, reference to "a component surface" includes reference to one or more such surfaces. When we refer to an element as being "connected" or "coupled" to another element, the one element may be directly connected or coupled to the other element, or the one element and the other element may establish a connection relationship through an intermediate element. In addition, "connected" or "coupled" as used herein may include wireless connection or wireless coupling.

[0050] The term "include" or "may include" refers to the presence of corresponding disclosed functions, operations, or components that can be used in various embodiments of the present application, rather than limiting the presence of one or more additional functions, operations, or features. In addition, the term "include" or "have" may be interpreted as indicating certain characteristics, numbers, steps, operations, constituent elements, components, or combinations thereof, but should not be interpreted as excluding the possibility of the presence of one or more other characteristics, numbers, steps, operations, constituent elements, components, or combinations thereof.

[0051] The term "or" used in various embodiments of the present application includes any of the listed terms and all combinations thereof. For example, "A or B" may include A, may include B, or may include both A and B. When describing multiple (two or more) items, if the relationship between the multiple items is not clearly defined, the multiple items may refer to one, multiple, or all of the multiple items. For example, the description of "parameter A includes A1, A2, A3" may be implemented as parameter A including A1 or A2 or A3, and may also be implemented as parameter A including at least two of the three items A1, A2, and A3.

[0052] Unless defined differently, all terms (including technical terms or scientific terms) used in this application have the same meanings understood by those skilled in the art of this application. Common terms as defined in dictionaries are interpreted as having meanings consistent with the context in the relevant technical field, and should not be interpreted ideally or overly formally, unless clearly defined in this application.

[0053] At least some functions of the device or electronic device provided in the embodiments of the present application can be implemented by an AI model, such as at least one module among multiple modules of the device or electronic device can be implemented by an AI model. Functions associated with AI can be performed by non-volatile memory, volatile memory and processor.

[0054] The processor may include one or more processors. In this case, the one or more processors may be general-purpose processors, such as a central processing unit (CPU), an application processor (AP), etc., or pure graphics processing units, such as a graphics processing unit (GPU), a visual processing unit (VPU), and / or an AI-specific processor, such as a neural processing unit (NPU).

[0055] The one or more processors control the processing of input data according to predefined operating rules or artificial intelligence (AI) models stored in non-volatile memory and volatile memory. The predefined operating rules or artificial intelligence models are provided by training or learning.

[0056] Here, providing by learning means obtaining a predefined operating rule or an AI model with desired characteristics by applying a learning algorithm to a plurality of learning data. The learning can be performed in the device or electronic device itself in which the AI ​​according to the embodiment is executed, and / or can be implemented by a separate server / system.

[0057] The AI ​​model may include multiple neural network layers. Each layer has multiple weight values, and each layer performs neural network calculations by calculating between the input data of the layer (such as the calculation results of the previous layer and / or the input data of the AI ​​model) and the multiple weight values ​​of the current layer. Examples of neural networks include, but are not limited to, convolutional neural networks (CNNs), deep neural networks (DNNs), recurrent neural networks (RNNs), restricted Boltzmann machines (RBMs), deep belief networks (DBNs), bidirectional recurrent deep neural networks (BRDNNs), generative adversarial networks (GANs), and deep Q networks.

[0058] A learning algorithm is a method of using a plurality of learning data to train a predetermined target device (e.g., a robot) to enable, allow, or control the target device to make a determination or prediction. Examples of the learning algorithm include, but are not limited to, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning.

[0059] The method provided in this application may involve one or more technical fields such as speech, language, image, video or data intelligence.

[0060] Optionally, when it comes to the field of images or videos, according to the present application, in a method performed in an electronic device, a method for image repair can obtain output data after the target repair area in the image is repaired by using image data as input data of an artificial intelligence model. The artificial intelligence model can be obtained through training. Here, "obtained by training" means obtaining a predefined operation rule or artificial intelligence model configured to perform the desired feature (or purpose) by training a basic artificial intelligence model with multiple training data through a training algorithm. The method of the present application may relate to the field of visual understanding of artificial intelligence technology, which is a technology for identifying and processing things like human vision, and includes, for example, object recognition, object tracking, image retrieval, human recognition, scene recognition, 3D reconstruction / positioning, or image enhancement.

[0061] The following describes several optional embodiments to illustrate the technical solutions of the embodiments of the present application and the technical effects produced by the technical solutions of the present application. It should be noted that the following embodiments can refer to, draw on or combine with each other, and the same terms, similar features and similar implementation steps in different embodiments will not be described repeatedly.

[0062] In an embodiment of the present application, a method executed by an electronic device is provided, such as Figure 1 As shown, the method includes:

[0063] Step S101: Acquire a first image, where the first image is an image obtained by adding noise to a second image, and the second image includes a target repair area;

[0064] In an embodiment of the present application, the target repair area refers to the area in the image to be repaired (also referred to as restoration), for example, it may be a lost area or a removed area, but is not limited thereto. Among them, the lost area may refer to an area where some information in the image is missing or damaged due to various factors such as occlusion, blur, transmission interference, etc. The removed area may refer to a blank area formed after erasing part of the target object (such as an object, a person, a building, etc., but not limited thereto) in the image. The erased target object may be automatically selected or manually selected, and the embodiment of the present application is not limited thereto. Optionally, a corresponding mask image may be determined for the target repair area, which is used to determine the position of the target repair area in various images.

[0065] In the embodiment of the present application, the second image refers to the image to be repaired including the target repair area. Optionally, the second image can directly use the image including the original target repair area, or the image including the original target repair area can be processed to a certain extent and then used as the second image for repair processing. As an example, if the area of ​​the original target repair area is too large, the image including the original target repair area can be cut to reduce the area of ​​the target repair area in the cut image, and the cut image can be used as a second image for repair processing; as an example, if the size of the image including the original target repair area is too large, the image including the original target repair area can also be cut or scaled or deformed to reduce the size of the cut image, and the cut image can be used as a second image for repair processing, thereby reducing the amount of calculation, which is not limited in the embodiment of the present application.

[0066] In the embodiment of the present application, the first image is an image after adding noise (noise) to the second image, that is, the noise and the second image are fused, and the first image obtained can be understood as an image completely transformed into noise. The added noise can be generated randomly or according to a predetermined algorithm, and the fusion method can be channel splicing or superposition, etc. The implementation of the present application does not specifically limit the noise generation method and fusion method.

[0067] Step S102: using the first AI network, performing at least one first denoising process on the first image to obtain a first denoising process result;

[0068] In the embodiment of the present application, for the input first image, the first AI network can be used to remove noise step by step, such as Figure 2a As shown, optionally, the first AI network can adopt a diffusion network, but is not limited thereto. The first AI network can only repair a little content at a time, and the current content repair is performed on the basis of the previous repair result. Figure 2a Taking 10% repair each time as an example, in actual applications, the degree of each repair of the first AI network (also known as the dynamic diffusion denoising time step) can be achieved by setting different training methods according to actual conditions. The embodiment of the present application does not specifically limit the degree of each repair of the first AI network.

[0069] Optionally, the first AI network can be trained in the following manner: obtain training samples, continuously add noise to sample images containing the target repair area, guide them to gradually become completely noise, and retain each image with added noise as a training sample, which is used to calculate the loss in the training stage to update the first AI network. Furthermore, in the process of using the training samples to train the first AI network, each time the first AI network only needs to predict the noise to be removed from the current input image, for example Figure 2b As shown, the 70% denoised image is input into the first AI network. The first AI network only needs to predict the 80% denoised image and compare it with the sample image with 20% noise added to calculate the loss to update the current model parameters of the first AI network. The updated first AI network is then used for prediction. Similar prediction and model training processes can be deduced by analogy and will not be described in detail here.

[0070] That is, in the embodiment of the present application, the first AI network converts the image restoration problem into a prediction problem. Each time denoising is performed, as long as the amount of noise remaining in the current input image and its denoising result can be accurately predicted, if the image is completely denoised, a restored image can be obtained, and a good restoration effect can be achieved. In other words, the first denoising processing result can include an image after the noise is completely removed using the first AI network.

[0071] Alternatively, the first denoising result may include an intermediate denoising result after using the first AI network to remove a certain degree of noise. This is to take into account that as the first AI network is continuously used to remove noise, the noise in the denoising result becomes less and less, while the restored semantic content becomes more and more. When the noise content is low to a certain level, using the first AI network again will increase the delay of the method. Therefore, the embodiment of the present application can set a judgment condition so that when the denoising result reaches a predetermined condition, the first AI network will no longer be used to continue the denoising process. In actual applications, those skilled in the art can set the predetermined conditions according to actual conditions, such as the amount of residual noise reaches a first predetermined value, the image quality (such as clarity, completeness, etc., but not limited to this) reaches a second predetermined value, the number of denoising processes reaches a third predetermined value, etc., and the embodiment of the present application is not limited here.

[0072] Step S103: Using the second AI network, based on the first denoising result, the target repair area is repaired to obtain a repaired image.

[0073] In an embodiment of the present application, the second AI network may have a different repair method than the first AI network. For example, the first AI network is a prediction type network model, and the second AI network may be a generation type network model, but is not limited thereto. The processing process of the generation type network model can be understood as using pixels from a known area to expand to an unknown area. Optionally, the second AI network may adopt a GAN (Generative Adversarial Networks) network. As an example, the second AI network is a generator composed of an encoder and a decoder. The first denoising process result is input into the generator for repair processing to obtain a repaired image, such as Figure 3aAs shown. In the restored image, the target restoration area has been filled with some content. Optionally, the training of the generator can be performed based on the loss value of the generated sample image and the original sample image calculated by the discriminator, such as Figure 3b As shown, it is obtained through adversarial training between the generator and the discriminator.

[0074] In other embodiments, the type of the second AI network is not limited thereto, and may also be other neural network models. Those skilled in the art may set it according to actual conditions, and the embodiments of the present application are not limited thereto.

[0075] In the embodiment of the present application, the first denoising result has good semantic content and may have some noise points. The restoration processing of the second AI network can better maintain the restoration quality of the image, such as removing noise points other than semantic content, and can significantly reduce latency and improve image restoration efficiency compared to using only the first AI network.

[0076] The method provided by the embodiment of the present application and executed by the electronic device uses a first AI network to perform a first denoising process to obtain a first denoising process result, and then uses a second AI network to process the first denoising process result, thereby significantly improving the image restoration efficiency while also maintaining the image restoration quality.

[0077] In the embodiment of the present application, before step S101, the following steps may also be included:

[0078] Step S201: receiving a first instruction from a user to select a target repair area in a third image;

[0079] Step S202: Based on the first instruction, determine a target repair area and a second image including the target repair area.

[0080] The third image refers to the image before the target repair area is removed from the second image.

[0081] As an example, after the user opens the application and selects the third image, the user can select the object area to be removed in the third image. Fig.14 As shown, after the object area is removed, a second image including the target repair area can be formed, and the removed object area is the target repair area.

[0082] In the embodiment of the present application, an optional implementation is provided for step S102, which may specifically include:

[0083] Step S1021: Determine target repair content related information corresponding to the target repair area;

[0084] In the embodiment of the present application, the repair content (or filling content) for the target repair area is settable. Optionally, the specific type of the repair content is represented by a content feature map, that is, the target repair content related information includes the content feature map corresponding to the target repair content. As an example, if the desired repair content is "flower", the target repair content related information can be a feature map corresponding to the image containing the content "flower". As another example, if the desired repair content is a clean background, the target repair content related information can be a feature map corresponding to the blank content image, but is not limited to this. In actual applications, those skilled in the art can set the repairable content and the corresponding target repair content related information according to actual conditions, and the embodiment of the present application is not limited here.

[0085] Step S1022: Based on the information related to the target restoration content, use the first AI network to perform at least one first denoising process on the first image.

[0086] As an example, taking the target repair content-related information using a content feature map as an example, a process example of denoising based on a content feature map is as follows: Figure 4a As shown, it may specifically include: inputting the second image (or the scaled second image) into a VAE (Variational Auto-Encoder) encoder. In this example, the input size is 512*512*3, but it is not limited to this. It can also be other sizes (for example, 256*256*3, determined by the set downsampling multiple). *3 represents the channel dimension. For the convenience of description, this dimension may be omitted in the following text. With the help of the VAE encoder, the 512*512*3 image is compressed to a latent space of 64*64*4, and the denoising operation of the first AI network is performed in this low-dimensional space (i.e., the first denoising process). It can be understood that VAE is optional. VAE includes two parts, encoding and decoding, which are used to encode the image into a form that is easy to represent, such as 64*64*4. This form is configurable and can be decoded back to the original real image as losslessly as possible. After adding noise to the image output by the VAE encoder, the first image can be obtained. The first image and the content feature Figure 1 The first AI network is inputted to perform the first denoising process, and the first AI network outputs the denoising result. The denoising result can be inputted into the first AI network again for denoising process, until the target denoising result is outputted after multiple denoising processes through the first AI network. The target denoising result is then converted to a 512*512*3 space (the same as the VAE encoder input size) by the VAE decoder to obtain the first denoising process result.

[0087] In the embodiment of the present application, an optional implementation is provided for step S1021, which may specifically include:

[0088] Step SA1: providing the user with category information of the repair content;

[0089] Step SA2: receiving a second instruction from a user to select target category information;

[0090] Step SA3: Based on the second instruction, determine the target repair content-related information corresponding to the target repair area.

[0091] That is, in the embodiment of the present application, the second instruction is a selection instruction, that is, selectable options are provided to the user, corresponding to the category information of the repair content respectively. When the user clicks on the option of the target category information, the second instruction of the user selecting the target category information can be received.

[0092] In other embodiments, the second instruction may also be generated according to a user-defined input operation.

[0093] Among them, the content type carried by the second instruction can be text, picture, voice, etc. The embodiment of the present application does not limit the content type of the second instruction.

[0094] For ease of understanding, the embodiment of the present application takes the second instruction as a selection instruction, and the content type of the second instruction is text as an example for introduction. The processing methods of other content types can be deduced by analogy, and will not be repeated. Specifically, the user can be provided with selectable input texts in the interface, which correspond to the category information of the repair content respectively, and receive the second instruction of the user selecting the target category information. In response to the second instruction, the input text selected by the user is obtained, and the target repair content related information corresponding to the target repair area is determined according to the input text.

[0095] For the embodiment of the present application, if the target repair content-related information adopts a content feature map, the target repair content-related information may also be referred to as a text feature map.

[0096] Optionally, the association relationship between each selectable input text and its corresponding text feature graph may be stored in a database. According to the input text selected by the user, the corresponding text feature graph is found in the database.

[0097] As an example, a process of performing denoising based on a text feature map selected by a user is as follows: Figure 4b As shown, it may specifically include: according to the input text "XXXX" selected by the user, searching for the text feature map corresponding to the input text "XXXX" in the defined text feature map database, and inputting the text feature map together with the first image into the first AI network for the first denoising process. Since the classification of the repaired content can be fixed and limited, the text feature map can be saved in the database. Among them, Figure 4b The method of obtaining the first image and the subsequent processing method of the first AI network can be referred to in Figure 4aFor example, if the user selects "Flowers", the repair content of the target repair area will include "flowers".

[0098] Optionally, step S1021 may further include:

[0099] Step SB: If the target category information does not exist, the feature map of the empty character is used as the target repair content related information.

[0100] In an embodiment of the present application, target category information does not exist, which can be understood as the user has not selected or input target category information, that is, the user does not want the repair content, then the repair content of the target repair area can be random, so the feature map of the empty character can be used as information related to the target repair content.

[0101] As an example, another process example of denoising based on target repair content related information is as follows: Figure 4c As shown, image restoration includes a text feature map database, a VAE network, a first AI network, and a noise scoring network. The specific process may include: the user can select a pre-defined input text or not select it directly. If the user selects the input text "XXXX", the text feature map corresponding to the input text "XXXX" is searched in the text feature map database as the target restoration content related information. If the user does not select any input text, the feature map of the empty character is used as the target restoration content related information. The second image (or the scaled second image) is input into a VAE encoder. In this example, the input size is 256*256*3. With the help of the VAE encoder, the second image is compressed to a latent space of 64*64*4, and the denoising operation of the first AI network is performed in this low-dimensional space (ie, the first denoising process). Specifically, after adding noise to the image output by the VAE encoder, the first image can be obtained. The text feature map and the first image are input into the first AI network together for the first denoising process, and the first AI network outputs the processing result. The noise scoring network can be used to determine whether the processing result meets the predetermined conditions. If not, the first AI network is input again to continue the first denoising process. If so, the denoising result is converted to a 512*512*3 space using the VAE decoder to obtain the first denoising result. This process can be understood as a dynamic denoising process exit mechanism to solve the problem of large computational complexity and high latency.

[0102] In other embodiments, the target repair content related information may also be generated in real time. For example, a target repair content related information generation network generates the target repair content related information in real time according to the user's second instruction. If the user does not perform the operation of selecting target category information, the generated target repair content related information may be a feature map of an empty character. The subsequent processing method of the target repair content related information generated in real time is the same as that of Figure 4a and Figure 4c The subsequent processing flow is similar and will not be repeated here.

[0103] In an embodiment of the present application, a fast and lightweight first AI network is provided, so that the model is small and the amount of calculation is small, which solves the problem that the denoising model is usually large and the amount of calculation is large. Optionally, the first AI network includes an attention module, wherein step S1022 may specifically include: using at least one attention module, through Fourier transform, calculating the attention between the noise feature map corresponding to the first image and the target restoration content related information, the attention representation is the degree of denoising of the noise feature map according to the target restoration content related information, and performing at least one first denoising process on the first image based on the attention calculation result.

[0104] The attention module may adopt a Fast Fourier Attention (FFA) module, but is not limited thereto.

[0105] In an embodiment of the present application, information related to the target repair content can be input into the first AI network by FFA, so as to guide the first AI network to generate specified content in the target repair area. Specifically, after merging the information related to the target repair content into the noise feature map corresponding to the first image, the attention module can calculate the attention between the two feature maps, and the first AI network can perform specific denoising based on the input operation (such as input text) of the repair content of the target repair area. Specifically, the attention module uses Fourier transform to convert the two input feature maps into the frequency domain space, performs attention processing on the converted feature maps, and outputs the processed feature maps.

[0106] In the implementation of this application, the attention between the noise feature map corresponding to the first image and the target restoration content related information is calculated through Fourier transform, specifically including:

[0107] Step S501: performing a fast Fourier transform on the noise feature map corresponding to the first image and the target restoration content related information to obtain a Fourier feature map;

[0108] That is, through FFT (Fast Fourier Transform), the information related to the target repair content is merged into the noise feature map corresponding to the first image, and the obtained feature map is converted from the time domain space to the Fourier space (frequency domain space). Among them, the noise feature map corresponding to the first image can refer to a feature map extracted from the first image, or it can refer to a feature map extracted from the denoising result obtained by performing a first denoising process on the first image any number of times using the first artificial intelligence AI network.

[0109] Step S502: performing a first convolution operation on the Fourier feature map to obtain a Fourier space query (Query) feature, a Fourier space key (Key) feature, and a Fourier space value (Value) feature;

[0110] In the embodiment of the present application, the first convolution operation may refer to using a 1*1 convolution kernel (Conv 1*1) to perform a convolution operation. This is because the feature map in the time domain is converted to the frequency domain space using Fourier transform, so that each value of the feature map can represent the global features of the image. Based on this feature, using a 1*1 convolution kernel for convolution operation can achieve the global feature map information that can only be obtained using a fully connected layer in the time domain space.

[0111] Step S503: performing a second convolution operation on the Fourier space query feature and the Fourier space key feature to obtain a first attention weight coefficient;

[0112] Specifically, the Fourier space query feature and the Fourier space key feature may be directly concatenated (Concat) together, and a second convolution operation may be performed on the concatenated result.

[0113] In the embodiment of the present application, the second convolution operation may also refer to the use of a 1*1 convolution kernel (Conv1*1) for convolution operations. In the time domain, a huge matrix multiplication operation is required to fuse the Query feature graph, the Key feature graph, and the Value feature graph. In the embodiment of the present application, after the feature graph of the time domain space is mapped to the frequency domain space, the information of different channels can be mixed in the frequency domain space through the convolution operation of the 1*1 convolution kernel, so as to achieve the effect of approximating the matrix multiplication of the Query feature graph and the Key feature graph, thereby reducing the amount of calculation.

[0114] Step S504: performing an inverse fast Fourier transform on the first attention weight coefficient to obtain a second attention weight coefficient;

[0115] That is, through IFFT (Inverse Fast Fourier Transform), the first attention weight coefficient is converted from the frequency domain space back to the time domain space to obtain the second attention weight coefficient.

[0116] Step S505: Obtaining the attention calculation result based on the second attention weight coefficient and the Fourier space value feature.

[0117] The values ​​in the Fourier space value feature are weighted according to the second attention weight coefficient to obtain the attention calculation result as the calculation result output by FFA.

[0118] For the embodiment of the present application, since FFT and IFFT are parameter-free, their time complexity is negligible compared to other computational costs, so the increase in computational complexity caused by FFT and IFFT can be ignored. Therefore, the parameters of the model can be greatly reduced, thereby reducing the size of the model.

[0119] As an example, a processing flow of an attention module is as follows: Figure 5 As shown, it is assumed that the noise feature map corresponding to the first image and the target restoration content related information are used to obtain a CNN (Convolutional Neural Network) feature map of 1,320,64,64 (where 1 represents the number of images, 320 is the channel dimension, and the channel dimension can be represented in the front or in the back as needed), and the feature map is passed through FFT to obtain a Fourier feature map of 1,320,64,64. The Query, Key and Value feature maps of the Fourier space of the global receptive field of 1,320,64,64 are calculated using a 1*1 convolution operation, where the Value feature map can also be represented as 1,4096,320. The Query and Key feature maps are merged in the channel dimension using the Concat operation, and then the merged 1,640,64,64 feature map is output to the 1,4096,64,64 dimension using a 1*1 convolution to obtain the first attention weight coefficient. Then the first attention weight coefficient is processed by IFFT to obtain the result of 1,4096,64,64 in the time domain space, and the weight coefficients of Query and Key can be processed by deformation, normalization, Softmax and other operations to obtain the second attention weight coefficient of 1,4096,4096. The value in the Value feature map (1,4096,320) is weighted according to the second attention weight coefficient, and finally the calculation result of FFA of 1,4096,320 is output.

[0120] Among them, Figure 6 As shown in the figure, since the feature map is converted from the time domain space (1,320,64,64) to the Fourier space (1,320,64,64), each pixel in the Fourier space is the receptive field of the entire image area, so the 1*1 convolution (the first convolution operation) can be used to calculate the feature maps (1,320,64,64) of the Query, Key and Value of the global receptive field, which can greatly reduce the amount of calculation compared to the full connection operation in the time domain space.

[0121] And, if Figure 7As shown, the matrix multiplication result of the Query and Key feature maps in the time domain space can be used as the weight coefficient of the Value feature map to determine which value of the Value feature map is important. Since the feature maps of FFA all contain global information, the frequency domain space can use the channel fusion method to calculate the feature map. Specifically, the Query and Key feature maps in the Fourier space are merged (fused) in the channel dimension using the concat operation, and then the fused (1,640,64,64) feature map is output as an attention weight coefficient of (1,4096,64,64) using a 1*1 convolution (the second convolution operation). Compared with the use of a huge matrix multiplication to fuse the Query and Key feature maps in the time domain space, the present application scheme can greatly reduce the amount of calculation.

[0122] The fast and lightweight first AI network provided by the embodiment of the present application uses a fast Fourier attention (FFA) module to convert the pixel space into Fourier space, and the value of each position of the feature map corresponds to the characteristics of a certain image range in the pixel space. 1*1conv can be used to implement the attention mechanism of the conversion, thereby significantly reducing the size of the model. And the global receptive field of the image in Fourier space can be extracted to reduce memory consumption. Based on the CNN attention calculation results in Fourier space, similar attention calculation results can be achieved for time-domain space matrix multiplication, reducing model size and latency.

[0123] In an embodiment of the present application, the fast and lightweight first AI network may include not only an attention module, but also a multi-layer residual network (Deep Residual Network, ResNet) module.

[0124] Among them, the residual network module can avoid the gradient vanishing or gradient exploding problems faced in the first AI network training and accelerate model convergence.

[0125] Furthermore, the encoding part and the decoding part of the first AI network may include jump connections. Jump connections can effectively utilize the intermediate features of the network layer, solve the gradient vanishing problem that occurs when the number of network layers is deep, and at the same time facilitate the back propagation of the gradient, so that the model converges faster.

[0126] Furthermore, the first AI network may include an intermediate part between the encoding part and the decoding part, and the intermediate part may more accurately find the noise points in the first image.

[0127] As an example, an architecture example of a first AI network is as follows: Figure 8As shown, it may include: encoding the second image into a 64*64*4 image using a VAE encoder. Randomly generate a noise data image of 64*64*4 dimensions. The two groups of images are spliced ​​together in the channel dimension and input into the first AI network, wherein the first AI network may be a U-net network (a network containing jump connections) composed of a multi-layer residual network module and an FFA. Specifically, a group of modules consisting of a residual network and an FFA, the encoding part and the decoding part of the first AI network both contain the same group (at least one group) of modules, and the output results of each group of modules in the encoding part are connected to a group of modules in the decoding part corresponding to the same scale feature, that is, the output results of the Xth group of the encoding part can be input into the N-X+1th group of the decoding part through a jump connection, 1≤X≤N. Figure 8 In the example, the encoding part and the decoding part both contain three groups (N=3) of modules. The output result of the first group of modules (X=1) of the encoding part is input into the third group of modules (N-X+1=3) of the decoding part through a skip connection, the output result of the second group of modules (X=2) of the encoding part is input into the second group of modules (N-X+1=2) of the decoding part through a skip connection, and the output result of the third group of modules (X=3) of the encoding part is input into the first group of modules (N-X+1=1) of the decoding part through a skip connection. The case of more or fewer groups of modules can be deduced by analogy and will not be repeated here. The first AI network may also include an intermediate part (optional) between the encoding part and the decoding part, Figure 8 The middle part of the algorithm includes a residual network, an FFA and a residual network connected together. The middle part processes the output of the encoding part and inputs it into the decoding part. Before the encoding part and after the decoding part, a convolution (CNN) module is also included. The CNN module and the ResNet module can extract the latent space noise feature map. Figure 8 In the CNN, information related to the target restoration content is input into the network by each FFA (merged into the noise feature map), and the attention of the two feature maps is calculated to guide the network to perform specific denoising processing to generate specified content in the target restoration area.

[0128] The fast and lightweight first AI network provided in the embodiment of the present application can reduce the amount of calculation and calculation parameters, improve the efficiency of the model while maintaining the module performance, and the model is very easy to train.

[0129] In the embodiment of the present application, in step S102, in the process of using the first AI network to perform at least one first denoising process on the first image, at least one first AI network can be used to perform the first denoising process on the first image in sequence.

[0130] Among them, for each processing result output by the first AI network, the first feature map of the processing result and the second feature map of the third image with a predetermined proportion of noise added can be extracted, the third image is the image of the second image before the target repair area is removed, and based on the similarity between the first feature map and the second feature map, it is determined whether to continue using the first AI network to perform the first denoising process, that is, to determine whether each processing result (denoising result) of the first AI network meets the predetermined conditions, thereby realizing a dynamic denoising process exit mechanism.

[0131] The third image refers to the original image without losing or removing the target repair area. A predetermined proportion of random noise is fused with the third image, such as 30%, to obtain a third image with a predetermined proportion of noise added (for ease of description, it may be referred to as the fourth image below). The value of the predetermined proportion can be set according to actual conditions, and the embodiment of the present application is not limited here.

[0132] In an embodiment of the present application, a first feature map of the processing result is extracted, and a second feature map of the fourth image is extracted, so as to determine whether the noise content of the denoising result is lower than the target based on the similarity between the first feature map and the second feature map.

[0133] In an embodiment of the present application, the step of extracting the first feature map of the processing result may specifically include: using a third AI network to extract at least two levels of first feature maps of the processing result, and calculating the autocorrelation coefficient of each first feature map.

[0134] In the embodiment of the present application, the third AI network can be understood as a noise scoring network, which is used to extract high-level feature maps and calculate the autocorrelation coefficient representing the semantic content and contextual relationship of the feature maps. Among them, the noise scoring network can be a CNN-based network, for example, it can include a connected multi-layer CNN network, each layer of CNN is used to extract the first feature map of different levels.

[0135] Optionally, the processing result of the first AI network and the original image (second image) containing the target repair area are input into the noise scoring network to extract first feature maps of different levels. Optionally, the partial image corresponding to the target repair area in the denoising result and the original image (second image) containing the target repair area are spliced ​​together to form a first composite image in which the target repair area contains the denoising result content and other areas contain the original image content. The first composite image is input into the noise scoring network to extract first feature maps of different levels. Alternatively, the target repair area of ​​the processing result of the first AI network can be directly input into the noise scoring network to extract first feature maps of different levels.

[0136] Optionally, the autocorrelation coefficient of each first feature map is calculated using a Gram matrix. That is, a symmetric matrix composed of the inner products of k first feature maps (corresponding to the extracted level) in a multidimensional Euclidean space is used as the autocorrelation coefficient of the k first feature maps, but it is not limited to this, and other autocorrelation coefficient calculation methods can also be used. The Gram matrix can represent the semantic content and contextual relationship of the image. Therefore, when the autocorrelation coefficient calculation result is similar to the target, it means that the denoising result has similar semantic content and channel relationship as the target.

[0137] Furthermore, the step of extracting the second feature map of the third image with noise added in a predetermined proportion may specifically include: using a third AI network to extract at least two levels of second feature maps of the third image with noise added in a predetermined proportion, and calculating the autocorrelation coefficient of each second feature map.

[0138] Specifically, the fourth image and the original image (second image) containing the target repair area are input into the noise scoring network to extract second feature maps of different levels. Optionally, the partial image corresponding to the target repair area in the fourth image and the original image (second image) containing the target repair area are spliced ​​together to form a second composite image in which the target repair area contains a predetermined proportion of noise content and other areas contain the original image content. The second composite image is input into the noise scoring network to extract second feature maps of different levels. Alternatively, if each first feature map is obtained by inputting the target repair area of ​​the processing result of the first AI network into the noise scoring network, the target repair area of ​​the fourth image can also be directly input into the noise scoring network to extract second feature maps of different levels.

[0139] Optionally, the third AI network used to extract each first feature map and the third AI network used to extract each second feature map may be the same, and the autocorrelation coefficient calculation method adopted may be the same, which facilitates comparison.

[0140] It is worth noting that the step of extracting the second feature map of the third image with a predetermined proportion of noise added can be performed only once. For example, the autocorrelation coefficient of the second feature map can be calculated only once, and the result can be directly obtained and used to determine whether to output the first denoising processing result when compared with the autocorrelation coefficient of the processing result of the first AI network next time, without the need to calculate it again.

[0141] Furthermore, based on the similarity between the first feature map and the second feature map, the step of determining whether to continue using the first AI network to perform the first denoising process may specifically include: determining the distance between the autocorrelation coefficients of each first feature map and the autocorrelation coefficients of each second feature map; based on the relationship between the distance and the first threshold, determining whether to continue using the first AI network to perform the first denoising process.

[0142] Calculate the distance between the autocorrelation coefficient of each first feature graph and the autocorrelation coefficient of each second feature graph, such as the L1 distance, but not limited to this. Optionally, if the distance < the first threshold, it is judged that the processing result meets the predetermined condition, and the first denoising result can be output, and the first AI network is no longer used for denoising. Otherwise, continue to use the first AI network for denoising. In actual applications, those skilled in the art can set the value of the first threshold (such as 0.1) according to actual conditions, and the embodiments of the present application are not limited here.

[0143] Based on this, the number of times the first AI network performs the first denoising process can be reduced, thereby reducing the operation delay.

[0144] Optionally, the determination of whether to continue using the first AI network for the first denoising process may be based on a combination of multiple judgment conditions. As an example, if the number of times the first AI network is used for the first denoising process does not exceed a predetermined number of times, the distance of the above-mentioned autocorrelation coefficient is used to determine whether the denoising result meets the exit condition. If the number of times the first AI network is used for the first denoising process reaches a predetermined number of times, the denoising process of the first AI network is directly exited. In practical applications, those skilled in the art may set the combination of multiple judgment conditions according to actual conditions, and the embodiments of the present application are not limited thereto.

[0145] Based on this, it is possible to dynamically determine whether the intermediate denoising result output by the first AI network meets the predetermined exit condition, and output the first denoising processing result if it is met, thereby reducing the time step and delay.

[0146] As an example, a process example of a dynamic denoising process exit mechanism is as follows: Fig. 9As shown, it can include: ①: Take the partial image corresponding to the target repair area of ​​the processing result (denoising result) output by the first AI network, splice it into the original image (second image) of the missing area (target repair area) in the 256*256*3 hidden code space, and input the spliced ​​image into the noise scoring network (third AI network) based on CNN, extract the first feature maps of different levels, and use the Gram matrix to calculate the autocorrelation coefficient between each first feature map (feature map of the denoising result). ②: Fuse the random noise with the original image (third image) of the 256*256*3 hidden code space without missing areas (the target repair area is not missing) in a predetermined ratio, such as fusion 30%. Then splice the partial image corresponding to the target repair area of ​​the fused image (fourth image) and the original image (second image) of the 256*256*3 hidden code space containing missing areas (target repair area) to obtain a spliced ​​image of the target noise content, and extract the autocorrelation coefficient between each second feature map (feature map of the spliced ​​image) through the noise scoring network. ③: Compare the distance between the autocorrelation coefficients of each first feature map obtained in ① and the autocorrelation coefficients of each second feature map (stitched image) obtained in ②, such as the L1 distance. If the distance is less than the first threshold (such as 0.1), the result is output to the VAE decoder to obtain the intermediate first denoising result of 256*256*3. Otherwise, continue to input the first AI network for denoising.

[0147] The dynamic denoising exit mechanism provided in the embodiment of the present application uses a noise scoring network to estimate the denoising result of the first AI network. When the denoising result meets the requirements, the first denoising processing result is output, and then the second AI network is used to repair the intermediate first denoising processing result, thereby significantly improving the operating efficiency while maintaining the quality of the repaired image.

[0148] In the embodiment of the present application, an optional implementation is provided for step S103, which may specifically include:

[0149] Step S1031: performing a second denoising process on the first denoising process result to obtain a second denoising process result;

[0150] In an embodiment of the present application, the second denoising process can expand the content of the unknown area (i.e., the remaining noise points in the first denoising process result) based on the known area in the first denoising process result to remove these remaining noise points and obtain the second denoising process result (fuzzy denoising result).

[0151] Step S1032: determining texture information of the second image from the region that does not need to be repaired;

[0152] In the embodiment of the present application, the texture information of the second image can be generated based on the known area in the second image (the area that does not need to be repaired) by borrowing the texture of the known area for repairing and finding a suitable texture.

[0153] Step S1033: extracting a third feature map of the target restoration area in the second denoising result;

[0154] Optionally, performing a fourth convolution operation on the target repair area in the second denoising result to extract a third feature map;

[0155] Optionally, the fourth convolution operation may refer to partial convolution, that is, using partial convolution to extract a feature map of the target repair area in the second denoising processing result.

[0156] Step S1034: Based on the texture information and the third feature map, the target repair area is repaired to obtain a repaired image.

[0157] In the implementation of this application, the texture information includes a more suitable texture of the target repair area. Combining it with the third feature map can generate an image with better texture for the target repair area of ​​the second denoising processing result, thereby obtaining an image after the target repair area is repaired.

[0158] As an example, a processing flow of the second AI network is as follows: Fig.10 As shown, it can include: inputting the 1024*1024 original image (second image) containing the missing area (target repair area) and the 512*512 (or 256*256) first denoising result into the second AI network (an improved GAN can be used), and the second AI network performs a second denoising on the first denoising result to remove the remaining noise points in the first denoising result to obtain a 1024*1024 second denoising result, and determines the texture information of the second image based on the known area of ​​the second image, and then generates a texture image of the target repair area of ​​the second denoising result based on the texture information and the third feature map, and fuses the second denoising result and the texture image (Add), and finally outputs a 1024*1024 repaired image.

[0159] In the embodiment of the present application, an optional implementation is provided for step S1032, which may specifically include the following steps:

[0160] Step S1101: using a third AI network to extract a fourth feature map of at least one previous level from the first denoising result;

[0161] The third AI network used in this step may be the same or similar to the third AI network used to extract the first feature map. Using the third AI network to extract the shallow feature map of the first denoising result, directly using the third AI network to extract the first feature map can save certain training resources. Taking the third AI network including a connected multi-layer CNN network as an example, the shallow feature map refers to the feature map extracted by the first few layers of CNN networks of the third AI network, which may also be referred to as a low-level feature map or a low-level feature map. Compared with the deep feature map, the shallow feature map contains more texture information. In practical applications, those skilled in the art can set the number of levels corresponding to the required low-level feature map according to actual conditions, and the embodiments of the present application are not limited here.

[0162] Step S1102: performing a third convolution operation on the second image and each fourth feature map to determine non-texture information of the second image from the region that does not need to be repaired.

[0163] In the embodiment of the present application, texture information can be understood as a high-frequency feature map, and non-texture information can be understood as a low-frequency feature map. Among them, a general image includes high-frequency components and low-frequency components. The high-frequency component corresponds to the part with a large frequency change, usually corresponding to information such as image contour and texture, and the frequency change of the low-frequency image is relatively uniform or very slow, usually corresponding to information such as image color.

[0164] The third convolution operation may refer to a fuzzy convolution, which may specifically include an upsampling stage and a downsampling stage. Because the first denoising result (the fourth feature map) includes less high-frequency information, and the second image contains more high-frequency information, when scaling the two images, the original high-frequency information can be released, thereby obtaining a low-frequency feature map.

[0165] Step S1103: Determine texture information of the second image based on the second image and the non-texture information of the second image.

[0166] In an embodiment of the present application, the extracted shallow feature map can be combined with the second image, and based on the known areas in the second image and the shallow feature map, the texture information of the target repair area of ​​the second denoising processing result can be generated by borrowing the texture restoration of the known areas.

[0167] In an embodiment of the present application, the second AI network may include a denoising branch subnetwork and at least one texture enhancement branch subnetwork, wherein the denoising branch subnetwork may be used to perform the above steps S1032 to S1034 (step S1101 may not be included).

[0168] The second AI network provided in the embodiment of the present application uses a denoising branch to eliminate noise points, and then combines it with the shallow feature map of the first denoising processing result and the known area of ​​the original image (second image) containing the target repair area to form a rich texture. Finally, the repaired image is output by element point addition, that is, based on the image generation technology, the intermediate first denoising processing result is directly converted into a high-resolution generation result to ensure the image quality of the repaired image.

[0169] As an example, another processing flow using the second AI network is as follows: Fig.11 As shown, it may include: on the one hand, using the VAE decoder to upsample the first denoising result of 256*256*3 (or 512*512*3) to 1024*1024*3, and then input it into the denoising branch sub-network of the second AI network (which may use a convolution-based U-net, but is not limited to this), remove the remaining noise points in the 1024*1024 image, and the denoising branch sub-network outputs the second denoising result of 1024*1024*3. On the other hand, the first denoising result of 256*256*3 is input into the third AI network (noise score network) to extract the shallow feature map (fourth feature map) of the first denoising result, Fig.11 The first two layers are taken as examples, which are not limited to this in practical applications. The extracted shallow feature map, the original image (second image) containing the missing area (target repair area) of 1024*1024*3, and the second denoising result of 1024*1024*3 are input into the connected texture enhancement (RT, Render Texture) branch subnetwork. Among them, the extracted shallow feature map can be input into each texture enhancement branch subnetwork (RT module), and the last texture enhancement branch subnetwork outputs an enhanced texture image of 1024*1024*3. The texture image and the second denoising result are fused, such as adding dots to the original elements, to obtain a 1024*1024*3 repaired image.

[0170] In an embodiment of the present application, an optional implementation scheme is provided for the denoising branch subnetwork. Specifically, the denoising branch subnetwork can adopt a U-net structure, including an encoder and a decoder, mainly including a CNN, and may also contain a transformer attention model. The input of this branch is an upsampled image of the 1024*1024*3 first denoising result (which may contain noise points) output by the middle of the first AI network. Among them, the upsampling method can be an interpolation method, but is not limited to this. The output is a second denoising result of 1024*1024*3 (without noise points).

[0171] In an embodiment of the present application, the noise contained in the first denoising result outputted by the first AI network can be removed by image generation technology, without the first AI network having to continue running multiple times to remove the noise points. This can significantly improve the operating efficiency while maintaining the image quality.

[0172] In the embodiment of the present application, an optional implementation is provided for step S1034, which may specifically include the following steps:

[0173] Step S1201: Divide the feature map corresponding to the texture information and the third feature map into image blocks of the same size to obtain each first sub-feature map and each second sub-feature map;

[0174] That is, the high-frequency feature map and the third feature map extracted by partial convolution are divided into blocks of the same height and width. The specific segmentation size (height and width) can be set according to actual conditions, and the embodiment of the present application is not limited here.

[0175] It should be noted that when the shape of the target repair area is irregular, the total area of ​​the segmented second sub-feature map should be able to cover the irregular target repair area.

[0176] Step S1202: Calculate the similarity between each first sub-feature graph and each second sub-feature graph, and obtain an adaptive weight corresponding to each pair of the first sub-feature graph and the second sub-feature graph;

[0177] Optionally, a cosine similarity algorithm is used to calculate the similarity between each first sub-feature graph and each second sub-feature graph, but is not limited thereto.

[0178] Further optionally, a normalization algorithm such as SoftMax is used to normalize the similarity settlement results to obtain adaptive weights (a total of M×N) corresponding to each first sub-feature graph (assuming there are M) and each second sub-feature graph (assuming there are N).

[0179] Step S1203: fusing each first sub-feature map corresponding to each second sub-feature map with the corresponding adaptive weights to obtain each second sub-feature map after texture enhancement;

[0180] That is, for a second sub-feature map, its corresponding M first sub-feature maps are fused with corresponding adaptive weights to obtain a texture-enhanced second sub-feature map, and finally N texture-enhanced second sub-feature maps are obtained.

[0181] Among them, for each second sub-feature graph, the fusion method of the corresponding first sub-feature graphs and the corresponding adaptive weights includes but is not limited to weighted addition.

[0182] Step S1204: Based on each second sub-feature map after texture enhancement, the target repair area is repaired to obtain a repaired image.

[0183] Each second sub-feature map can be used as a high-frequency feature map of the target repair area after texture enhancement output by a texture enhancement branch sub-network (RT module).

[0184] Specifically, each second sub-feature map and the target repair area of ​​the second denoising process result can be element-dotted to obtain a repaired image. That is, the high-frequency feature map after texture enhancement of the target repair area can be fused with the second denoising process result, such as element dot addition, and then input into the next texture enhancement branch sub-network (RT module) with the second image for further texture enhancement processing.

[0185] Alternatively, for the last texture enhancement branch sub-network (RT module), the high-frequency feature map of the target repair area after texture enhancement can be used as a texture image and fused with the second denoising result, such as adding elements, to obtain the repaired image of the target repair area.

[0186] As an example, a processing flow using a texture enhancement branch sub-network (RT module) is as follows: Fig.12As shown, it can include: on the one hand, the original image (second image) containing the missing area (target repair area) of 1024*1024 is combined with the shallow feature map output by the noise scoring network, and a low-frequency feature map (non-texture information) is obtained by fuzzy convolution, and the original image (second image) containing the missing area (target repair area) and the low-frequency feature map are element-wise subtracted (Sub) to obtain a high-frequency feature map (texture information). On the other hand, a third feature map of the second denoising result is extracted using partial convolution. Further, the high-frequency feature map and the third feature map extracted by partial convolution are divided into blocks of the same height and width, such as abdc (each second sub-feature map) and 1234567 (each first sub-feature map). The cosine similarity algorithm is used to calculate the similarity between each abcd second sub-feature map and each 1234567 first sub-feature map to obtain an adaptive weight value, and the adaptive weight value is normalized (SoftMax) to obtain a normalized adaptive weight. Multiply the 1234567 position image blocks of group a with the adaptive weights by each 1234567 first sub-feature map, and then add them all together to get the enhanced a. Perform similar operations on other groups bcd. Place the enhanced abcd in the target repair area (other areas can be empty) to get a texture-enhanced high-frequency feature map (enhanced texture image). Output the second denoising result, and perform element-wise addition operation on the second denoising result and the texture-enhanced high-frequency feature map to get a result map with rich texture in the target repair area. Then add the 1024*1024 original image (second image) containing the target repair area to the result map with rich texture in the target repair area. Figure 1 Starting from, the next texture enhancement branch sub-network is input, wherein the result map with rich texture of the target repair area can be used as the second denoising result input by the next texture enhancement branch sub-network.

[0187] It should be noted that Fig.12 The texture enhancement branch subnetwork (RT module) shown corresponds to Fig.11 For any texture enhancement branch subnetwork except the last one shown in Fig.11 The last texture enhancement branch subnetwork shown in the figure can be directly used as the enhanced texture image after obtaining the high-frequency feature map of texture enhancement. The second denoising result is added to the enhanced texture image by element-wise addition to obtain the image after the target repair area is repaired.

[0188] The texture enhancement branch subnetwork provided in the embodiment of the present application uses fuzzy convolution to simulate the generation mode of the denoising branch to ensure that the appropriate high-frequency information is found. Then learn and calculate the adaptive weights between the high-frequency feature map and the third feature map of the second denoising process result. According to the adaptive weights, the texture information in the high-frequency feature map is fused to finally obtain a texture-enhanced high-frequency feature map, that is, the high-frequency texture of the target repair area is repaired by borrowing the high-frequency texture of the known area to obtain better texture performance of the repaired image.

[0189] In an embodiment of the present application, in order to avoid the pattern of the blurred image in the process of searching for high-frequency information being inconsistent with the blurred pattern of the 1024*1024 second denoising result obtained by the denoising branch, resulting in a mismatch between the high-frequency image obtained later and affecting texture enhancement. Therefore, an embodiment of the present application uses a third convolution operation (such as fuzzy convolution) to extract a low-frequency feature map. The fuzzy convolution includes a small U-net in the downsampling stage and the upsampling stage. Specifically, step S1102 may include: performing a corresponding number of downsampling operations and upsampling operations on the second image to obtain non-texture information of the second image; wherein, for at least one downsampling operation, the downsampling result and the fourth feature map of the corresponding scale are element-wise averaged to obtain an average feature map, and the next downsampling operation or upsampling operation is performed on the average feature map.

[0190] Among them, the number of downsampling operations and upsampling operations can be set according to actual conditions, and the embodiments of the present application are not limited here. Optionally, the downsampling stage uses a convolution layer step size of 2 or other pooling operations for downsampling, and the upsampling uses a convolution plus interpolation upsampling method or a deconvolution step size of 2 for upsampling, but is not limited to this.

[0191] As an example, taking the number of downsampling operations and upsampling operations as 4 as an example, a processing flow of a third convolution operation (such as fuzzy convolution) is as follows: Fig.13 As shown, it may include: inputting the 1024*1024 original image (second image) containing the missing area (target repair area) into the downsampling stage, and performing 4 downsampling operations through 4 CNN layers in the downsampling stage. Among them, after the downsampling operation of the 1st and 3rd layers (which can be set, not limited to this) extracts the feature map, the feature average of each element point is calculated for the extracted feature map and the shallow feature map output by the third AI network (noise scoring network) of the corresponding scale, and the average feature map is obtained, which is input into the next CNN layer to perform the next downsampling operation. The result of the downsampling stage is input into the upsampling stage, and finally a low-frequency feature map (non-texture information) is obtained.

[0192] In the embodiment of the present application, after the fuzzy convolution performs a mean operation on the feature map of the second image and the shallow layer feature map of the noise scoring network, the fuzzy convolution can obtain a similar blur pattern to the image generated by the denoising branch of the second AI network using the first denoising result, thereby obtaining a more reasonable high-frequency feature map.

[0193] In the embodiment of the present application, an optional implementation is provided for the execution of step S101. Specifically, before acquiring the first image, the following preprocessing steps may also be included:

[0194] Step S1501: Acquire a fifth image including a target repair area;

[0195] Among them, the fifth image can be obtained by removing the object area selected by the user in the third image (original image), or it can be an original lost image in which some information is missing or damaged due to various factors such as occlusion, blur, transmission interference, etc., but is not limited to this.

[0196] Step S1502: If the size of the fifth image is greater than the second threshold, obtain the second image from the fifth image by at least one of the following cropping methods:

[0197] (1) If the area of ​​the target repair region is smaller than the third threshold and the length of the target repair region is smaller than the fourth threshold, the fifth image is cropped into an image with a size of the second threshold with the target repair region as the center, thereby obtaining a second image;

[0198] The area or size of the target repair region can be determined by its corresponding mask image.

[0199] Specifically, when the area of ​​the target repair area is smaller than the third threshold (for example, 1024*1024*80%), and the length of the target repair area is smaller than the fourth threshold (for example, 1024), that is, the target repair area is relatively small, the original image is cropped with the center point of the target repair area as the center into an image with a size of the second threshold (for example, 1024*1024), and input into the image post-repair processing.

[0200] Optionally, if the target repair area is close to the edge of the fifth image, cropping with the center point of the target repair area as the center may include areas outside the image. The cropped area can be translated into the fifth image, or the fifth image can be mirrored before cropping.

[0201] (2) If the area of ​​the target repair region is smaller than the third threshold, the length of the target repair region is greater than the fourth threshold, or if the area of ​​the target repair region is greater than the third threshold, determine that the size of the image region in the fifth image is the second threshold, and the area of ​​the target repair region in the image region is not greater than the fifth threshold, crop the image region to obtain a second image, and use the target repair region in the image region as the target repair region of the second image.

[0202] Specifically, if the area of ​​the target repair area is smaller than the third threshold (for example, 1024*1024*80%), the length of the target repair area is greater than the fourth threshold (for example, 1024), that is, the target repair area is relatively slender, or, if the area of ​​the target repair area is greater than the third threshold, that is, the target repair area is relatively large, determine that the size of the image area in the fifth image is the second threshold (for example, 1024*1024), and the area of ​​the target repair area in the image area is not greater than the fifth threshold (for example, 1024*1024*80%).

[0203] Optionally, from left to right and from top to bottom (other orders are also possible), find a matrix region with a size of the second threshold (such as 1024*1024), where the area of ​​the target repair region in the region is equal to or less than the fifth threshold (such as 1024*1024*80%), crop the region, and input it into the image post-repair processing.

[0204] Furthermore, for the latter case, the steps may also be included: fusing the image after the target repair area corresponding to the image area is repaired into the fifth image to obtain the fifth image after the target repair area is updated; and obtaining the second image from the fifth image after the target repair area is updated through at least one of the above-mentioned cropping methods.

[0205] That is, if the target repair area is relatively slender or relatively large, one cropping may not be able to complete the filling of all target repair areas. The image after the repair of the previous cropped area can be pasted back to the fifth image of the original image, and the target repair area and its mask image can be updated. The same process can be followed to continue adaptive cropping and input into the image post-repair processing.

[0206] (3) If the size of the fifth image is smaller than the second threshold, it can be directly input into the image post-restoration process according to the model situation, or the fifth image can be mirrored and then the second image can be obtained through the cropping method (1) and input into the image post-restoration process.

[0207] For the cropped image, the image after the cropped area is repaired can be pasted back to the fifth image to obtain the final repaired image.

[0208] In actual applications, those skilled in the art can set the values ​​of the second threshold, the third threshold, the fourth threshold and the fifth threshold according to actual conditions, and the embodiments of the present application are not limited thereto.

[0209] As an example, a processing flow example of adaptive image cropping can be as follows: Fig.15 As shown, it may include:

[0210] 1) According to the user's selection operation of erasing the object area, a fifth image containing the missing area (target repair area) and a mask image S corresponding to the missing area are obtained, and the area of ​​the mask image S is calculated.

[0211] 2) When the area of ​​S is smaller than the third threshold (eg, 1024*1024*80%), the original image is cropped with S as the center into an image with a size of the second threshold (eg, 1024*1024), and input into the image post-repair processing.

[0212] 3) When the area of ​​S is larger than the third threshold (e.g. 1024*1024*80%), find a matrix region with a size of the second threshold (e.g. 1024*1024) from left to right and from top to bottom, where the area of ​​the missing region in the region is equal to or smaller than the fifth threshold (e.g. 1024*1024*80%), crop the region, and input it into the image post-repair process. Paste the repaired image of the region back to the original fifth image, and update the mask image S.

[0213] 4) If the updated S is less than the third threshold (such as 1024*1024*80%), execute step 2) again, otherwise execute step 3) again. The subsequent same processing will not be repeated.

[0214] In the embodiment of the present application, adaptive image cropping allows the model to process images with large resolution, such as images with pixels exceeding a second threshold (for example, 1024*1024), thereby avoiding the problem of insufficient memory when the model processes images with pixels exceeding the second threshold, and solving the problem that large-resolution images require large computation and cannot be run on the terminal.

[0215] Based on at least one of the above embodiments, the embodiments of the present application are Fig.16, shows an example of a complete image restoration process flow, specifically, it can include: obtaining the original image, in the user editing stage, the user can choose to erase or lose the area, and select the repair content (such as through text) to guide the model to generate the corresponding content. In the preprocessing stage, adaptive image cropping is performed according to the ratio of the missing area (target repair area) to the original image. In the image restoration stage, the corresponding text feature map is found in the database according to the text selected by the user, and combined with the preprocessing result, it is input into the fast and lightweight first AI network for denoising, and the first denoising result is output in combination with the dynamic denoising exit mechanism. In the post-processing stage, the first denoising result and the preprocessed image are input into the second AI network, and the restored image with a resolution of 1024*1024 is output by blurring denoising and enhancing texture. The restored image is pasted back to the original image to synthesize the final restored image.

[0216] Among them, the fast and lightweight first AI network converts the feature map of the time domain space into the Fourier space, extracts the receptive field of the image range in the Fourier space, and then uses a lightweight and fast CNN-based method to replace the attention operations of numerous matrix multiplications in the time domain space, significantly reducing the model size.

[0217] The dynamic denoising exit mechanism calculates the autocorrelation coefficient of the first feature map based on the noise scoring network (the third AI network), which takes into account the semantic content and context of the denoising result. When the autocorrelation coefficient reaches the defined threshold, the intermediate first denoising result that meets the requirements can be dynamically output, avoiding a large amount of denoising processing and significantly improving efficiency.

[0218] The first AI network is combined with the second AI network and uses fuzzy convolution to simulate the generation mode of the denoising branch. It can find more suitable high-frequency information to help the target repair area generate better high-frequency images, and then use the high-frequency information of the known area to enrich the first denoising processing results to ensure image quality.

[0219] The specific implementation methods of each processing stage can be found in the above introduction and will not be repeated here.

[0220] Among them, a processing flow of the user editing stage is as follows Fig.17As shown, it mainly includes: first, select the target to be removed, that is, according to the user operation, select the object area or the user-defined area as the removal area. Furthermore, then select the content that needs to be repaired in the removal area. If the user selects text (the text "white cloud" is taken as an example in this example), the text feature map corresponding to the input text is input for preprocessing, and finally the image related to the selected text is repaired. For another example, if the user chooses to fill the content as "Flowers", the removal area will be filled with the content of flowers. If the user does not select text, the image of the target repair area is finally randomly repaired.

[0221] The method provided by the embodiment of the present application and executed by an electronic device uses artificial intelligence generation technology to erase objects or targets such as people in an image, and can generate designated new targets or background content in the erased target area.

[0222] Compared with directly using an adversarially trained GAN network to repair an image, the image repair method provided in the embodiment of the present application makes it easier for the network to converge to a good state, and for images with large missing areas (target repair areas), the repaired content is more diverse, more natural, and more reasonable.

[0223] Compared with directly using a diffusion network to repair an image, the image restoration method provided in the embodiment of the present application does not need to use a large number of transformer structures (which have the characteristics of many model parameters, large models, and large amount of calculation) to calculate global features, so that the model parameters are fewer, the model is smaller, and the amount of calculation is less. In addition, due to the use of a dynamic denoising process exit mechanism, the model running time is shorter, the delay is significantly reduced, the operating efficiency is higher, and it is very easy to run on mobile devices.

[0224] The present application also provides a method executed by an electronic device, such as Fig.18a As shown, the method includes:

[0225] Step S1701: Acquire the target repair area selected by the user in the image;

[0226] In the embodiment of the present application, the user can select the object area to be removed in the image, such as Fig.14 As shown, after the object area is removed, the second image containing the target repair area mentioned above can be formed, and the removed object area is the target repair area.

[0227] Step S1702: providing the user with category information of the repaired content, and obtaining target category information selected by the user;

[0228] In an embodiment of the present application, selectable options can be provided to the user, corresponding to the category information of the repair content. When the user clicks on the option of the target category information, an instruction for the user to select the target category information (i.e., the second instruction above) can be received.

[0229] Step S1703: using the first AI network to perform repair processing on the target repair area in the image based on the target category information selected by the user;

[0230] Optionally, an image after adding noise (i.e., the first image mentioned above) is obtained, and a first AI network is used to perform at least one first denoising process on the target repair area in the image after adding noise based on the target category information selected by the user.

[0231] Specifically, based on the target category information selected by the user, the target repair content-related information corresponding to the target repair area can be determined, and then the first AI network is used to repair the target repair area in the image based on the target repair content-related information.

[0232] In the embodiment of the present application, the first AI network may be a Diffusion network. For the target repair area in the image, the first AI network may be used to repair the image step by step. The processing method and training method of the first AI network may refer to the above description and will not be repeated here.

[0233] Step S1704: using the second AI network to perform repair processing on the repair result of the first AI network;

[0234] Among them, the restoration result of the first AI network is the first denoising result mentioned above.

[0235] In the embodiment of the present application, the second AI network may be repaired in a different manner than the first AI network. For example, the second AI network may use a GAN network, and the processing process may be understood as using pixels from a known area to expand to an unknown area, but is not limited thereto. The processing method and training method of the second AI network can be found in the above description and will not be repeated here.

[0236] Step S1705: Provide the restored image to the user.

[0237] The method performed by the electronic device provided in the embodiment of the present application can be combined with the target repair content-related information corresponding to the repair content of the target repair area in the above text, a fast and lightweight denoising network, a dynamic denoising process exit mechanism and other processing methods to perform image restoration processing. The specific functional description and the beneficial effects produced can be found in the description of the corresponding method shown in the above text, which will not be repeated here.

[0238] The present application also provides a method executed by an electronic device, such as Fig.18b As shown, the method includes:

[0239] Step S1801: Acquire a first image, where the first image is an image after adding noise to the second image, and the second image includes a target repair area;

[0240] In an embodiment of the present application, the target repair area refers to the area in the image to be repaired (also referred to as restoration), for example, it may be a lost area or a removed area, but is not limited thereto. Among them, the lost area may refer to an area where some information in the image is missing or damaged due to various factors such as occlusion, blur, transmission interference, etc. The removed area may refer to a blank area formed after erasing part of the target object (such as an object, a person, a building, etc., but not limited thereto) in the image. The erased target object may be automatically selected or manually selected, and the embodiment of the present application is not limited thereto. Optionally, a corresponding mask image may be determined for the target repair area, which is used to determine the position of the target repair area in various images.

[0241] In the embodiment of the present application, the second image refers to the image to be repaired including the target repair area. Optionally, the second image can directly use the image including the original target repair area, or the image including the original target repair area can be processed to a certain extent and then used as the second image for repair processing. As an example, if the area of ​​the original target repair area is too large, the image including the original target repair area can be cut to reduce the area of ​​the target repair area in the cut image, and the cut image can be used as a second image for repair processing; as an example, if the size of the image including the original target repair area is too large, the image including the original target repair area can also be cut or scaled or deformed to reduce the size of the cut image, and the cut image can be used as a second image for repair processing, thereby reducing the amount of calculation, which is not limited in the embodiment of the present application.

[0242] In the embodiment of the present application, the first image is an image after adding noise (noise) to the second image, that is, the noise and the second image are fused, and the first image obtained can be understood as an image completely transformed into noise. The added noise can be generated randomly or according to a predetermined algorithm, and the fusion method can be channel splicing or superposition, etc. The implementation of the present application does not specifically limit the noise generation method and fusion method.

[0243] Step S1802: using an attention module to calculate the attention corresponding to the first image through Fourier transform, where the attention represents the degree of denoising of the first image;

[0244] The attention module may adopt an FFA module, but is not limited thereto.

[0245] In an embodiment of the present application, the attention module can calculate the self-attention of the first image, which is used to perform specific denoising on the target repair area based on the information of the first image itself.

[0246] Step S1803: Perform at least one third denoising process based on the attention calculation result on the first image to obtain an image after the target repair area is repaired.

[0247] In the embodiment of the present application, for the first input image, noise can be removed step by step, that is, only a little content can be repaired at a time, and the current content repair is performed on the previous repair result. As the denoising process is continuously performed based on the attention calculation results, the noise in the denoising result becomes less and less, and the restored semantic content becomes more and more, until the denoising is completed and the image of the target repair area is obtained.

[0248] In the implementation of the present application, an optional implementation is provided for step S1802. Specifically, calculating the attention corresponding to the first image through Fourier transform includes:

[0249] Step S18021: performing a fast Fourier transform on the noise feature map corresponding to the first image to obtain a Fourier feature map;

[0250] That is, through FFT, the noise feature map corresponding to the first image is converted from the time domain space to the Fourier space (frequency domain space). The noise feature map corresponding to the first image may refer to a feature map extracted from the first image, or may refer to a feature map extracted from a denoising result obtained by performing denoising processing on the first image any number of times.

[0251] Step S18022: performing a first convolution operation on the Fourier feature map to obtain a Fourier space query (Query) feature, a Fourier space key (Key) feature, and a Fourier space value (Value) feature;

[0252] In the embodiment of the present application, the first convolution operation may refer to using a 1*1 convolution kernel (Conv 1*1) to perform a convolution operation. This is because the feature map in the time domain is converted to the frequency domain space using Fourier transform, so that each value of the feature map can represent the global features of the image. Based on this feature, using a 1*1 convolution kernel for convolution operation can achieve the global feature map information that can only be obtained using a fully connected layer in the time domain space.

[0253] Step S18023: performing a second convolution operation on the Fourier space query feature and the Fourier space key feature to obtain a first attention weight coefficient;

[0254] Specifically, the Fourier space query feature and the Fourier space key feature may be directly concatenated (Concat) together, and a second convolution operation may be performed on the concatenated result.

[0255] In the embodiment of the present application, the second convolution operation may also refer to the use of a 1*1 convolution kernel (Conv1*1) for convolution operations. In the time domain, a huge matrix multiplication operation is required to fuse the Query feature graph, the Key feature graph, and the Value feature graph. In the embodiment of the present application, after the feature graph of the time domain space is mapped to the frequency domain space, the information of different channels can be mixed in the frequency domain space through the convolution operation of the 1*1 convolution kernel, so as to achieve the effect of approximating the matrix multiplication of the Query feature graph and the Key feature graph, thereby reducing the amount of calculation.

[0256] Step S18024: performing an inverse fast Fourier transform on the first attention weight coefficient to obtain a second attention weight coefficient;

[0257] That is, through IFFT (Inverse Fast Fourier Transform), the first attention weight coefficient is converted from the frequency domain space back to the time domain space to obtain the second attention weight coefficient.

[0258] Step S18025: Based on the second attention weight coefficient and the Fourier space value feature, the attention calculation result is obtained.

[0259] The values ​​in the Fourier space value feature are weighted according to the second attention weight coefficient to obtain the attention calculation result as the calculation result output by FFA.

[0260] For the embodiment of the present application, since FFT and IFFT are parameter-free, their time complexity is negligible compared to other computational costs, so the increase in computational complexity caused by FFT and IFFT can be ignored. Therefore, the parameters of the model can be greatly reduced, thereby reducing the size of the model.

[0261] The image restoration method provided in the embodiment of the present application uses a fast and lightweight network to make the model smaller and the amount of calculation smaller, thereby solving the problem that the denoising model is usually large in size and the amount of calculation is large.

[0262] The fast and lightweight denoising network provided in the embodiment of the present application can be used in the above-mentioned first AI network, and can be combined with the target repair content-related information corresponding to the repair content of the target repair area, the dynamic denoising process exit mechanism, and the second AI network to perform image restoration processing. The specific functional description and the beneficial effects produced can be found in the description of the corresponding method shown in the previous text, which will not be repeated here.

[0263] The fast and lightweight first AI network provided in the embodiment of the present application is very effective in reducing the amount of calculation, and the fast Fourier attention module significantly reduces the model calculation parameters and the amount of calculation.

[0264] The combination of the dynamic denoising process exit mechanism and the second AI network provided in the embodiment of the present application is very effective in improving operating efficiency, the model is significantly smaller, and the operating time is significantly reduced.

[0265] In addition, the traditional diffusion network mainly processes 512*512 images, and the processing results of 1024*1024 images need to be enlarged by interpolation, which leads to blur. And for larger target repair areas, the repair processing results of the traditional GAN ​​network contain a lot of artifacts. Compared with the traditional diffusion network on the 1024*1024 image repair processing results, the present application scheme has a clearer and more natural repair effect. Compared with the image repair processing results of the traditional GAN ​​network, the repair content is more diverse, more natural, and the content is more reasonable.

[0266] The image restoration method provided in the embodiment of the present application can be used to infer the content of the target restoration area after the image is rotated, can be used to generate perspective images, and can be used to restore super-resolution images, etc., but is not limited to this.

[0267] The technical solution provided in the embodiments of the present application can be applied to various electronic devices, including but not limited to mobile terminals, smart terminals, etc., such as smart phones, tablet computers, laptops, smart wearable devices (such as watches, glasses, etc.), smart speakers, vehicle terminals, personal digital assistants, portable multimedia players, navigation devices, etc., but not limited to these. It can be understood by those skilled in the art that, in addition to components specifically used for mobile purposes, the structure according to the embodiments of the present application can also be applied to fixed-type terminals, such as digital televisions, desktop computers, etc.

[0268] The technical solution provided in the embodiments of the present application can also be applied to image restoration in a server, such as an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), as well as big data and artificial intelligence platforms.

[0269] Specifically, the technical solution provided in the embodiments of the present application can be applied to image AI editing applications on various electronic devices to improve the performance of image repair in large target repair areas.

[0270] An embodiment of the present application also provides an electronic device, which includes a processor and, optionally, may also include a transceiver and / or a memory coupled to the processor, and the processor is configured to execute the steps of the method provided in any optional embodiment of the present application.

[0271] Fig.19 A schematic diagram of the structure of an electronic device applicable to an embodiment of the present invention is shown in FIG. Fig.19 As shown, Fig.19 The electronic device 4000 shown includes: a processor 4001 and a memory 4003. The processor 4001 and the memory 4003 are connected, such as through a bus 4002. Optionally, the electronic device 4000 may also include a transceiver 4004, which can be used for data interaction between the electronic device and other electronic devices, such as data transmission and / or data reception. It should be noted that in actual applications, the transceiver 4004 is not limited to one, and the structure of the electronic device 4000 does not constitute a limitation on the embodiments of the present application. Optionally, the electronic device may be a first network node, a second network node, or a third network node.

[0272] Processor 4001 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array) or other programmable logic devices, transistor logic devices, hardware components or any combination thereof. It may implement or execute various exemplary logic blocks, modules and circuits described in conjunction with the disclosure of this application. Processor 4001 may also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, etc.

[0273] The bus 4002 may include a path to transmit information between the above components. The bus 4002 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus. The bus 4002 may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Fig.19 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0274] The memory 4003 may be a ROM (Read Only Memory) or other types of static storage devices that can store static information and instructions, a RAM (Random Access Memory) or other types of dynamic storage devices that can store information and instructions, or an EEPROM (Electrically Erasable Programmable Read Only Memory), a CD-ROM (Compact Disc Read Only Memory) or other optical disk storage, optical disk storage (including compressed optical disk, laser disk, optical disk, digital versatile disk, Blu-ray disk, etc.), magnetic disk storage media, other magnetic storage devices, or any other medium that can be used to carry or store computer programs and can be read by a computer, without limitation herein.

[0275] The memory 4003 is used to store the computer program for executing the embodiment of the present application, and the execution is controlled by the processor 4001. The processor 4001 is used to execute the computer program stored in the memory 4003 to implement the steps shown in the above method embodiment.

[0276] An embodiment of the present application provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps and corresponding contents of the aforementioned method embodiment can be implemented.

[0277] The embodiment of the present application also provides a computer program product, including a computer program, which can implement the steps and corresponding contents of the aforementioned method embodiment when executed by a processor.

[0278] The terms "first", "second", "third", "fourth", "1", "2", etc. (if any) in the specification and claims of this application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the numbers used in this way can be interchanged where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than that shown or described in the drawings.

[0279] It should be understood that, although each operation step is indicated by arrows in the flowchart of the embodiment of the present application, the implementation order of these steps is not limited to the order indicated by the arrows. Unless clearly stated herein, in some implementation scenarios of the embodiment of the present application, the implementation steps in each flowchart can be performed in other orders according to demand. In addition, some or all of the steps in each flowchart may include multiple sub-steps or multiple stages based on actual implementation scenarios. Some or all of these sub-steps or stages may be executed at the same time, and each sub-step or stage in these sub-steps or stages may also be executed at different times respectively. In different scenarios of execution time, the execution order of these sub-steps or stages may be flexibly configured according to demand, and the embodiment of the present application does not limit this.

[0280] The above text and drawings are provided only as examples to help readers understand the present application. They are not intended to and should not be interpreted as limiting the scope of the present application in any way. Although certain embodiments and examples have been provided, based on the contents disclosed herein, it is obvious to those skilled in the art that the embodiments and examples shown can be changed without departing from the scope of the present application, and other similar implementation means based on the technical ideas of the present application are adopted, which also fall within the protection scope of the embodiments of the present application.

Claims

1. A method performed by an electronic device, characterized in that: include: Acquire a first image, where the first image is an image obtained by adding noise to the second image, and the second image includes a target repair area; Using a first artificial intelligence AI network, performing at least one first denoising process on the first image to obtain a first denoising process result; Using a second AI network, based on the first denoising result, the target repair area is repaired to obtain a repaired image.

2. The method according to claim 1, characterized in that Also includes: receiving a first instruction from a user to select a target repair area in a third image; Based on the first instruction, the target repair area and a second image including the target repair area are determined.

3. The method according to claim 1 or 2, characterized in that: The using the first AI network to perform at least one first denoising process on the first image includes: Determine the target repair content related information corresponding to the target repair area; Based on the target restoration content related information, a first AI network is used to perform at least one first denoising process on the first image.

4. The method according to claim 3, characterized in that: The determining of the target repair content related information corresponding to the target repair area includes: Provide users with category information of repaired content; receiving a second instruction from a user to select target category information; Based on the second instruction, information related to the target repair content corresponding to the target repair area is determined.

5. The method according to claim 3 or 4, characterized in that: The target repair content related information includes a content feature map corresponding to the target repair content.

6. The method according to any one of claims 3 to 5, characterized in that: The first AI network includes at least one attention module, and the first AI network is used to perform at least one first denoising process on the first image based on the target restoration content related information, including: Using the at least one attention module, the attention between the noise feature map corresponding to the first image and the target restoration content related information is calculated through Fourier transform, the attention representation indicates the degree of denoising of the noise feature map according to the target restoration content related information, and the first image is subjected to at least one first denoising process based on the attention calculation result.

7. The method according to claim 6, characterized in that Calculating the attention between the noise feature map corresponding to the first image and the target restoration content related information through Fourier transform, including: Performing a fast Fourier transform on the noise feature map corresponding to the first image and the target restoration content related information to obtain a Fourier feature map; Performing a first convolution operation on the Fourier feature map to obtain a Fourier space query feature, a Fourier space key feature, and a Fourier space value feature; Performing a second convolution operation on the Fourier space query feature and the Fourier space key feature to obtain a first attention weight coefficient; Performing an inverse fast Fourier transform on the first attention weight coefficient to obtain a second attention weight coefficient; Based on the first attention weight coefficient and the Fourier space value feature, an attention calculation result is obtained.

8. The method according to any one of claims 1 to 7, characterized in that: The using the first AI network to perform at least one first denoising process on the first image includes: Using at least one first AI network, sequentially performing a first denoising process on the first images; For each processing result output by the first AI network, a first feature map of the processing result and a second feature map of a third image with a predetermined proportion of noise added are extracted, where the third image is an image of the second image before the target repair area is removed, and based on the similarity between the first feature map and the second feature map, it is determined whether to continue using the first AI network to perform the first denoising process.

9. The method according to claim 8, characterized in that Extracting a first feature map of the processing result includes: Using a third AI network, extracting at least two levels of first feature graphs from the processing result, and calculating the autocorrelation coefficient of each first feature graph; Extracting a second feature map of a third image to which noise of a predetermined proportion is added, comprising: Using a third AI network, extracting at least two levels of second feature maps from the third image to which noise of a predetermined proportion is added, and calculating the autocorrelation coefficient of each second feature map; The determining whether to continue to use the first AI network to perform the first denoising process based on the similarity between the first feature map and the second feature map includes: Determining the distance between the autocorrelation coefficients of the first characteristic graphs and the autocorrelation coefficients of the second characteristic graphs; Based on the relationship between the distance and the first threshold, it is determined whether to continue using the first AI network to perform the first denoising process.

10. The method according to any one of claims 1 to 9, characterized in that: The using the second AI network to perform restoration processing on the target restoration area based on the first denoising processing result includes: Performing a second denoising process on the first denoising process result to obtain a second denoising process result; determining texture information of the second image from the region not requiring restoration; Extracting a third feature map of the target repair area from the second denoising result; Based on the texture information and the third feature map, the target repair area is repaired to obtain a repaired image.

11. The method according to claim 10, characterized in that The step of performing a repair process on the target repair area based on the texture information and the third feature map to obtain a repaired image includes: Dividing the feature map corresponding to the texture information and the third feature map into image blocks of the same size to obtain first sub-feature maps and second sub-feature maps; Calculate the similarity between each first sub-feature map and each second sub-feature map, and obtain an adaptive weight corresponding to each pair of the first sub-feature map and the second sub-feature map; The first sub-feature maps corresponding to each second sub-feature map are respectively fused with the corresponding adaptive weights to obtain the second sub-feature maps after texture enhancement; Based on each of the second sub-feature maps after texture enhancement, the target repair area is repaired to obtain a repaired image.

12. The method according to claim 11, characterized in that The step of performing a repair process on the target repair area based on each second sub-feature map after texture enhancement to obtain a repaired image includes: Each second sub-feature map and the target restoration area of ​​the second denoising process result are element-added to obtain a restored image.

13. The method according to claim 10, characterized in that The step of determining texture information of the second image from the region that does not need to be repaired comprises: Using a third AI network, extracting a fourth feature map of at least one previous level from the first denoising result; Performing a third convolution operation on the second image and each fourth feature map to determine non-texture information of the second image from the region that does not need to be repaired; Texture information of the second image is determined based on the second image and non-texture information of the second image.

14. The method according to claim 13, characterized in that The performing a third convolution operation on the second image and each fourth feature map comprises: Performing a corresponding number of downsampling operations and upsampling operations on the second image to obtain non-texture information of the second image; For at least one downsampling operation, the downsampling result and the fourth feature map of the corresponding scale are element-wise averaged to obtain an average feature map, and the next downsampling operation or upsampling operation is performed on the average feature map.

15. The method according to any one of claims 1 to 14, characterized in that: Before acquiring the first image, the method further includes: Acquire a fifth image including the target repair area; If the size of the fifth image is greater than a second threshold, obtaining the second image from the fifth image by at least one of the following cropping methods; If the area of ​​the target repair region is smaller than the third threshold, and the length of the target repair region is smaller than the fourth threshold, the fifth image is cropped into an image with a size of the second threshold with the target repair region as the center, to obtain the second image; If the area of ​​the target repair area is smaller than the third threshold, the length of the target repair area is greater than the fourth threshold, or if the area of ​​the target repair area is greater than the third threshold, determine that the size of the image area in the fifth image is the second threshold, and the area of ​​the target repair area in the image area is not greater than the fifth threshold, crop the image area to obtain the second image, and use the target repair area in the image area as the target repair area of ​​the second image.

16. The method according to claim 15, characterized in that Also includes: The image after the target repair area corresponding to the image area is repaired is fused into the fifth image to obtain a fifth image after the target repair area is updated; The second image is acquired from the fifth image after the target repair area is updated through the at least one cropping method.

17. A method performed by an electronic device, characterized in that: include: Get the target repair area selected by the user in the image; Providing the user with category information of the repaired content, and obtaining the target category information selected by the user; Using a first AI network, based on target category information selected by a user, repairing a target repair area in the image; Using the second AI network, repairing the repair result of the first AI network; The restored image is provided to the user.

18. The method according to claim 17, characterized in that The first AI network is a diffusion network.

19. A method performed by an electronic device, characterized in that: include: Acquire a first image, where the first image is an image obtained by adding noise to the second image, and the second image includes a target repair area; Using an attention module, by Fourier transform, calculating attention corresponding to the first image, wherein the attention represents a degree of denoising of the first image; Perform at least one third denoising process based on the attention calculation result on the first image to obtain an image after the target repair area is repaired.

20. An electronic device comprising a memory, a processor and a computer program stored in the memory, characterized in that: The processor executes the computer program to implement the method according to any one of claims 1 to 19.

21. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 19 is implemented.

22. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the method according to any one of claims 1 to 19 is implemented.

Images