Image inpainting method, device, electronic equipment, storage medium and program product

Abstract

Embodiments of the present application provide a kind of image restoration method, device, electronic equipment, storage medium and program product.The method comprises: obtaining to-be-processed image, and based on superpixel segmentation technology, to-be-processed image is carried out multi-granularity segmentation processing, obtain multiple segmented to-be-processed images;According to the multiple image blocks included in each segmented to-be-processed image, to-be-processed image is carried out noise identification processing, obtain the result of noise identification processing, according to the result of noise identification processing and to-be-processed image, generate multiple noise mask images;Wherein, each noise mask image indicates different granularity;The granularity indicated by noise mask image corresponds to the granularity indicated by segmented to-be-processed image;According to multiple noise mask images, to-be-processed image is carried out noise restoration processing, obtain the to-be-processed image of repair completion.The method is used to reach the effect of improving the accuracy of image restoration.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, and particularly relates to a method and device for image restoration, an electronic device, a storage medium and a program product. BACKGROUND

[0002] With the development of science and technology, the images in the film can be processed through scanning technology to generate digital images. However, in the process of using the film, due to the aging of the film material, repeated playing, improper handling and other reasons, scratches and cracks, color distortion and fading, and dirt and dust may appear on the surface of the film. Therefore, the generated digital pictures may also have the above problems.

[0003] In the prior art, an autoencoder is used to perform image restoration processing on the digital image. The autoencoder is a neural network that learns data encoding by compressing and decompressing data. In the image restoration process, the autoencoder learns a low-dimensional representation of the damaged picture and then reconstructs the restored picture.

[0004] However, the autoencoder may lose some high-frequency details in the reconstruction process, resulting in a restored image that is too smooth and lacks details, and is not accurate enough. SUMMARY

[0005] The embodiments of the present application provide a method and device for image restoration, an electronic device, a storage medium and a program product to improve the accuracy of image restoration.

[0006] In a first aspect, the embodiments of the present application provide a method for image restoration, comprising:

[0007] obtaining a to-be-processed image, and performing multi-granularity segmentation processing on the to-be-processed image based on a superpixel segmentation technology to obtain a plurality of segmented to-be-processed images; wherein each segmented to-be-processed image indicates a different granularity; each segmented to-be-processed image includes a plurality of image blocks; wherein the granularity of each image block included in each segmented to-be-processed image corresponds to the granularity indicated by the corresponding segmented to-be-processed image;

[0008] performing noise identification processing on the to-be-processed image according to the plurality of image blocks included in each segmented to-be-processed image to obtain a noise identification processing result, and generating a plurality of noise mask images according to the noise identification processing result and the to-be-processed image; wherein each noise mask image indicates a different granularity; the granularity indicated by the noise mask image corresponds to the granularity indicated by the segmented to-be-processed image;

[0009] According to the plurality of noise mask images, noise repair processing is performed on the to-be-processed image, and a repaired to-be-processed image is obtained.

[0010] In a possible implementation, according to the plurality of image blocks included in each of the segmented to-be-processed images, noise identification processing is performed on the to-be-processed image, and a noise identification processing result is obtained. The noise identification processing includes: obtaining an image block feature vector of each of the plurality of image blocks included in each of the segmented to-be-processed images; and performing noise identification processing on the to-be-processed image according to a noise identification algorithm and the image block feature vector of each of the plurality of image blocks included in each of the segmented to-be-processed images, to obtain the noise identification processing result. The noise identification processing result includes an image block noise identification parameter of each of the plurality of image blocks included in each of the segmented to-be-processed images. The image block noise identification parameter indicates whether the image block contains noise.

[0011] In a possible implementation, according to the noise identification processing result and the to-be-processed image, a plurality of noise mask images are generated. The generation includes: performing noise mask image generation processing on the to-be-processed image according to the image block noise identification parameter of each of the plurality of image blocks included in each of the segmented to-be-processed images included in the noise identification processing result, to obtain a plurality of initial noise mask images; obtaining a position feature vector of each of the plurality of image blocks included in each of the segmented to-be-processed images; and performing correction processing on the obtained plurality of initial noise mask images according to the position feature vector of each of the plurality of image blocks included in each of the segmented to-be-processed images, to obtain the plurality of noise mask images.

[0012] In a possible implementation, the position feature vector of each of the plurality of image blocks included in each of the segmented to-be-processed images is obtained by: calculating a position similarity parameter between each of the plurality of image blocks included in each of the segmented to-be-processed images and each of the plurality of image blocks included in other segmented to-be-processed images; and generating the position feature vector of each of the plurality of image blocks included in each of the segmented to-be-processed images according to the calculated position similarity parameter.

[0013] In a possible implementation, the method further includes: determining image blocks in the same region range in each of the processed images according to the position feature vector of each of the image blocks included in each of the processed images; and performing correction processing on the plurality of initial noise mask images according to the image block noise identification parameters of the image blocks in the same region range in each of the processed images to obtain the plurality of noise mask images.

[0014] In a possible implementation, the method further includes: obtaining a position feature vector of each of the image blocks included in each of the processed images; determining a plurality of to-be-identified regions of the image to be processed according to the position feature vector of each of the image blocks included in each of the processed images; wherein the to-be-identified region includes at least one image block in at least two of the processed images; obtaining an image block feature vector of each of the image blocks included in each of the processed images, and determining a region feature vector of each of the to-be-identified regions according to the image block feature vector of each of the image blocks included in each of the processed images; and performing noise identification processing on the image to be processed according to a noise identification algorithm and the region feature vector of each of the to-be-identified regions to obtain a noise identification processing result; wherein the noise identification processing result includes a region noise identification parameter of each of the to-be-identified regions; and wherein the region noise identification parameter indicates whether the to-be-identified region contains noise.

[0015] In a possible implementation, the method further includes: calculating a position similarity parameter between each of the image blocks included in each of the processed images and each of the image blocks included in other processed images; and generating the position feature vector of each of the image blocks included in each of the processed images according to the calculated position similarity parameter.

[0016] In a possible implementation, the method further includes: generating a plurality of noise mask images according to the results of the noise identification processing and the to-be-processed image, including: determining, according to a region noise identification parameter of each to-be-identified region included in the results of the noise identification processing, an image block noise identification parameter of each image block in a plurality of image blocks included in each of the to-be-processed images after the segmentation processing; and generating a plurality of noise mask images according to the image block noise identification parameter of each image block in the plurality of image blocks included in each of the to-be-processed images after the segmentation processing and the to-be-processed image.

[0017] In a possible implementation, the method further includes: performing noise repair processing on the to-be-processed image according to the plurality of noise mask images to obtain a to-be-processed image after repair completion, including: performing, respectively, a masking processing on the to-be-processed image and each of the plurality of noise mask images to obtain a to-be-processed image after the masking processing corresponding to each of the plurality of noise mask images; performing, respectively, a noise optimization processing on each of the to-be-processed images after the masking processing to obtain a to-be-processed image after the noise optimization processing corresponding to each of the to-be-processed images after the masking processing; and performing, based on an attention mechanism, a noise edge blurring processing on all the to-be-processed images after the noise optimization processing to obtain the to-be-processed image after repair completion.

[0018] In a possible implementation, the method further includes: performing, respectively, a noise optimization processing on each of the to-be-processed images after the masking processing to obtain a to-be-processed image after the noise optimization processing corresponding to each of the to-be-processed images after the masking processing, including: performing, respectively, a non-structure noise optimization processing on each of the to-be-processed images after the masking processing to obtain a to-be-processed image after the non-structure noise optimization processing corresponding to each of the to-be-processed images after the masking processing; and performing, respectively, a structure noise optimization processing on each of the to-be-processed images after the non-structure noise optimization processing to obtain a to-be-processed image after the noise optimization processing corresponding to each of the to-be-processed images after the masking processing.

[0019] In a possible implementation, the method further includes: determining a loss function value according to the to-be-processed image and the image after completion of the repairing; and optimizing algorithm parameters of each algorithm in the image repairing process according to the loss function value.

[0020] In a possible implementation, the method further includes: determining a loss function value according to the to-be-processed image and the image after completion of the repairing; and optimizing algorithm parameters of each algorithm in the image repairing process according to the loss function value.

[0021] In a possible implementation, the method further includes: determining a loss function value according to the to-be-processed image and the image after completion of the repairing; and optimizing algorithm parameters of each algorithm in the image repairing process according to the loss function value.

[0022] In a possible implementation, the method further includes: determining a loss function value according to the to-be-processed image and the image after completion of the repairing; and optimizing algorithm parameters of each algorithm in the image repairing process according to the loss function value.

[0023] In a second aspect, an embodiment of the present application provides an image repairing device, including:

[0024] The segmentation module is configured to acquire a to-be-processed image, and perform multi-granularity segmentation processing on the to-be-processed image based on a superpixel segmentation technology to obtain a plurality of to-be-processed images after segmentation processing; each to-be-processed image after segmentation processing indicates a different granularity; each to-be-processed image after segmentation processing includes a plurality of image blocks; and the granularity of each image block included in each to-be-processed image after segmentation processing corresponds to the granularity indicated by the corresponding to-be-processed image after segmentation processing.

[0025] The identification module is configured to perform noise identification processing on the to-be-processed image according to the plurality of image blocks included in each to-be-processed image after segmentation processing to obtain a result of noise identification processing, and generate a plurality of noise mask images according to the result of noise identification processing and the to-be-processed image; each noise mask image indicates a different granularity; and the granularity indicated by the noise mask image corresponds to the granularity indicated by the to-be-processed image after segmentation processing.

[0026] The repair module is configured to perform noise repair processing on the to-be-processed image according to the plurality of noise mask images to obtain a to-be-processed image after repair.

[0027] In a third aspect, an electronic device is provided, including a memory and a processor.

[0028] The memory stores computer-executable instructions.

[0029] The processor executes the computer-executable instructions stored in the memory, so that the processor performs the method according to the first aspect and / or various possible implementation manners of the first aspect.

[0030] In a fourth aspect, a computer-readable storage medium is provided, which stores computer-executable instructions. When the computer-executable instructions are executed by a processor, the computer-executable instructions are used to implement the method according to the first aspect and / or various possible implementation manners of the first aspect.

[0031] In a fifth aspect, a computer program product is provided, which includes a computer program. When the computer program is executed by a processor, the computer program implements the method according to the first aspect and / or various possible implementation manners of the first aspect.

[0032] The image restoration method, apparatus, electronic device, storage medium, and program product provided in this application embodiment acquire an image to be processed and perform multi-granularity segmentation processing on the image to be processed based on superpixel segmentation technology to obtain multiple segmented images to be processed. Based on the multiple image blocks included in each segmented image to be processed, noise recognition processing is performed on the image to be processed to obtain the noise recognition processing result. Based on the noise recognition processing result and the image to be processed, multiple noise mask images are generated. Based on the multiple noise mask images, noise restoration processing is performed on the image to be processed to obtain the restored image to be processed. In this way, performing noise recognition processing and noise restoration processing based on multiple segmented images to be processed obtained from multi-granularity segmentation processing can improve the accuracy of image restoration. Attached Figure Description

[0033] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0034] Figure 1 Flowchart of the image restoration method provided in this application Figure One ;

[0035] Figure 2 Flowchart of the image restoration method provided in this application Figure Two ;

[0036] Figure 3 Flowchart of the image restoration method provided in this application Figure Three ;

[0037] Figure 4 Flowchart of the image restoration method provided in this application Figure Four ;

[0038] Figure 5 Flowchart of the image restoration method provided in this application Figure Five ;

[0039] Figure 6 Flowchart of the image restoration method provided in this application Figure Six ;

[0040] Figure 7 This is a schematic diagram of the network structure for noise optimization processing of the image to be processed after masking.

[0041] Figure 8 Here is an example diagram of the structure of a Fast Fourier Convolutional Layer;

[0042] Figure 9 A schematic diagram of the image restoration apparatus provided in this application;

[0043] Figure 10 A schematic diagram of the structure of the electronic device provided in this application.

[0044] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0045] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0046] In existing technologies, image inpainting is performed on digitized images using autoencoders, which are neural networks that learn data encoding by compressing and decompressing data. During image inpainting, the autoencoder learns a low-dimensional representation of the damaged image and then reconstructs the restored image. However, autoencoders may lose some high-frequency details during reconstruction, resulting in an overly smooth, detail-lacking, and inaccurate restored image.

[0047] The image restoration method provided in this application acquires an image to be processed and performs multi-granularity segmentation processing on the image to be processed based on superpixel segmentation technology to obtain multiple segmented images to be processed. Based on the multiple image blocks included in each segmented image to be processed, noise recognition processing is performed on the image to be processed to obtain the noise recognition processing result. Based on the noise recognition processing result and the image to be processed, multiple noise mask images are generated. Based on the multiple noise mask images, noise restoration processing is performed on the image to be processed to obtain the restored image to be processed. In this way, performing noise recognition processing and noise restoration processing based on multiple segmented images to be processed obtained from multi-granularity segmentation processing can improve the accuracy of image restoration.

[0048] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will now be described with reference to the accompanying drawings.

[0049] Figure 1 Flowchart of the image restoration method provided in this application Figure OneAs shown in Figure 1 The method comprises the following steps:

[0050] In step S101, a to-be-processed image is acquired, and a multi-granularity segmentation processing is performed on the to-be-processed image based on a superpixel segmentation technology, to obtain a plurality of to-be-processed images after segmentation processing.

[0051] Specifically, a to-be-processed image can be acquired, and a multi-granularity segmentation processing is performed on the to-be-processed image based on a superpixel segmentation technology, to obtain a plurality of to-be-processed images after segmentation processing. The to-be-processed image is an image to be processed, for example, in the processing process of the film strip described above, the to-be-processed image can be an image obtained after scanning the film strip. The superpixel segmentation technology (superpixelsSegmentation) is also called computer vision technology. In the field of computer vision, image segmentation (Segmentation) refers to the process of subdividing a digital image into a plurality of image sub-regions (a set of pixels) (also known as superpixels). The superpixels are composed of a series of small regions of adjacent pixels with similar color, brightness, texture and other characteristics. These small regions mostly retain effective information for further image segmentation, and generally do not damage the boundary information of objects in the image. Specifically, the result of image segmentation is a set of sub-regions on the image (the entire set of sub-regions covers the entire image), or a set of contour lines extracted from the image (such as edge detection). Each pixel in a sub-region is similar in the measurement of a certain characteristic or the calculated characteristic, such as color, brightness, texture. Adjacent regions are very different in the measurement of a certain characteristic.

[0052] In the present application, the process of acquiring the to-be-processed image is not limited. Based on the above description of the to-be-processed image, the process of acquiring the to-be-processed image is different when the type of the to-be-processed image is different. For example, in the processing process of the film strip described above, a plurality of to-be-processed images can be obtained by scanning the film strip according to a scanning technology, such as a high-resolution scanning technology.

[0053] The granularity is the size of the particle, also known as "grain size" or "diameter". When the physical characteristics or physical behavior of the measured particle is most similar to that of a homogeneous sphere (or combination) of a certain diameter, the diameter of the sphere (or combination) is taken as the equivalent particle diameter (or particle size distribution) of the measured particle. In this application, the granularity refers to the granularity of the image block obtained after superpixel segmentation processing. The multi-granularity segmentation processing refers to superpixel segmentation processing of the to-be-processed image based on multiple granularities. In this application, the number of to-be-processed images after segmentation processing, i.e. the number of granularities corresponding to the multi-granularity segmentation processing, is not limited. Any number of granularities corresponding to the to-be-processed image after segmentation processing can be obtained based on the obtained multiple to-be-processed images after segmentation processing, which can be used as the number of to-be-processed images after segmentation processing, i.e. the number of granularities corresponding to the multi-granularity segmentation processing. Each to-be-processed image after segmentation processing indicates a different granularity. Each to-be-processed image after segmentation processing includes multiple image blocks. The granularity of each image block included in each to-be-processed image after segmentation processing corresponds to the granularity indicated by the corresponding to-be-processed image after segmentation processing. The number of image blocks included in the to-be-processed image after segmentation processing corresponds to the size of the to-be-processed image and the granularity of the superpixel segmentation processing. Specifically, for the same to-be-processed image, the smaller the granularity of the superpixel segmentation processing, the more the number of image blocks included in the corresponding to-be-processed image after segmentation processing, and the larger the granularity of the superpixel segmentation processing, the fewer the number of image blocks included in the corresponding to-be-processed image after segmentation processing.

[0054] The sub-region, i.e. the image block, obtained by segmentation processing through the superpixel segmentation technology is larger than the traditional pixel processing unit, which can better reflect the local features and structural information of the image, thereby improving the accuracy of noise recognition and noise repair and further improving the accuracy of image repair.

[0055] The superpixel technology also simplifies the complexity of the image, which is particularly important when processing large-scale image data. By reducing the amount of data processed, the noise detection and segmentation tasks can be completed faster, thereby improving the overall efficiency and response speed of the algorithm. Specifically, the simplified image structure not only speeds up the noise processing process, but also provides a clearer working basis for subsequent image reconstruction and optimization.

[0056] In step S102, noise recognition processing is performed on the to-be-processed image according to the multiple image blocks included in each to-be-processed image after segmentation processing, to obtain a noise recognition processing result, and multiple noise mask images are generated based on the noise recognition processing result and the to-be-processed image.

[0057] Specifically, according to the plurality of segmented processed to-be-processed images obtained in step S101, noise recognition processing is performed on the to-be-processed images, and a noise recognition processing result is obtained. The process of performing noise recognition processing on the to-be-processed images to obtain the noise recognition processing result is not limited in the present application, and any process capable of performing noise recognition processing on the to-be-processed images to obtain the noise recognition processing result can be used as the process of performing noise recognition processing on the to-be-processed images to obtain the noise recognition processing result provided in the present application. Optionally, an image block feature vector of each image block in the plurality of image blocks included in each segmented processed to-be-processed image can be obtained first; then, noise recognition processing is performed on the to-be-processed images according to a noise recognition algorithm and the image block feature vector of each image block in the plurality of image blocks included in each segmented processed to-be-processed image, and a noise recognition processing result is obtained; wherein the noise recognition processing result includes an image block noise recognition parameter of each image block in the plurality of image blocks included in each segmented processed to-be-processed image; wherein the image block noise recognition parameter indicates whether the image block contains noise.

[0058] Specifically, after obtaining the noise recognition processing result, a plurality of noise mask images can be generated according to the noise recognition processing result and the to-be-processed images. Each noise mask image indicates a different granularity. The granularity indicated by the noise mask image corresponds to the granularity indicated by the segmented processed to-be-processed image, that is, the noise mask image and the segmented processed to-be-processed image correspond to each other, and the corresponding noise mask image and segmented processed to-be-processed image indicate the same granularity. The process of generating a plurality of noise mask images according to the noise recognition processing result and the to-be-processed images is not limited in the present application, and any process capable of generating a plurality of noise mask images according to the noise recognition processing result and the to-be-processed images can be used as the process of generating a plurality of noise mask images according to the noise recognition processing result and the to-be-processed images provided in the present application. Optionally, noise mask image generation processing can be performed on the to-be-processed images according to the image block noise recognition parameter of each image block in the plurality of image blocks included in each segmented processed to-be-processed image included in the noise recognition processing result, and a plurality of initial noise mask images are obtained; then, a position feature vector of each image block in the plurality of image blocks included in each segmented processed to-be-processed image is obtained; then, the plurality of initial noise mask images obtained are corrected according to the position feature vector of each image block in the plurality of image blocks included in each segmented processed to-be-processed image, and a plurality of noise mask images are obtained.

[0059] Step S103: performing noise repair processing on the to-be-processed images according to the plurality of noise mask images, and obtaining a repaired to-be-processed image.

[0060] Specifically, according to the plurality of noise mask images obtained in step S102, the noise repair processing can be performed on the to-be-processed image to obtain the to-be-processed image after repair. Wherein, the process of performing the noise repair processing on the to-be-processed image according to the plurality of noise mask images to obtain the to-be-processed image after repair is not limited in the present application, any process capable of performing the noise repair processing on the to-be-processed image according to the plurality of noise mask images to obtain the to-be-processed image after repair can be used as the process of performing the noise repair processing on the to-be-processed image according to the plurality of noise mask images to obtain the to-be-processed image after repair provided in the present application. Optionally, the to-be-processed image and each noise mask image of the plurality of noise mask images can be subjected to the masking processing respectively to obtain the to-be-processed image after masking processing corresponding to each noise mask image; then, the to-be-processed image after masking processing of each to-be-processed image can be subjected to the noise optimization processing respectively to obtain the to-be-processed image after noise optimization processing corresponding to each to-be-processed image after masking processing; and then, the noise edge blurring processing can be performed on all the to-be-processed images after noise optimization processing based on the attention mechanism to obtain the to-be-processed image after repair.

[0061] The method for image repair provided in the embodiments of the present application can obtain the to-be-processed image, perform the multi-granularity segmentation processing on the to-be-processed image based on the superpixel segmentation technology to obtain a plurality of to-be-processed images after segmentation processing, perform the noise identification processing on the to-be-processed image according to a plurality of image blocks included in each to-be-processed image after segmentation processing to obtain the result of noise identification processing, generate a plurality of noise mask images according to the result of noise identification processing and the to-be-processed image, and perform the noise repair processing on the to-be-processed image according to the plurality of noise mask images to obtain the to-be-processed image after repair, wherein, the noise identification processing and the noise repair processing are performed based on the plurality of to-be-processed images after segmentation processing obtained by the multi-granularity segmentation processing, which can improve the accuracy of image repair.

[0062] Figure 2 The flowchart of the method for image repair provided in the present application is shown in Figure Two The embodiments of the present application are based on Figure 1 The process of performing the noise identification processing on the to-be-processed image according to a plurality of image blocks included in each to-be-processed image after segmentation processing to obtain the result of noise identification processing is described in detail based on the embodiments shown in Figure 2 The process of performing the noise identification processing on the to-be-processed image according to a plurality of image blocks included in each to-be-processed image after segmentation processing to obtain the result of noise identification processing includes:

[0063] In step S201, the image block feature vector of each image block in each to-be-processed image after segmentation processing is obtained.

[0064] Specifically, an image block feature vector of each of the plurality of image blocks included in each of the segmented to-be-processed images can be acquired. The image block feature vector is a vector representing the features of the image block. The image block feature vector is not limited in the present application, and any vector representing the features of the image block can be used as the image block feature vector provided in the present application. Optionally, the formula of the image block feature vector can be as follows:

[0065]

[0066] wherein, is the image block feature vector, St R is the standard deviation of the red (R) channel, St G is the standard deviation of the green (G) channel, St B is the standard deviation of the blue (B) channel, Me R is the average value of the red (R) channel, Me G is the average value of the green (G) channel, Me B is the average value of the blue (B) channel, Max R is the maximum value of the red (R) channel, Max G is the maximum value of the green (G) channel, Max B is the maximum value of the blue (B) channel, List c is the color feature extraction vector, List b is the gray scale feature extraction vector.

[0067] In step S202, the to-be-processed image is subjected to noise recognition processing according to the noise recognition algorithm and the image block feature vector of each of the plurality of image blocks included in each of the segmented to-be-processed images, to obtain a noise recognition processing result.

[0068] Specifically, the to-be-processed image can be subjected to noise recognition processing according to the noise recognition algorithm and the image block feature vector of each of the plurality of image blocks included in each of the segmented to-be-processed images obtained in step S201, to obtain a noise recognition processing result. The noise recognition processing result includes the image block noise recognition parameter of each of the plurality of image blocks included in each of the segmented to-be-processed images, wherein the image block noise recognition parameter represents whether the image block contains noise.

[0069] In the present application, the process of performing noise recognition processing on the to-be-processed image according to the noise recognition algorithm and the image block feature vector of each image block included in the to-be-processed image after each segmentation processing, to obtain the result of the noise recognition processing, is not limited, and any process of performing noise recognition processing on the to-be-processed image according to the noise recognition algorithm and the image block feature vector of each image block included in the to-be-processed image after each segmentation processing, to obtain the result of the noise recognition processing, can be used as the process of performing noise recognition processing on the to-be-processed image according to the noise recognition algorithm and the image block feature vector of each image block included in the to-be-processed image after each segmentation processing, to obtain the result of the noise recognition processing. Optionally, the image block feature vector of each image block included in the to-be-processed image after each segmentation processing can be embedded to obtain an image block feature embedding vector of each image block included in the to-be-processed image after each segmentation processing; and then the to-be-processed image can be processed according to the noise recognition algorithm and the image block feature embedding vector of each image block included in the to-be-processed image after each segmentation processing, to obtain the result of the noise recognition processing.

[0070] In the present application, the noise recognition algorithm is not limited, and any noise recognition algorithm that can perform noise recognition processing on the to-be-processed image according to the noise recognition algorithm and the image block feature vector of each image block included in the to-be-processed image after each segmentation processing, to obtain the result of the noise recognition processing, can be used as the noise recognition algorithm provided in the present application. Optionally, the formula of the noise recognition algorithm provided in the embodiments of the present application is as follows:

[0071]

[0072] In the present application, the noise recognition function S represents the noise recognition function, S m is an image block noise recognition parameter. Optionally, the image block noise recognition parameter can be 1 or 0. Specifically, if the image block noise recognition parameter is 1, it indicates that the image block contains noise, and if the image block noise recognition parameter is 0, it indicates that the image block does not contain noise.

[0073] The method provided in the embodiments of the present application comprises the following steps: obtaining an image block feature vector of each image block included in each segmented image; performing noise recognition processing on the image to be processed according to a noise recognition algorithm and the image block feature vector of each image block included in each segmented image included in the image to be processed, to obtain a noise recognition processing result; and generating a plurality of noise mask images according to the noise recognition processing result and the image to be processed.

[0074] Figure 3 The method for image restoration provided in the embodiments of the present application Figure Three The embodiments of the present application Figure 1 Or Figure 2 The embodiments of the present application Figure 3 The embodiments of the present application

[0075] Step S301: performing noise mask image generation processing on the image to be processed according to the image block noise recognition parameter of each image block included in each segmented image included in the noise recognition processing result, to obtain a plurality of initial noise mask images.

[0076] Specifically, according to the image block noise recognition parameter of each image block in the plurality of image blocks included in each segmented processed to-be-processed image included in the result of the noise recognition processing, noise mask image generation processing can be performed on the to-be-processed image to obtain a plurality of initial noise mask images. Wherein, the process of performing noise mask image generation processing on the to-be-processed image according to the image block noise recognition parameter of each image block in the plurality of image blocks included in each segmented processed to-be-processed image included in the result of the noise recognition processing to obtain a plurality of initial noise mask images is not limited, any process of performing noise mask image generation processing on the to-be-processed image according to the image block noise recognition parameter of each image block in the plurality of image blocks included in each segmented processed to-be-processed image included in the result of the noise recognition processing to obtain a plurality of initial noise mask images can be used as the process of performing noise mask image generation processing on the to-be-processed image according to the image block noise recognition parameter of each image block in the plurality of image blocks included in each segmented processed to-be-processed image included in the result of the noise recognition processing to obtain a plurality of initial noise mask images provided by the present application. Wherein, if the image block noise recognition parameter of each image block in the plurality of image blocks included in each segmented processed to-be-processed image included in the result of the noise recognition processing is different, the corresponding process of performing noise mask image generation processing on the to-be-processed image according to the image block noise recognition parameter of each image block in the plurality of image blocks included in each segmented processed to-be-processed image included in the result of the noise recognition processing to obtain a plurality of initial noise mask images is also different.

[0077] Optionally, if the image block noise recognition parameter of each image block in the plurality of image blocks included in each segmented processed to-be-processed image included in the result of the noise recognition processing is as shown in step S202, that is, the image block noise recognition parameter can be 1 or 0, specifically, if the image block noise recognition parameter is 1, it indicates that the image block contains noise, and if the image block noise recognition parameter is 0, it indicates that the image block does not contain noise, then the process of performing noise mask image generation processing on the to-be-processed image according to the image block noise recognition parameter of each image block in the plurality of image blocks included in each segmented processed to-be-processed image included in the result of the noise recognition processing to obtain a plurality of initial noise mask images can include: according to the image block noise recognition parameter of each image block in the plurality of image blocks included in each segmented processed to-be-processed image, setting the pixels of the image blocks in the to-be-processed image whose image block noise recognition parameter is 1 to 1, and setting the pixels of the image blocks in the to-be-processed image whose image block noise recognition parameter is 0 to 0, to obtain a plurality of initial noise mask images.

[0078] Step S302, obtaining the position feature vector of each image block in the plurality of image blocks included in each segmented processed to-be-processed image.

[0079] Specifically, a position feature vector of each image block included in each segmented to-be-processed image can be acquired. Wherein, the process of acquiring the position feature vector of each image block included in each segmented to-be-processed image is not limited in the present application, any process capable of acquiring the position feature vector of each image block included in each segmented to-be-processed image can be used as the process of acquiring the position feature vector of each image block included in each segmented to-be-processed image provided by the present application. Alternatively, a position similarity parameter between each image block included in each segmented to-be-processed image and each image block included in other segmented to-be-processed image can be calculated first. Then, the position feature vector of each image block included in each segmented to-be-processed image is generated according to the calculated position similarity parameter.

[0080] Wherein, the process of calculating the position similarity parameter between each image block included in each segmented to-be-processed image and each image block included in other segmented to-be-processed image is not limited in the present application, and the formula of calculating the position similarity parameter between each image block included in each segmented to-be-processed image and each image block included in other segmented to-be-processed image provided by the present application is as follows:

[0081]

[0082] Wherein, represents the position similarity parameter, specifically, represents the position similarity parameter between the image block a with granularity p and the image block b with granularity q, wherein the larger the position similarity parameter is, the higher the position similarity between the two image blocks is, that is, the closer the position is. represents the pixel index set of the image block with granularity p, represents the number of pixels possessed in the image block with granularity p, represents the pixel index set of the image block with granularity q, represents the exclusive or (XOR) operation.

[0083] Wherein, the application does not limit the process of generating the position feature vector of each image block in the plurality of image blocks included in each segmented processed image to be processed according to the calculated position similarity parameter, and optionally, the process of generating the position feature vector of each image block in the plurality of image blocks included in each segmented processed image to be processed according to the calculated position similarity parameter provided by the application is as follows:

[0084]

[0085] Wherein, represents the position feature vector, and specifically represents the position feature vector of the image block a with granularity p.

[0086] Step S303, according to the position feature vector of each image block in the plurality of image blocks included in each segmented processed image to be processed, the plurality of initial noise mask images obtained are corrected to obtain a plurality of noise mask images.

[0087] Specifically, according to the position feature vector of each image block in the plurality of image blocks included in each segmented processed image to be processed obtained in step S302, the plurality of initial noise mask images obtained in step S301 can be corrected to obtain a plurality of noise mask images. Wherein, the application does not limit the process of generating the position feature vector of each image block in the plurality of image blocks included in each segmented processed image to be processed, and any process that can generate the position feature vector of each image block in the plurality of image blocks included in each segmented processed image to be processed can be used as the process of generating the position feature vector of each image block in the plurality of image blocks included in each segmented processed image to be processed provided by the application. Optionally, the image blocks in the same region range in each segmented processed image to be processed can be determined according to the position feature vector of each image block in the plurality of image blocks included in each segmented processed image to be processed. Then, according to the image block noise identification parameter of the image blocks in the same region range in each segmented processed image to be processed, the plurality of initial noise mask images are corrected to obtain a plurality of noise mask images.

[0088] Wherein, the application does not limit the process of determining the image blocks in the same area range in each of the segmented processed to-be-processed images according to the position feature vector of each of the image blocks in the plurality of image blocks included in each of the segmented processed to-be-processed images. Optionally, if the position feature vector of each of the image blocks in the plurality of image blocks included in each of the segmented processed to-be-processed images is as shown in step S302, the image blocks in the same area range in each of the segmented processed to-be-processed images can be determined according to the description of the position similarity parameter in step S302.

[0089] Wherein, the application does not limit the process of correcting the plurality of initial noise mask images according to the image block noise identification parameter of the image blocks in the same area range in each of the segmented processed to-be-processed images. Optionally, whether the image block noise identification parameter of the image block with small granularity is different from the image block noise identification parameter of the image block with large granularity in the image block noise identification parameter of the image blocks in the same area range in each of the segmented processed to-be-processed images, wherein, if the image block noise identification parameter of the image block with small granularity is 1, but the image block noise identification parameter of the image block with large granularity is 0, the image block with large granularity in the initial noise mask image can be corrected, that is, the pixels of the image block with large granularity in the initial noise mask image are corrected to 1. If the image block noise identification parameter of the image block with small granularity is the same as the image block noise identification parameter of the image block with large granularity, no correction is performed.

[0090] The application embodiment provides a process of generating a plurality of noise mask images according to the results of noise identification processing and to-be-processed images. The process includes: performing noise mask image generation processing on the to-be-processed images according to the image block noise identification parameter of each of the image blocks in the plurality of image blocks included in each of the segmented processed to-be-processed images included in the results of noise identification processing, to obtain a plurality of initial noise mask images; obtaining the position feature vector of each of the image blocks in the plurality of image blocks included in each of the segmented processed to-be-processed images; and correcting the plurality of initial noise mask images obtained according to the position feature vector of each of the image blocks in the plurality of image blocks included in each of the segmented processed to-be-processed images, to obtain a plurality of noise mask images. First, a plurality of initial noise mask images are generated according to the results of noise processing, and then the plurality of initial noise mask images generated are repaired according to the position feature vector of each of the image blocks in the plurality of image blocks included in each of the segmented processed to-be-processed images. This can improve the accuracy of the plurality of noise mask images obtained, further improve the accuracy of noise repair processing, and thus improve the accuracy of image repair.

[0091] Figure 4 Flowchart of the method of image repair provided by the applicationFigure Four The embodiment is based on the embodiments shown in Figure 1 or Figure 2 or Figure 3 The embodiment further describes the process of performing noise recognition processing on the plurality of image blocks included in each of the segmented processed to-be-processed images to obtain the noise recognition processing result, as shown in Figure 4 The process of performing noise recognition processing on the plurality of image blocks included in each of the segmented processed to-be-processed images to obtain the noise recognition processing result, as shown in

[0092] In step S401, the position feature vector of each of the plurality of image blocks included in each of the segmented processed to-be-processed images is obtained.

[0093] Specifically, the position feature vector of each of the plurality of image blocks included in each of the segmented processed to-be-processed images is obtained. The process of obtaining the position feature vector of each of the plurality of image blocks included in each of the segmented processed to-be-processed images can refer to the description in step S302, which will not be repeated here.

[0094] The process of obtaining the position feature vector of each of the plurality of image blocks included in each of the segmented processed to-be-processed images is not limited in the present application. Alternatively, the position similarity parameter between each of the plurality of image blocks included in each of the segmented processed to-be-processed images and each of the plurality of image blocks included in other segmented processed to-be-processed images can be calculated first. Then, the position feature vector of each of the plurality of image blocks included in each of the segmented processed to-be-processed images is generated according to the calculated position similarity parameter. The description of the above process can refer to the description in step S301, which will not be repeated here.

[0095] In step S402, a plurality of to-be-recognized regions of the to-be-processed image are determined according to the position feature vector of each of the plurality of image blocks included in each of the segmented processed to-be-processed images.

[0096] Specifically, according to the position feature vector of each image block in the plurality of image blocks included in each segmented to-be-processed image obtained in step S401, a plurality of to-be-identified regions of the to-be-processed image can be determined. Wherein, at least one image block in at least two segmented to-be-processed images is included in each to-be-identified region, that is, in one to-be-identified region, a plurality of image blocks with different granularity levels are included, and the plurality of image blocks with different granularity levels included in one to-be-identified region respectively belong to at least two segmented to-be-processed images included in the to-be-identified region.

[0097] Step S403, obtaining an image block feature vector of each image block in the plurality of image blocks included in each segmented to-be-processed image, and determining a region feature vector of each to-be-identified region according to the image block feature vector of each image block in the plurality of image blocks included in each segmented to-be-processed image.

[0098] Specifically, the image block feature vector of each image block in the plurality of image blocks included in each segmented to-be-processed image can be obtained, and the region feature vector of each to-be-identified region can be determined according to the image block feature vector of each image block in the plurality of image blocks included in each segmented to-be-processed image.

[0099] Wherein, the process of obtaining the image block feature vector of each image block in the plurality of image blocks included in each segmented to-be-processed image can refer to the description in step S201, which will not be repeated here.

[0100] Wherein, the process of determining the region feature vector of each to-be-identified region according to the image block feature vector of each image block in the plurality of image blocks included in each segmented to-be-processed image is not limited in the present application, and optionally, the image block feature vector of the image block corresponding to each to-be-identified region can be first screened from the image block feature vector of each image block in the plurality of image blocks included in each segmented to-be-processed image, and then the image block feature vector of the image block corresponding to each to-be-identified region is embedded to obtain the region feature vector of each to-be-identified region.

[0101] Step S404, performing noise identification processing on the to-be-processed image according to the noise identification algorithm and the region feature vector of each to-be-identified region, to obtain a noise identification processing result.

[0102] Specifically, according to the noise recognition algorithm and the region feature vector of each to-be-recognized region determined in step S403, noise recognition processing is performed on the to-be-processed image to obtain a noise recognition processing result. The noise recognition processing result includes a region noise recognition parameter of each to-be-recognized region. The region noise recognition parameter indicates whether the to-be-recognized region contains noise.

[0103] Specifically, the process of performing noise recognition processing on the to-be-processed image according to the noise recognition algorithm and the region feature vector of each to-be-recognized region to obtain a noise recognition processing result can refer to the description of the process of performing noise recognition processing on the to-be-processed image according to the noise recognition algorithm and the image block feature vector of each image block included in the plurality of image blocks in the to-be-processed image after segmentation in step S202, which is not described herein again. Specifically, the region noise recognition parameter can refer to the description of the image block noise recognition parameter in step S202, which is not described herein again.

[0104] The process of performing noise recognition processing on the to-be-processed image according to the plurality of image blocks included in each to-be-processed image after segmentation to obtain a noise recognition processing result provided by the embodiments of the present application can improve the accuracy of the obtained noise recognition processing result and further improve the accuracy of noise repair processing, thereby improving the accuracy of image repair, by obtaining the position feature vector of each image block included in the plurality of image blocks in each to-be-processed image after segmentation, determining a plurality of to-be-recognized regions of the to-be-processed image according to the position feature vector of each image block included in the plurality of image blocks in each to-be-processed image after segmentation, obtaining an image block feature vector of each image block included in the plurality of image blocks in each to-be-processed image after segmentation, and determining a region feature vector of each to-be-recognized region according to the image block feature vector of each image block included in the plurality of image blocks in each to-be-processed image after segmentation, performing noise recognition processing on the to-be-processed image according to the noise recognition algorithm and the region feature vector of each to-be-recognized region to obtain a noise recognition processing result.

[0105] Figure 5 Flowchart of the method of image repair provided by the present application Figure Five The embodiments of the present application are based on the embodiments shown in Figure 1 or Figure 2 or Figure 3 or Figure 4 The process of generating a plurality of noise mask images according to the noise recognition processing result and the to-be-processed image in the embodiments of the present application is described in detail, as shown in Figure 5As shown, according to the result of the noise identification processing and the to-be-processed image, a process of generating a plurality of noise mask images includes:

[0106] In step S501, according to the region noise identification parameter of each to-be-identified region included in the result of the noise identification processing, the image block noise identification parameter of each image block in the plurality of image blocks included in each segmented to-be-processed image is determined.

[0107] Specifically, according to the region noise identification parameter of each to-be-identified region included in the result of the noise identification processing, the image block noise identification parameter of each image block in the plurality of image blocks included in each segmented to-be-processed image can be determined. Wherein, the process of determining the image block noise identification parameter of each image block in the plurality of image blocks included in each segmented to-be-processed image according to the region noise identification parameter of each to-be-identified region included in the result of the noise identification processing is not limited by the present application, and optionally, the image block corresponding to each to-be-identified region and the at least one image block included in each segmented to-be-processed image corresponding to the image block can be determined first, and then the image block noise identification parameter of each image block in the at least one image block included in each segmented to-be-processed image corresponding to the image block corresponding to each to-be-identified region is determined according to the region noise identification parameter of each to-be-identified region included in the result of the noise identification processing, wherein the image block noise identification parameter of each image block in the at least one image block included in each segmented to-be-processed image corresponding to the image block corresponding to each to-be-identified region is the image block noise identification parameter of each image block in the plurality of image blocks included in each segmented to-be-processed image.

[0108] In step S502, according to the region noise identification parameter of each image block in the plurality of image blocks included in each segmented to-be-processed image and the to-be-processed image, a plurality of noise mask images are generated.

[0109] Specifically, according to the region noise identification parameter of each image block in the plurality of image blocks included in each segmented to-be-processed image determined in step S501 and the to-be-processed image, a plurality of noise mask images can be generated. Wherein, the process of generating a plurality of noise mask images according to the region noise identification parameter of each image block in the plurality of image blocks included in each segmented to-be-processed image and the to-be-processed image can refer to the description of the process of performing noise mask image generation processing on the to-be-processed image according to the image block noise identification parameter of each image block in the plurality of image blocks included in each segmented to-be-processed image included in the result of the noise identification processing in step S301 to obtain a plurality of initial noise mask images, which will not be described here.

[0110] The process provided in this application embodiment for generating multiple noise mask images based on the noise recognition processing result and the image to be processed involves determining the image block noise recognition parameters of each image block in the multiple image blocks included in each segmented image to be processed based on the region noise recognition parameters of each region to be identified included in the noise recognition processing result. Multiple noise mask images are then generated based on the region noise recognition parameters of each image block in the multiple image blocks included in each segmented image to be processed and the image to be processed. The generated noise mask images are more accurate due to the region noise recognition parameters of each region to be identified included in the noise recognition processing result, further improving the accuracy of noise restoration processing and thus enhancing the accuracy of image restoration.

[0111] Figure 6 Flowchart of the image restoration method provided in this application Figure Six In this embodiment Figure 1 or Figure 2 or Figure 3 or Figure 4 or Figure 5 Based on the illustrated embodiment, the process of performing noise inpainting on the image to be processed using multiple noise mask images to obtain the repaired image to be processed is described in detail, such as... Figure 6 As shown, the process of performing noise inpainting on the image to be processed based on multiple noise mask images to obtain the inpainted image includes:

[0112] Step S601: Perform masking processing on each noise mask image in the image to be processed and the multiple noise mask images to obtain the masked image to be processed corresponding to each noise mask image.

[0113] Specifically, each noise mask image in the image to be processed and each of the multiple noise mask images can be masked separately to obtain a masked image to be processed corresponding to each noise mask image. The masked image to be processed is a color image with a mask. The image to be processed has three channels, and the noise mask images have one channel; therefore, the masked image to be processed has four channels. This application does not limit the process of masking each noise mask image in the image to be processed and each of the multiple noise mask images to obtain a masked image to be processed corresponding to each noise mask image. Any process that can mask each noise mask image in the image to be processed and each of the multiple noise mask images to obtain a masked image to be processed corresponding to each noise mask image can be used as the process provided in this application for masking each noise mask image in the image to be processed and obtaining a masked image to be processed corresponding to each noise mask image.

[0114] Step S602, respectively, on each shielding processing after the processing of the image noise optimization processing, get each shielding processing after the processing of the image corresponding to the noise optimization processing after the processing of the image.

[0115] Specifically, on each shielding processing after the processing of the image noise optimization processing, get each shielding processing after the processing of the image corresponding to the noise optimization processing after the processing of the image. Wherein, the process of the application to each shielding processing after the processing of the image noise optimization processing, get each shielding processing after the processing of the image corresponding to the noise optimization processing after the processing of the image is not limited, any process of each shielding processing after the processing of the image noise optimization processing, get each shielding processing after the processing of the image corresponding to the noise optimization processing after the processing of the image, can be as the process of the application to each shielding processing after the processing of the image noise optimization processing, get each shielding processing after the processing of the image corresponding to the noise optimization processing after the processing of the image. Optionally, each shielding processing after the processing of the image can be first processed by non-structure noise optimization processing, and each shielding processing after the processing of the image corresponding to the non-structure noise optimization processing after the processing of the image is obtained. Then, each non-structure noise optimization processing after the processing of the image is processed by structure noise optimization processing, and each shielding processing after the processing of the image corresponding to the noise optimization processing after the processing of the image is obtained. Through the two processes of non-structure noise optimization processing and structure noise optimization processing, the accuracy of each shielding processing after the processing of the image corresponding to the noise optimization processing after the processing of the image can be improved.

[0116] Wherein, the process of the application to each shielding processing after the processing of the image noise optimization processing, get each shielding processing after the processing of the image corresponding to the noise optimization processing after the processing of the image is not limited, any process of each shielding processing after the processing of the image noise optimization processing, get each shielding processing after the processing of the image corresponding to the noise optimization processing after the processing of the image, can be as the process of the application to each shielding processing after the processing of the image noise optimization processing, get each shielding processing after the processing of the image corresponding to the noise optimization processing after the processing of the image. Optionally, each shielding processing after the processing of the image can be first processed by non-structure noise optimization processing, and each shielding processing after the processing of the image corresponding to the non-structure noise optimization processing after the processing of the image is obtained. Then, each non-structure noise optimization processing after the processing of the image is processed by structure noise optimization processing, and each shielding processing after the processing of the image corresponding to the noise optimization processing after the processing of the image is obtained. Through the two processes of non-structure noise optimization processing and structure noise optimization processing, the accuracy of each shielding processing after the processing of the image corresponding to the noise optimization processing after the processing of the image can be improved.

[0117] Wherein, the application does not limit the process of performing structure noise optimization processing on each non-structure noise optimization processing image, and obtaining the noise optimization processing image corresponding to each shielding processing image. Any process that can perform structure noise optimization processing on each non-structure noise optimization processing image, and obtain the noise optimization processing image corresponding to each shielding processing image, can be used as the process of performing structure noise optimization processing on each non-structure noise optimization processing image, and obtaining the noise optimization processing image corresponding to each shielding processing image provided by the application. Optionally, each non-structure noise optimization processing image can be first down-sampled to obtain a down-sampled image corresponding to each shielding processing image, wherein the down-sampled image corresponding to each shielding processing image is a reduced shielding processing image. Then, each down-sampled image is subjected to global structure noise optimization processing and local structure noise optimization processing to obtain a global structure noise optimization processing and local structure noise optimization processing image corresponding to each shielding processing image. Then, each global structure noise optimization processing and local structure noise optimization processing image is up-sampled to obtain a noise optimization processing image corresponding to each shielding processing image. Wherein, the noise optimization processing image corresponding to each shielding processing image is an enlarged global structure noise optimization processing and local structure noise optimization processing image. Wherein, through the two processes of global structure noise optimization processing and local structure noise optimization processing, the accuracy of the noise optimization processing image corresponding to each shielding processing image obtained can be improved.

[0118] Wherein, the application does not limit the process of performing global structure noise optimization processing and local structure noise optimization processing on each down-sampled image. Optionally, two-layer fast Fourier convolution FFC and skip connection can be used to perform global structure noise optimization processing and local structure noise optimization processing on each down-sampled image. Specifically, the down-sampled image after two-layer fast Fourier convolution FFC and the down-sampled image can be subjected to skip connection processing to obtain a noise optimization processing image. Wherein, through fast Fourier convolution FFC and skip connection, the efficiency of global structure noise optimization processing and local structure noise optimization processing can be improved. Figure 7A structural schematic diagram of a network for performing noise optimization processing on a to-be-processed image after occlusion processing. The network for performing noise optimization processing on the to-be-processed image after occlusion processing includes a non-structured noise optimization processing layer and a structured noise optimization processing layer. The structured noise optimization processing layer includes a down-scaling layer, two fast Fourier convolution layers (FFC), a skip connection, and an up-scaling layer.

[0119] In this application, the structure of the fast Fourier convolution FFC is not limited. Optionally, the fast Fourier convolution FFC can include a global processing branch and a local processing branch. The global processing branch is used for global structured noise optimization processing, and the local processing branch is used for local structured noise optimization processing. Specifically, the to-be-processed image after each down-scaling processing can be divided into global image information and local image information. The global image information is subjected to fast Fourier transform (FFT) processing and 3*3 convolution processing through the global processing branch, and the local image information is subjected to 3*3 convolution processing through the local processing branch. The global image information after FFT processing and the local image information after 3*3 convolution processing are subjected to skip connection processing and convolution processing in sequence to obtain cross-fused global image information. The global image information after 3*3 convolution processing and the local image information after 3*3 convolution processing are subjected to skip connection processing and convolution processing in sequence to obtain cross-fused local image information. The cross-fused global image information and the cross-fused local image information are spliced based on a channel to obtain the to-be-processed image after down-scaling processing after one-time fast Fourier convolution processing. The to-be-processed image after down-scaling processing after one-time fast Fourier convolution processing is input into the fast Fourier convolution FFC described above again to obtain the to-be-processed image after down-scaling processing after two-time fast Fourier convolution processing. The to-be-processed image after down-scaling processing after two-time fast Fourier convolution processing and the to-be-processed image after down-scaling processing are subjected to skip connection processing to obtain the to-be-processed image after noise optimization processing.

[0120] Figure 8This is an example diagram of the structure of a Fast Fourier Transform (FFT) convolutional layer. The FFT convolutional layer includes a global processing branch and local processing branches. The global processing branch includes two global information layers (Global), a first global processing sub-branch, a second global processing sub-branch, skip connections, and a normalized rectified layer (BN-ReLU). The local processing branch includes two local information layers (local), a first local processing sub-branch, a second local processing sub-branch, skip connections, and a normalized rectified layer (BN-ReLU). The first global processing sub-branch includes two convolutional normalized rectified layers (Conv-BN-ReLU), a real Fourier transform layer (Real-FFT2d), an inverse real Fourier transform layer (Inv-Real-FFT2d), skip connections, and a 1x1 convolutional layer. The second global processing sub-branch is a 3x3 convolutional layer, and both the first and second local processing sub-branches are 3x3 convolutional layers.

[0121] Among them, Figure 8 The processing formula for the Real-FFT2d layer included in the first sub-branch of the global processing in the Fast Fourier Convolutional layer shown is as follows:

[0122]

[0123] Among them, Figure 8 The processing formula for the second convolutional normalized rectified layer (Conv-BN-ReLU) included in the first sub-branch of the global processing in the Fast Fourier Convolutional layer shown is as follows:

[0124]

[0125] Among them, Figure 8 The processing formula for the Inverse Real Fourier Transform (Inv-Real-FFT2d) layer included in the first sub-branch of the global processing in the Fast Fourier Convolutional layer shown is as follows:

[0126]

[0127] Step S603: Based on the attention mechanism, perform noise edge blurring on the obtained image to be processed after all noise optimization processing to obtain the repaired image to be processed.

[0128] Specifically, the obtained all noise-optimized to-be-processed images can be subjected to noise edge blurring processing based on an attention mechanism to obtain a to-be-processed image with repair completed. The process of subjecting the obtained all noise-optimized to-be-processed images to noise edge blurring processing based on the attention mechanism is not limited in the present application. Optionally, the processing formula of subjecting the obtained all noise-optimized to-be-processed images to noise edge blurring processing based on the attention mechanism is as follows:

[0129]

[0130] wherein, represents the i th pixel of the noise edge of the finest granularity level in the image, represents the i th pixel in the image with the k th granularity level, represents a set of pixels in the image with different granularity levels corresponding to the same pixel i. represents the 8-neighborhood of the i th pixel in the image with the k th granularity level, represents the n th pixel of the finest granularity level in the mask image. In the mask image, a pixel value of 0 indicates that reconstruction is needed, and a pixel value of 1 indicates that reconstruction is not needed. N' k represents a set of all granularity levels. In the merging of the pixels in the blurred edge, After that, the corresponding pixel in the mask image is set to 1. The above formula will be repeatedly calculated until the value of all pixels in the mask image is 1.

[0131] The process of performing noise repair processing on a to-be-processed image according to multiple noise mask images to obtain a to-be-processed image with repair completed provided by the embodiments of the present application includes: performing masking processing on the to-be-processed image and each noise mask image of the multiple noise mask images respectively to obtain a to-be-processed image after masking processing corresponding to each noise mask image; performing noise optimization processing on each to-be-processed image after masking processing respectively to obtain a to-be-processed image after noise optimization processing corresponding to each to-be-processed image after masking processing; and performing noise edge blurring processing on all to-be-processed images after noise optimization processing based on an attention mechanism to obtain a to-be-processed image with repair completed. Through generating multiple to-be-processed images after noise optimization processing and performing noise edge blurring processing on all to-be-processed images after noise optimization processing based on the attention mechanism to obtain a to-be-processed image with repair completed, the accuracy of image repair can be improved.

[0132] In a possible embodiment, after obtaining the to-be-processed image with repair completed, the loss function value can be determined according to the to-be-processed image and the to-be-processed image with repair completed. Then, the algorithm parameters of each algorithm in the image repair processing process can be optimized according to the loss function value.

[0133] In the present application, the process of determining the loss function value according to the to-be-processed image and the to-be-processed image after completion of the repair is not limited, and the formula for determining the loss function value according to the to-be-processed image and the to-be-processed image after completion of the repair is as follows:

[0134]

[0135] In the present application, the process of determining the loss function value according to the to-be-processed image and the to-be-processed image after completion of the repair is not limited, and the formula for determining the loss function value according to the to-be-processed image and the to-be-processed image after completion of the repair is as follows: The loss function value is represented by L, and α and β represent the balance factor, and α+β=1. The image division loss function value is represented by L1. The image noise optimization loss function value is represented by L2.

[0136] Specifically, in the process of optimizing the algorithm parameters of each algorithm in the image repair processing according to the loss function value, the algorithm parameters of each algorithm in the image repair processing can be optimized to reduce the loss function.

[0137] In one possible embodiment, the image repair method provided by the present embodiment can be based on a multi-view local reconstruction network (MLRN). Specifically, the multi-view local reconstruction network described above can be obtained after network supervised training based on the training set and the loss function value described above.

[0138] In one possible embodiment, determining the loss function value according to the to-be-processed image and the to-be-processed image after completion of the repair can include: first determining the image division loss function value and the image noise optimization loss function value according to the to-be-processed image and the to-be-processed image after completion of the repair. Then, the loss function value is determined according to the image division loss function value and the image noise optimization loss function value.

[0139] In the present application, the process of determining the loss function value according to the to-be-processed image and the to-be-processed image after completion of the repair is not limited, and the formula for determining the loss function value according to the to-be-processed image and the to-be-processed image after completion of the repair is as follows:

[0140]

[0141] In the present application, the process of determining the loss function value according to the to-be-processed image and the to-be-processed image after completion of the repair is not limited, and the formula for determining the loss function value according to the to-be-processed image and the to-be-processed image after completion of the repair is as follows: The image division loss function value is represented by L1, and α1 and α2 are weight coefficients. The gradient loss function is represented by L3. The clustering loss function value is represented by L4.

[0142] In the present application, the process of determining the loss function value according to the to-be-processed image and the to-be-processed image after completion of the repair is not limited, and the formula for determining the loss function value according to the to-be-processed image and the to-be-processed image after completion of the repair is as follows:

[0143]

[0144] where, denotes the image noise optimization loss function value. β1, β2, β3 are weight coefficients. denotes the local loss function value, denotes the global loss function value. We also introduce a gradient penalty loss R1.

[0145] Specifically, the loss function value can be determined based on the above-described formula for determining the loss function value, and the determined image partition loss function value and image noise optimization loss function value.

[0146] In one possible embodiment, determining the image partition loss function value and the image noise optimization loss function value according to the to-be-processed image and the to-be-processed image after completion of the repair can include: first determining a gradient loss function value, a clustering loss function value, a global loss function value, and a local loss function value according to the to-be-processed image and the to-be-processed image after completion of the repair. Then, determining the image partition loss function value according to the gradient loss function value and the clustering loss function value. Then, determining the image noise optimization loss function value according to the global loss function value and the local loss function value.

[0147] wherein the formula for determining the gradient loss function value according to the to-be-processed image and the to-be-processed image after completion of the repair is as follows:

[0148]

[0149] wherein, corresponding to the mean square error (MSE) loss of the color feature. respectively represent the reconstructed R, G, B values of pixel n, represent the true R, G, B values of pixel n. measure the MSE loss of the structure feature. and respectively represent the reconstructed and true structure features of pixel n.

[0150] wherein the formula for determining the clustering loss function value according to the to-be-processed image and the to-be-processed image after completion of the repair is as follows:

[0151]

[0152] wherein ξ represents a regularization term. and respectively represent the reference probability distribution and the true probability distribution

[0153] pixel i is clustered to center k. denotes the KL divergence between the two probability distributions. Clustering is an unsupervised model, as shown in the formula. This method can penalize those pixels that deviate too far, thereby enhancing the purity of clustering.

[0154] wherein the formula for determining the local loss function value according to the to-be-processed image and the to-be-processed image after completion of inpainting is as follows:

[0155]

[0156] wherein, is the local loss function value, and β 11 , β 12 , β 13 is a weight coefficient, is the generation loss, is the discrimination loss, is the feature matching loss, and ψ() is a PatchGAN discriminator, wherein the PatchGAN discriminator is used to distinguish between “real” samples and “fake” samples, for example, the mask area is marked as “fake”, and otherwise is marked as “real”, wherein the feature matching loss is used to measure the mean absolute error (MAE) loss between the discriminator features extracted from real and fake samples, such a loss function is commonly used in the training of GAN networks and has been proven to have a positive impact on image generation. X and denote the real image and the predicted image. M is a Boolean mask matrix, in which 1 and 0 represent the mask and non-mask areas, respectively. ⊙ represents the multiplication between elements.

[0157] wherein the formula for determining the global loss function value according to the to-be-processed image and the to-be-processed image after completion of inpainting is as follows:

[0158]

[0159] wherein Φ hrf represents a pre-trained ResNet50 for segmentation with extended convolution, which can be achieved by using Fourier or extended convolution.

[0160] Specifically, the image division loss function value can be determined based on the formula for determining the image division loss function value described above, and the gradient loss function value and the clustering loss function value determined. Specifically, the image noise optimization loss function value can be determined based on the formula for determining the image noise optimization loss function value described above, and the global loss function value and the local loss function value determined.

[0161] Figure 9 The structure diagram of the image inpainting device provided in the present application is shown in FIG. 1, and the image inpainting device 90 provided in the present embodiment includes: Figure 9

[0162] ​The segmentation module 901 is configured to acquire a to-be-processed image, and perform multi-granularity segmentation processing on the to-be-processed image based on a superpixel segmentation technology to obtain a plurality of to-be-processed images after segmentation processing. Each to-be-processed image after segmentation processing indicates a different granularity. Each to-be-processed image after segmentation processing includes a plurality of image blocks. The granularity of each image block included in each to-be-processed image after segmentation processing corresponds to the granularity indicated by the corresponding to-be-processed image after segmentation processing.

[0163] The identification module 902 is configured to perform noise identification processing on the to-be-processed image according to the plurality of image blocks included in each to-be-processed image after segmentation processing to obtain a result of noise identification processing, and generate a plurality of noise mask images according to the result of noise identification processing and the to-be-processed image. Each noise mask image indicates a different granularity. The granularity indicated by the noise mask image corresponds to the granularity indicated by the to-be-processed image after segmentation processing.

[0164] The repair module 903 is configured to perform noise repair processing on the to-be-processed image according to the plurality of noise mask images to obtain a to-be-processed image after repair.

[0165] The device for image repair provided in this embodiment can execute the method provided in the method embodiment, and has similar implementation principles and technical effects, which will not be described here in detail.

[0166] Figure 10 A structural schematic diagram of an electronic device provided in this application is shown in FIG. 1. As shown in FIG. 1, the electronic device 100 provided in this embodiment includes at least one processor 1001 and a memory 1002. Optionally, the device 100 further includes a communication component 1003. The processor 1001, the memory 1002 and the communication component 1003 are connected through a bus 1004. Figure 10 In the specific implementation process, the at least one processor 1001 executes the computer execution instructions stored in the memory 1002, so that the at least one processor 1001 executes the method described above.

[0167] The specific implementation process of the processor 1001 can refer to the method embodiment described above, which has similar implementation principles and technical effects, and will not be described here in detail.

[0168]

[0169] ​In the above embodiments, it should be understood that the processor can be a central processing unit (CPU) and can also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), etc. The general-purpose processor can be a microprocessor or can also be any conventional processor. The steps of the method disclosed in combination with the application can be directly embodied as hardware processor execution or combined with hardware and software modules in the processor for execution.

[0170] The memory can include a random access memory (RAM) and can also include a non-volatile memory (NVM), such as at least one disk memory.

[0171] The bus can be an industry standard architecture (ISA) bus, a peripheral component (PCI) bus, an extended industry standard architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, the bus in the drawings of the present application does not limit to only one bus or one type of bus.

[0172] The present application also provides a computer program product comprising a computer program which, when executed by a processor, implements the above method.

[0173] The present application also provides a computer readable storage medium having computer execution instructions stored therein, when the processor executes the computer execution instructions, the above method is implemented.

[0174] The above readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk. The readable storage medium can be any available medium that can be accessed by a general-purpose or special-purpose computer.

[0175] An example readable storage medium is coupled to the processor such that the processor can read information from the readable storage medium and can write information to the readable storage medium. Of course, the readable storage medium can also be a part of the processor. The processor and the readable storage medium can be located in an application specific integrated circuit (ASIC). Of course, the processor and the readable storage medium can also exist as discrete components in the device.

[0176] The division of units is only a logical functional division, and in actual implementation, there can be another division manner, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the units shown or discussed can be indirect coupling or communication connection through some interfaces, devices or units, and can be electrical, mechanical or other forms.

[0177] The units described as separate components can or can not be physically separated, and the components shown as units can or can not be physical units, that is, can be located in one place, or can be distributed on a plurality of network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiment scheme.

[0178] In addition, the functional units in each embodiment of the present application can be integrated in one processing unit, or each unit can be physically present separately, or two or more units can be integrated in one unit.

[0179] If the functions are realized in the form of software function units and sold or used as independent products, they can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application or the part of the present application that essentially contributes to the prior art or the part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the embodiments of the present application. The foregoing storage medium includes a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various program code storage media.

[0180] Those skilled in the art can understand that all or part of the steps of the above-mentioned method embodiments can be completed by program instruction related hardware. The foregoing program can be stored in a computer readable storage medium. The program executes to perform the steps of the above-mentioned method embodiments; and the foregoing storage medium includes various media capable of storing program codes, such as ROM, RAM, magnetic disk, or optical disk.

[0181] Finally, it should be noted that other embodiments of the present application will readily occur to those skilled in the art upon consideration of the specification and practice of the present application disclosed herein. The present application is intended to include all such variations and modifications as fall within the scope of the present application, which is defined by the following claims, as well as the full scope of equivalents to which such claims are entitled. It is intended, therefore, that the present application be considered as including all possibilities falling within the scope of the application and their equivalents.

Claims

1. A method of image inpainting, characterized by, The method comprises the following steps: obtaining a to-be-processed image, and performing multi-granularity segmentation processing on the to-be-processed image based on a superpixel segmentation technology to obtain a plurality of segmented to-be-processed images; wherein each segmented to-be-processed image indicates a different granularity; each segmented to-be-processed image comprises a plurality of image blocks; wherein the granularity of each image block in each segmented to-be-processed image corresponds to the granularity indicated by the corresponding segmented to-be-processed image; performing noise identification processing on the to-be-processed image according to the plurality of image blocks included in each segmented to-be-processed image to obtain a noise identification processing result, and generating a plurality of noise mask images according to the noise identification processing result and the to-be-processed image; wherein each noise mask image indicates a different granularity; the granularity indicated by the noise mask image corresponds to the granularity indicated by the segmented to-be-processed image; performing masking processing on the to-be-processed image and each noise mask image in the plurality of noise mask images respectively to obtain a to-be-processed image after masking processing corresponding to each noise mask image; performing noise optimization processing on each to-be-processed image after masking processing respectively to obtain a to-be-processed image after noise optimization processing corresponding to each to-be-processed image after masking processing; performing noise edge blurring processing on all the to-be-processed images after noise optimization processing based on an attention mechanism to obtain a repaired to-be-processed image.

2. The method of claim 1, wherein, According to the plurality of image blocks included in each segmented to-be-processed image, performing noise identification processing on the to-be-processed image to obtain a noise identification processing result, comprising: obtaining an image block feature vector of each image block in the plurality of image blocks included in each segmented to-be-processed image; performing noise identification processing on the to-be-processed image according to a noise identification algorithm and the image block feature vector of each image block in the plurality of image blocks included in each segmented to-be-processed image to obtain a noise identification processing result; wherein the noise identification processing result comprises an image block noise identification parameter of each image block in the plurality of image blocks included in each segmented to-be-processed image; wherein the image block noise identification parameter represents whether the image block contains noise.

3. The method of claim 2, wherein, According to the noise identification processing result and the to-be-processed image, generating a plurality of noise mask images, comprising: performing noise mask image generation processing on the to-be-processed image according to the image block noise identification parameter of each image block in the plurality of image blocks included in each segmented to-be-processed image included in the noise identification processing result to obtain a plurality of initial noise mask images; obtaining a position feature vector of each image block in the plurality of image blocks included in each segmented to-be-processed image; According to the position feature vector of each image block in the plurality of image blocks included in each of the plurality of segmented processed images to be processed, the plurality of initial noise mask images are modified to obtain the plurality of noise mask images.

4. The method of claim 3, wherein, The position feature vector of each image block in the plurality of image blocks included in each of the plurality of segmented processed images to be processed is obtained, including: The position similarity parameter between each image block in the plurality of image blocks included in each of the plurality of segmented processed images to be processed and each image block in the plurality of image blocks included in other segmented processed images to be processed is calculated. According to the calculated position similarity parameter, the position feature vector of each image block in the plurality of image blocks included in each of the plurality of segmented processed images to be processed is generated.

5. The method of claim 3, wherein, According to the position feature vector of each image block in the plurality of image blocks included in each of the plurality of segmented processed images to be processed, the plurality of initial noise mask images are modified to obtain the plurality of noise mask images, including: According to the position feature vector of each image block in the plurality of image blocks included in each of the plurality of segmented processed images to be processed, the image blocks in the same area range in each of the segmented processed images to be processed are determined. According to the image block noise identification parameter of the image blocks in the same area range in each of the segmented processed images to be processed, the plurality of initial noise mask images are modified to obtain the plurality of noise mask images.

6. The method of claim 1, wherein, According to the plurality of image blocks included in each of the plurality of segmented processed images to be processed, noise identification processing is performed on the image to be processed to obtain a noise identification processing result, including: The position feature vector of each image block in the plurality of image blocks included in each of the plurality of segmented processed images to be processed is obtained. According to the position feature vector of each image block in the plurality of image blocks included in each of the plurality of segmented processed images to be processed, a plurality of to-be-identified areas of the image to be processed are determined; wherein the to-be-identified area includes at least one image block in at least two of the plurality of segmented processed images to be processed. The image block feature vector of each image block in the plurality of image blocks included in each of the plurality of segmented processed images to be processed is obtained, and the region feature vector of each of the to-be-identified areas is determined according to the image block feature vector of each image block in the plurality of image blocks included in each of the plurality of segmented processed images to be processed. According to the noise identification algorithm and the region feature vector of each of the to-be-identified areas, noise identification processing is performed on the image to be processed to obtain a noise identification processing result; wherein the noise identification processing result includes a region noise identification parameter of each to-be-identified area; wherein the region noise identification parameter represents whether the to-be-identified area contains noise.

7. The method of claim 6, wherein, The position feature vector of each image block in the plurality of image blocks included in each of the plurality of segmented processed images to be processed is obtained, including: calculating a position similarity parameter between each image block in the plurality of image blocks included in each of the segmented processed to-be-processed images and each image block in the plurality of image blocks included in other segmented processed to-be-processed images; generating a position feature vector of each image block in the plurality of image blocks included in each of the segmented processed to-be-processed images according to the calculated position similarity parameter.

8. The method of claim 6, wherein, According to the results of the noise identification processing and the to-be-processed image, a plurality of noise mask images are generated, including: According to the region noise identification parameter of each to-be-identified region included in the results of the noise identification processing, an image block noise identification parameter of each image block in the plurality of image blocks included in each of the segmented processed to-be-processed images is determined. According to the image block noise identification parameter of each image block in the plurality of image blocks included in each of the segmented processed to-be-processed images and the to-be-processed image, a plurality of noise mask images are generated.

9. The method according to any one of claims 1 to 8, characterized in that, For each of the masked processed to-be-processed images, noise optimization processing is performed to obtain a noise optimized processed to-be-processed image corresponding to each of the masked processed to-be-processed images, including: For each of the masked processed to-be-processed images, non-structural noise optimization processing is performed to obtain a non-structural noise optimized processed to-be-processed image corresponding to each of the masked processed to-be-processed images. For each of the non-structural noise optimized processed to-be-processed images, structural noise optimization processing is performed to obtain a noise optimized processed to-be-processed image corresponding to each of the masked processed to-be-processed images.

10. The method of claim 9, wherein, For each of the non-structural noise optimized processed to-be-processed images, structural noise optimization processing is performed to obtain a noise optimized processed to-be-processed image corresponding to each of the masked processed to-be-processed images, including: For each of the non-structural noise optimized processed to-be-processed images, downsampling processing is performed to obtain a downsampling processed to-be-processed image corresponding to each of the masked processed to-be-processed images. For each of the downsampling processed to-be-processed images, global structural noise optimization processing and local structural noise optimization processing are performed to obtain a global structural noise optimized and local structural noise optimized processed to-be-processed image corresponding to each of the masked processed to-be-processed images. For each of the global structural noise optimized and local structural noise optimized processed to-be-processed images, upsampling processing is performed to obtain a noise optimized processed to-be-processed image corresponding to each of the masked processed to-be-processed images.

11. The method of claim 10, wherein, The method further includes: determining a loss function value according to the to-be-processed image and the repaired to-be-processed image; optimizing algorithm parameters of algorithms in the image repair processing according to the loss function value.

12. The method of claim 11, wherein, Determining a loss function value according to the to-be-processed image and the repaired to-be-processed image includes: determining an image division loss function value and an image noise optimization loss function value according to the to-be-processed image and the repaired to-be-processed image; determining the loss function value according to the image division loss function value and the image noise optimization loss function value.

13. The method of claim 12, wherein, According to the to-be-processed image and the repaired to-be-processed image, a gradient loss function value, a clustering loss function value, a global loss function value, and a local loss function value are determined. According to the to-be-processed image and the repaired to-be-processed image, a gradient loss function value, a clustering loss function value, a global loss function value, and a local loss function value are determined. According to the gradient loss function value and the clustering loss function value, the image division loss function value is determined. According to the global loss function value and the local loss function value, the image noise optimization loss function value is determined.

14. An apparatus for image inpainting, the apparatus comprising: The segmentation module is configured to: acquire a to-be-processed image, and perform multi-granularity segmentation processing on the to-be-processed image based on a superpixel segmentation technology to obtain a plurality of segmented to-be-processed images; each segmented to-be-processed image indicates a different granularity; each segmented to-be-processed image includes a plurality of image blocks; the granularity of each image block included in each segmented to-be-processed image corresponds to the granularity indicated by the corresponding segmented to-be-processed image; The identification module is configured to: perform noise identification processing on the to-be-processed image according to the plurality of image blocks included in each segmented to-be-processed image to obtain a noise identification processing result, and generate a plurality of noise mask images according to the noise identification processing result and the to-be-processed image; each noise mask image indicates a different granularity; the granularity indicated by a noise mask image corresponds to the granularity indicated by the segmented to-be-processed image; The repair module is configured to: perform masking processing on the to-be-processed image and each noise mask image of the plurality of noise mask images respectively to obtain a to-be-processed image after masking processing corresponding to each noise mask image; perform noise optimization processing on each to-be-processed image after masking processing respectively to obtain a to-be-processed image after noise optimization processing corresponding to each to-be-processed image after masking processing; and perform noise edge blurring processing on all to-be-processed images after noise optimization processing obtained based on an attention mechanism to obtain a repaired to-be-processed image. The memory stores computer execution instructions; and the processor executes the computer execution instructions stored in the memory, so that the processor executes the method in any one of claims 1-13.

15. An electronic device, comprising: The computer readable storage medium stores computer execution instructions, and the computer execution instructions are executed by the processor to implement the method in any one of claims 1-13.

17. A computer program product, comprising a computer program, which, when executed by a processor, implements the method in any one of claims 1-13. ​ 16. A computer-readable storage medium, characterized in that, ​ ​

Images