An image inpainting method, device, computer equipment, medium and program

Abstract

Embodiments of the present application disclose an image repairing method, device, computer equipment, medium and program; in the embodiments of the present application, a to-be-repaired image can be acquired, the to-be-repaired image contains a target object; repairing information identification is performed on the to-be-repaired image to extract first image features corresponding to the target object in the to-be-repaired image; based on the first image features, repairing features of the target object are determined; the repairing features and the first image features are fused to obtain fused repairing features; based on the fused repairing features, the target object in the to-be-repaired image is repaired to obtain a repaired image. The problem that the image repairing effect is not ideal is solved, and the effect of simultaneously improving the high definition and credibility of the to-be-repaired image is realized.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of image processing, in particular to an image repairing method and device, computer equipment, medium and program. BACKGROUND

[0002] At present, the image repairing technology mainly uses various prior knowledge methods to complete image repairing. For example, the prior knowledge methods commonly used in face image repairing include the following: 1. geometric prior, such as face key points, face segmentation map or face component heat map, but this method cannot provide sufficient detailed information to restore high-definition details of the face. 2. reference prior, that is, using a fixed reference high-definition face image, or constructing a high-definition dictionary according to the high-definition face image, but this method needs to have a high-definition face image as a reference, which limits the application scene. 3. face component dictionary-based reference prior, but this method needs to construct a high-definition face component prior in advance, and the limited dictionary limits the diversity and richness of the model results. Therefore, the existing image repairing technology is not ideal, and cannot improve the high-definition and credibility of the image at the same time. SUMMARY

[0003] The embodiments of the present application provide an image repairing method, device, computer equipment, medium and program, which can improve the effect of image repairing.

[0004] An image repairing method comprises:

[0005] acquiring a to-be-repaired image, wherein the to-be-repaired image contains a target object;

[0006] performing repairing information identification on the to-be-repaired image to extract a first image feature corresponding to the target object in the to-be-repaired image;

[0007] determining a repairing feature of the target object based on the first image feature;

[0008] performing fusion processing on the repairing feature and the first image feature to obtain a fused repairing feature;

[0009] performing repairing on the target object in the to-be-repaired image based on the fused repairing feature to obtain a repaired image.

[0010] Correspondingly, the embodiments of the present application provide an image repairing device, which comprises:

[0011] an acquisition unit configured to acquire a to-be-repaired image, wherein the to-be-repaired image contains a target object;

[0012] an extraction unit configured to perform repairing information identification on the to-be-repaired image to extract a first image feature corresponding to the target object in the to-be-repaired image;

[0013] determining, based on the first image feature, a repair feature of the target object;

[0014] fusing, by a fusion unit, the repair feature and the first image feature to obtain a fused repair feature;

[0015] repairing, by a repair unit, the target object in the image to be repaired based on the fused repair feature to obtain a repaired image.

[0016] Optionally, in some embodiments, the image repair apparatus can further include a fusion unit, and the construction unit can be specifically configured to perform up-sampling convolution processing on the repair feature to obtain multi-scale repair features, perform down-sampling processing on the first image feature to obtain multi-scale image features, and perform fusion processing on the multi-scale repair features and the multi-scale image features to obtain the fused repair feature at the multi-scale.

[0017] Optionally, in some embodiments, the image repair apparatus can further include a fusion unit, and the construction unit can be specifically configured to perform displacement convolution processing on the repair feature and the first image feature to determine a displacement feature of a convolution sampling position, and perform deformable convolution processing on the repair feature based on the displacement feature to obtain the fused repair feature.

[0018] Optionally, in some embodiments, the image repair apparatus can further include a decoding unit, and the construction unit can be specifically configured to perform decoding processing on the fused repair feature to repair the target object in the image to be repaired to obtain a repaired image.

[0019] Optionally, in some embodiments, the image repair apparatus can further include an up-sampling unit, and the construction unit can be specifically configured to obtain an up-sampling scale, and perform up-sampling convolution processing on the fused repair feature based on the up-sampling scale.

[0020] Optionally, in some embodiments, the image repair apparatus can further include an extraction unit, and the construction unit can be specifically configured to perform convolution operation on the image to be repaired to extract at least one of a texture, an outline, and a position of the target object in the image to be repaired, and determine a first image feature corresponding to the target object based on the at least one of the texture, the outline, and the position.

[0021] Optionally, in some embodiments, the image repairing apparatus can further comprise a determining unit, and the constructing unit can be specifically configured to perform down-sampling convolution processing on the first image feature to obtain a second image feature of the target object in the image to be repaired; and determine the repairing feature of the target object based on the second image feature.

[0022] Optionally, in some embodiments, the image repairing apparatus can further comprise a determining unit, and the constructing unit can be specifically configured to obtain a reference repairing feature matched with the target object in the image to be repaired; and determine the repairing feature of the target object based on the second image feature and the reference repairing feature.

[0023] Optionally, in some embodiments, the image repairing apparatus can further comprise a correcting unit, and the constructing unit can be specifically configured to calculate a similarity between the reference repairing feature and the second image feature; and correct the second image feature based on the reference repairing feature satisfying a preset condition to obtain the repairing feature of the target object.

[0024] Optionally, in some embodiments, the image repairing apparatus can further comprise a replacing unit, and the constructing unit can be specifically configured to replace the second image feature with the reference repairing feature having the highest similarity to obtain the repairing feature of the target object.

[0025] Optionally, in some embodiments, the image repairing apparatus can further comprise a determining unit, and the constructing unit can be specifically configured to calculate a feature distance between the reference repairing feature and the second image feature; and determine that a similarity between the reference repairing feature and the second image feature satisfies a preset condition if the feature distance is less than a preset threshold.

[0026] In addition, an electronic device is provided in an embodiment of the present application, which comprises a processor and a memory, the memory stores an application program, and the processor is configured to run the application program in the memory to implement the image repairing method provided in the embodiment of the present application.

[0027] In addition, a computer readable storage medium is provided in an embodiment of the present application, which stores a plurality of instructions, and the instructions are adapted to be loaded by a processor to execute the steps in any one of the image repairing methods provided in the embodiment of the present application.

[0028] In addition, a computer program product is provided in an embodiment of the present application, which comprises a computer program, and the computer program is executed by a processor to implement the image repairing method provided in any one of the embodiments of the present application.

[0029] In the embodiment of the present application, a computer device acquires a to-be-repaired image, the to-be-repaired image containing a target object; performs repair information identification on the to-be-repaired image to extract a first image feature corresponding to the target object in the to-be-repaired image; determines a repair feature of the target object based on the first image feature; performs fusion processing on the repair feature and the first image feature to obtain a fused repair feature; and performs repair on the target object in the to-be-repaired image based on the fused repair feature to obtain a repaired image. In this way, the repair feature determined based on the first image feature of the target object in the defective to-be-repaired image can improve the high-definition detail effect of image restoration, and the fusion processing on the repair feature and the first image feature improves the credibility of image restoration, thereby achieving the effect of simultaneously improving the high definition and credibility of the to-be-repaired image. BRIEF DESCRIPTION OF DRAWINGS

[0030] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative effort.

[0031] Figure 1 is a flowchart of an image repair method provided by the embodiment of the present application;

[0032] Figure 2 is another flowchart of an image repair method provided by the embodiment of the present application;

[0033] Figure 3 is a schematic diagram of an image repair processing module in an image repair device provided by the embodiment of the present application;

[0034] Figure 4 is a schematic diagram of a vector quantization module in an image repair device provided by the embodiment of the present application;

[0035] Figure 5 is a schematic diagram of a texture deformation module in an image repair device provided by the embodiment of the present application;

[0036] Figure 6 is a structural schematic diagram of an image repair device provided by the embodiment of the present application;

[0037] Figure 7 is a structural schematic diagram of an electronic device provided by the embodiment of the present application. DETAILED DESCRIPTION

[0038] With reference to the accompanying drawings, the technical solutions in the embodiments of the present application will be described clearly and completely. Obviously, the described embodiments are only a part of the embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work fall within the scope of the present application.

[0039] The embodiments of the present application provide an image repairing method and device, computer equipment, a medium and a program. The image repairing device can be integrated in the computer equipment, which can be an electronic device, such as a server or a terminal. The medium is a computer readable storage medium, and the program is a computer program product or a computer program.

[0040] The server can be a stand-alone physical server, a server cluster or a distributed system composed of multiple physical servers, a cloud server providing cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery network (CDN), and big data and artificial intelligence platforms, etc. The terminal can be a smart phone, a tablet computer, a notebook computer, a desktop computer, a smart speaker, a smart watch, etc. The terminal and the server can be directly or indirectly connected through wired or wireless communication, which is not limited in the present application.

[0041] For example, referring to Figure 1 For example, the electronic device can obtain an image to be repaired, which contains a target object. The electronic device can identify the repair information of the image to be repaired to extract the first image feature corresponding to the target object in the image to be repaired. Based on the first image feature, the electronic device can determine the repair feature of the target object. The electronic device can fuse the repair feature and the first image feature to obtain the fused repair feature. Based on the fused repair feature, the electronic device can repair the target object in the image to be repaired to obtain a repaired image.

[0042] The following will be described in detail. It should be noted that the order of the following embodiments is not limited as the preferred order of the embodiments.

[0043] The present embodiment will be described from the perspective of an image repairing device, which can be integrated in an electronic device, such as a server or a terminal. The terminal can include a tablet computer, a notebook computer, a personal computer (PC), a wearable device, a virtual reality device, or other smart devices that can obtain data.

[0044] An image repairing method comprises:

[0045] As Figure 1 shown, the specific process of the image repairing method is as follows:

[0046] 101. Obtain a to-be-repaired image, the to-be-repaired image containing a target object.

[0047] The to-be-repaired image can be a blurred low-quality image, and the to-be-repaired image can have a case of missing details or missing part components in the image. The target object can be one or more of a human face, an animal, and a plant.

[0048] The execution subject of the image repairing method can be an electronic device or a server. The to-be-repaired image can be obtained from the execution subject of the image repairing method or from a third-party electronic device or server independent of the execution subject. For example, the electronic device obtains a to-be-repaired image stored locally or obtains a to-be-repaired image from a server or obtains a to-be-repaired image from a third-party electronic device, or the server obtains a to-be-repaired image sent by the electronic device.

[0049] 102. Perform repair information identification on the to-be-repaired image to extract a first image feature corresponding to the target object in the to-be-repaired image.

[0050] The repair information can include at least one of the texture, contour, and position of the target object. The first image feature is an image feature related to the repair information. The first image feature can be multi-dimensional and can include multiple repair information. The first image feature can also be a set of image features composed of multiple image features related to the repair information. For example, taking a human face as the target object, the repair information can include at least one of the texture of the face, the contour of the face shape, the contour of the hairstyle, and the position of each facial component. The first image feature can include the extracted repair information. Different to-be-repaired images can not have the same repair information, and thus the first image features of different to-be-repaired images can not be the same. For example, some to-be-repaired images have severe texture loss and cannot identify the texture, so the first image feature can not include the texture feature.

[0051] Correspondingly, performing repair information identification on the to-be-repaired image to extract a first image feature corresponding to the target object in the to-be-repaired image can include: performing a convolution operation on the to-be-repaired image to extract at least one of the texture, contour, and position of the target object; and determining the first image feature corresponding to the target object based on the at least one of the texture, contour, and position.

[0052] In an embodiment, taking a texture as an example, the texture is one of the features inherent to an image, is a pattern generated by a certain form of spatial transformation of gray scale, sometimes with a certain periodicity, the pixel gray scale distribution of a texture region has a certain form, and a histogram is a powerful tool for describing the pixel gray scale distribution in an image, and thus the histogram can be used to describe the texture. For example, the first image feature of the target object in the image to be repaired can be extracted by using a preset image feature extraction algorithm, the image to be repaired is divided into small connected regions, then the gradient or edge direction histogram of each pixel point in the connected regions is collected, and finally these histograms are combined to form the texture feature of the image, where the preset image feature extraction algorithm can be a Histogram of Oriented Gradient (HOG) feature extraction algorithm.

[0053] For example, the image to be repaired is subjected to a convolution operation by using the Histogram of Oriented Gradient feature extraction algorithm, so as to extract at least one of the texture, contour and position of the target object in the image to be repaired, and the feature set composed of the at least one of the texture, contour and position of the target object in the image to be repaired is taken as the first image feature corresponding to the target object.

[0054] 103. Based on the first image feature, the repair feature of the target object is determined.

[0055] The repair feature is in a corresponding relationship with the first image feature, the repair feature can be a repair feature set composed of multiple and multi-dimensional repair features, and the repair feature can be used to correct the first image feature, so as to repair the image to be repaired. For example, if the first image feature is a contour feature of a target object with low quality, the repair feature can be a contour feature with high quality corresponding to the contour feature. By determining the corresponding repair feature based on the first image feature, the first image feature of the target object in the image to be repaired can be accurately corrected, so as to repair the image to be repaired.

[0056] In an embodiment, based on the first image feature, the repair feature of the target object can include:

[0057] S1. The first image feature is subjected to a down-sampling convolution process to obtain a second image feature of the target object in the image to be repaired.

[0058] The second image feature can be a multi-dimensional image feature set, and the down-sampling convolution processing can be hierarchical convolution processing, for example, a down-sampling scale can be obtained first, where the scale can refer to a spatial scale, such as a spatial scale of the first image feature in the image to be repaired. The first image feature is subjected to hierarchical down-sampling convolution processing according to the down-sampling scale, so that the scale of the first image feature is reduced, and a second image feature of the target object in the image to be repaired at a minimum scale is obtained. The second image feature can be referred to as a hidden feature, and the down-sampling convolution processing can be implemented by a hierarchical convolutional neural network.

[0059] It can be understood that when the image is processed, the electronic device cannot know the spatial scale of the target object in the image in advance, and therefore, the description of the image at multiple scales needs to be considered to determine the spatial scale of the target object. Therefore, the image can be constructed as a series of image sets at different spatial scales, and the image features are detected in different spatial scales. Generally, the larger the spatial scale of the image feature, the smaller the pixel area corresponding to each feature in the image feature, that is, the more information contained in the image feature, and the smaller the spatial scale of the image feature, the larger the pixel area corresponding to each feature in the image feature.

[0060] For example, if the original spatial scale information of the image to be repaired is 512x512 pixels, the down-sampling scale can be obtained by dividing the original scale information by 2, and after one down-sampling convolution processing of the first image feature, an image feature with a spatial scale of 256x256 pixels can be obtained. After 4 times of down-sampling convolution processing of the first image feature by the hierarchical convolutional neural network, a second image feature with a spatial scale of 16x16 pixels can be obtained.

[0061] S2, determining a repair feature of the target object based on the second image feature.

[0062] In this embodiment, the first image feature is subjected to hierarchical convolution processing to obtain the second image feature, and the repair feature of the target object is determined based on the second image feature, rather than directly using the first image feature to determine the repair feature. This is because the pixel area corresponding to each image feature in the first image feature is too small, and it can be a case that one image feature is one pixel. Such a small repair feature cannot be accurately matched, resulting in inaccurate repair features. Using the second image feature at the minimum scale to determine the repair feature can more easily determine the repair feature of the target object matched with the second image feature.

[0063] In an embodiment, determining the repair feature of the target object based on the second image feature can include:

[0064] (1) obtaining a reference repair feature matched with the target object in the image to be repaired.

[0065] The reference repair feature can be obtained from a vector quantization dictionary corresponding to the target object, the vector quantization dictionary including a large number of reference repair features corresponding to the target object and a feature matching algorithm, the reference repair features being extracted from a large number of high-quality image samples corresponding to real target objects and including real image detail information. The reference repair feature can be matched with the second image feature, so as to repair the target object in the image to be repaired.

[0066] In an embodiment, the electronic device can obtain the vector quantization dictionary corresponding to the target object from a local device or from a server, and obtain the reference repair feature matching the target object in the image to be repaired from the vector quantization dictionary.

[0067] (2) determining the repair feature of the target object based on the second image feature and the reference repair feature.

[0068] The repair feature is a feature in the reference repair feature that matches the second image feature.

[0069] For example, for each feature vector at each spatial position in the second image feature, the matching degree between the feature vector in the reference repair feature and the feature vector at each spatial position in the second image feature is calculated to determine the matching degree between the reference repair feature and the second image feature, and the reference repair feature that best matches the second image feature is taken as the repair feature of the target object.

[0070] Optionally, in an embodiment, the matching degree can be a similarity between features, and the higher the similarity, the higher the matching degree. Determining the repair feature of the target object based on the second image feature and the reference repair feature can include:

[0071] calculating the similarity between the reference repair feature and the second image feature; and based on the reference repair feature whose similarity satisfies a preset condition, correcting the second image feature to obtain the repair feature of the target object.

[0072] The similarity between the reference repair feature and the second image feature can be calculated by calculating a feature distance between the reference repair feature and a feature vector in the second image feature, wherein the feature distance can be an Euclidean distance, and determining the similarity between the reference repair feature and the feature vector in the second image feature according to the feature distance. In this embodiment, a mapping relationship between the feature distance and the similarity between the reference repair feature and the feature vector in the second image feature can be set in advance, so that the similarity between the reference repair feature and the feature vector in the second image feature can be quickly determined according to the mapping relationship after the feature distance between the reference repair feature and the feature vector in the second image feature is calculated.

[0073] In an embodiment, the second image feature is modified based on the reference repair feature satisfying the preset condition to obtain the repair feature of the target object, and the modification of the second image feature based on the reference repair feature satisfying the preset condition can include: calculating a feature distance between the reference repair feature and the second image feature; and determining that the similarity between the reference repair feature and the second image feature satisfies the preset condition if the feature distance is less than a preset threshold.

[0074] The preset threshold can be set in advance, and the preset threshold can be a fixed value or a threshold value calculated according to all feature distances by percentage. For example, if 100 feature distances are calculated between 100 reference repair features in the reference repair feature and an image feature A in the second image feature, the 100 feature distances can be sorted in ascending order, and the feature distances in the first 1% can be selected as the preset threshold. If 1000 feature distances are calculated between 1000 reference repair features in the reference repair feature and an image feature A in the second image feature, the 1000 feature distances can be sorted in ascending order, and the feature distances in the first 0.1% can be selected as the preset threshold. If the calculated preset threshold is 3, the reference repair feature with a feature distance less than 3 is considered to satisfy the preset condition.

[0075] In an embodiment, the reference repair feature with a feature distance less than the preset threshold can be multiple, and the second image feature is modified based on the reference repair feature satisfying the preset condition to obtain the repair feature of the target object, and the modification of the second image feature based on the reference repair feature satisfying the preset condition can include: modifying the second image feature based on multiple reference repair features with a feature distance less than the preset threshold after feature fusion of the multiple reference repair features, to obtain the repair feature of the target object, or selecting a reference repair feature with the smallest feature distance from the multiple reference repair features with a feature distance less than the preset threshold to modify the second image feature to obtain the repair feature of the target object.

[0076] In an embodiment, the reference repair features with feature distances less than the preset threshold can be multiple, and the reference repair feature with the highest similarity can be used to correct the second image feature to obtain the repair feature of the target object.

[0077] In an embodiment, the reference repair feature with the highest similarity can be used to correct the second image feature to obtain the repair feature of the target object, which can include: calculating the feature distance between the reference repair feature and the second image feature; and if the feature distance is the smallest, determining that the similarity between the reference repair feature and the second image feature meets the preset condition.

[0078] For example, the feature distances between the reference repair features A, B, C and D and the second image feature X are calculated as 2, 4, 3 and 1 respectively, and the feature distance between the reference repair feature D and the second image feature X is the smallest, so it is determined that the similarity between the reference repair feature D and the second image feature X meets the preset condition, and the second image feature X is corrected based on the reference repair feature D to obtain the repair feature of the target object.

[0079] In an embodiment, the reference repair feature with the highest similarity can be used to correct the second image feature to obtain the repair feature of the target object, which can include: replacing the second image feature with the reference repair feature with the highest similarity to obtain the repair feature of the target object.

[0080] For example, the feature distances between the reference repair features A, B, C and D and the second image feature X are calculated, and if the feature distance between the reference repair feature D and the second image feature X is the smallest, it is determined that the similarity between the reference repair feature D and the second image feature X is the highest, and the reference repair feature D is replaced with the second image feature X as the repair feature.

[0081] 104、fuse the repair feature and the first image feature to obtain a fused repair feature.

[0082] The fusion processing can be feature deformation processing, such as deforming the repair feature to the information corresponding to the first image feature to obtain the fused repair feature.

[0083] In the embodiment, the repair feature has high-quality detail features corresponding to the target object, but the repair feature is obtained by similarity matching of the reference repair feature, so the result contains the detail features of the target object, but has a certain degree of untrustworthiness, and the first image feature is directly extracted from the image to be repaired, so it has trustworthiness, therefore, the repair feature and the first image feature are fused to improve the quality and trustworthiness of the repaired image corresponding to the image to be repaired.

[0084] Optionally, the repair feature can include a multi-scale repair feature, and the first image feature can include a multi-scale image feature. The fusion repair feature obtained by fusing the repair feature and the first image feature can include:

[0085] S1, performing up-sampling convolution processing on the repair feature to obtain a multi-scale repair feature.

[0086] It should be noted that the first image feature and the second image feature are both single spatial scales, and the repair feature obtained based on the first image feature or the second image feature is also a single spatial scale feature, so the fusion repair feature obtained based on the single spatial scale repair feature and the single spatial scale first image feature cannot accurately repair the target object in the image to be repaired. Therefore, in order to more accurately determine the fusion repair feature, the repair feature and the first image feature can be scaled at all scales to obtain multi-scale repair features and multi-scale first image features.

[0087] For example, a preset sampling scale is obtained, and the repair feature is up-sampled and convolved according to the sampling scale to obtain a multi-scale repair feature. For example, if the preset sampling scale is the square of 2, the scale of the repair feature is 16x16 pixels, and the original scale of the image to be repaired is 512x512 pixels, then the repair feature is up-sampled and convolved once according to the preset sampling scale to obtain a repair feature with a scale of 32x32 pixels, and the repair feature is up-sampled and convolved multiple times to obtain repair features with scales of 16x16, 32x32, 64x64, 128x128, 256x256 and 512x512 pixels.

[0088] S2, performing down-sampling processing on the first image feature to obtain a multi-scale image feature.

[0089] It should be noted that in order to retain the features related to the image to be repaired in the first image feature, when the first image feature is scaled at each scale, no convolution processing is performed, and only simple difference calculation is performed on the first image feature to obtain the first image feature at all scales.

[0090] For example, a preset sampling scale is obtained, and the first image feature is down-sampled and processed according to the sampling scale, to obtain a multi-scale image feature. For example, if the preset sampling scale is 2, the scale of the first image feature is 512x512 pixels, and the original scale of the image to be repaired is 512x512 pixels, then the first image feature is processed once by down-sampling and convolution according to the preset sampling scale, to obtain the first image feature with a scale of 256x256 pixels, and the first image feature is processed multiple times by down-sampling and convolution, to obtain image features with scales of 16x16, 32x32, 64x64, 128x128, 256x256 and 512x512 pixels.

[0091] S3, the multi-scale repair features and the multi-scale image features are fused to obtain fused repair features at multiple scales.

[0092] For example, the repair features with scales of 16x16, 32x32, 64x64, 128x128, 256x256 and 512x512 pixels are fused with the first image features with scales of 16x16, 32x32, 64x64, 128x128, 256x256 and 512x512 pixels respectively, to obtain fused repair features with scales of 16x16, 32x32, 64x64, 128x128, 256x256 and 512x512 pixels.

[0093] Optionally, in an embodiment, the fusion of the repair features and the first image features to obtain the fused repair features can further include:

[0094] S4, displacement convolution is performed on the repair features and the first image features to determine displacement features of convolution sampling positions.

[0095] For example, for the repair features and the first image features at a scale of 16x16, if the dimension of the repair features is 256 and the dimension of the first image features is 256, the repair features and the first image features are connected in parallel in the dimension channel to obtain repair features and first image features with a dimension of 256 and a spatial scale of 16x16, and the repair features and the first image features are processed by displacement convolution to generate displacement features corresponding to the repair features and the first image features.

[0096] The repair features and the first image features can both be multi-scale, and the repair features and the first image features at multiple scales are processed by displacement convolution respectively, to generate multi-scale displacement features.

[0097] S5, the repair features are processed by deformable convolution according to the displacement features, to obtain fused repair features.

[0098] For example, the displacement feature and the repair feature are input into deformable convolution to obtain a fused repair feature. The displacement feature and the repair feature can be multi-scale displacement features and repair features, and the multi-scale displacement features and repair features are input into deformable convolution to obtain multi-scale fused repair features.

[0099] 105、based on the fused repair feature, repairing the target object in the image to be repaired to obtain a repaired image.

[0100] In an embodiment, based on the fused repair feature, repairing the target object in the image to be repaired to obtain a repaired image can include: decoding the fused repair feature to repair the target object in the image to be repaired to obtain a repaired image.

[0101] The fused repair feature can be a multi-scale fused repair feature, and decoding the fused repair feature includes: obtaining an up-sampling scale; and based on the up-sampling scale, performing up-sampling convolution processing on the fused repair feature.

[0102] In an embodiment, if the scales of the multi-scale fused repair feature are 16x16, 32x32, 64x64, 128x128, 256x256, and 512x512 pixels, and the scale of the image to be repaired is 512x512 pixels, then based on the up-sampling scale, performing up-sampling convolution processing on the fused repair feature can include: performing up-sampling convolution processing on the fused repair feature of each scale in ascending order, for example, if the up-sampling scale is 2, performing up-sampling convolution processing once on the fused repair feature of 16x16 pixels can obtain a fused repair feature of 32x32 scale, and performing up-sampling convolution processing again on the fused repair feature of 32x32 scale can obtain a fused repair feature of 64x64 scale, and the above steps are sequentially executed to finally obtain a fused repair feature of 512x512, and the fused repair feature of this scale is the repaired target object, and the repaired image can be determined according to the repaired target object.

[0103] In the technical scheme provided in the embodiment, the electronic device acquires a to-be-repaired image, the to-be-repaired image containing a target object; performs repair information identification on the to-be-repaired image to extract a first image feature corresponding to the target object in the to-be-repaired image; determines a repair feature of the target object based on the first image feature; performs fusion processing on the repair feature and the first image feature to obtain a fused repair feature; and performs repair on the target object in the to-be-repaired image based on the fused repair feature to obtain a repaired image. In this way, the repair feature determined based on the first image feature of the target object in the defective to-be-repaired image can improve the high-definition detail effect of image restoration, and the fusion processing on the repair feature and the first image feature improves the credibility of image restoration, thereby achieving the effect of simultaneously improving the high definition and credibility of the to-be-repaired image.

[0104] According to the method described in the above embodiment, the following will be further described by way of example.

[0105] In the embodiment, the image repair device is specifically integrated in an electronic device, such as Figure 2 As shown in the figure, an image repair method, the specific process is as follows:

[0106] 201, the electronic device acquires a to-be-repaired image, the to-be-repaired image containing a target object.

[0107] Among them, the to-be-repaired image can be a blurred low-quality image, low quality can refer to low definition or the to-be-repaired image has defects, the to-be-repaired image can exist the case of detail loss or part of the component loss in the image, in this embodiment, the target object takes the face as an example, the to-be-repaired image can be a low-quality face image.

[0108] For example, the electronic device can acquire the to-be-repaired image from the local storage or from the server or from the third-party electronic device.

[0109] 202, the electronic device performs repair information identification on the to-be-repaired image to extract a first image feature corresponding to the target object in the to-be-repaired image.

[0110] Among them, the repair information includes at least one of the texture, contour and position of the target object, the first image feature is the image feature related to the repair information, which can include a variety of repair information, the first image feature can be multi-dimensional, or a set of image features composed of a plurality of image features related to repair information. For example, taking the face as the target object, the repair information can include at least one of the face texture, face contour, hairstyle contour and position of each facial feature component, the first image feature can include the extracted repair information, and the repair information identified by different to-be-repaired images is not necessarily the same, so the first image feature of different to-be-repaired images is not necessarily the same.

[0111] For example, the electronic device can extract a first image feature corresponding to the target object in the image to be inpainted through an image feature extraction algorithm. The image feature extraction algorithm can be a Histogram of Oriented Gradient (HOG) feature extraction algorithm, which is not limited herein.

[0112] 203. The electronic device performs down-sampling convolution processing on the first image feature to obtain a second image feature of the target object in the image to be inpainted.

[0113] The first image feature is an image feature of an original scale of the image to be inpainted, and the second image feature is an image feature of a low scale obtained through convolution processing of the first image feature. The second image feature can be an image feature set composed of multi-dimensional image features. The down-sampling convolution processing can be hierarchical convolution processing. For example, a down-sampling scale can be obtained first, where the scale can refer to a spatial scale, such as a spatial scale of the first image feature in the image to be inpainted. The first image feature is subjected to hierarchical down-sampling convolution processing according to the down-sampling scale, so as to reduce the scale of the first image feature and obtain a second image feature of the target object in the image to be inpainted at a minimum scale. The second image feature can be referred to as a hidden feature, and the down-sampling convolution processing can be implemented through a hierarchical convolutional neural network.

[0114] It can be understood that when performing related processing on an image, the electronic device cannot know the spatial scale of the target object in the image in advance. Therefore, the description of the image at multiple scales needs to be considered to determine the spatial scale of the target object. Therefore, the image can be constructed as a series of image sets at different spatial scales, and image features are detected in different spatial scales. Generally, the larger the spatial scale of an image feature, the smaller the pixel area corresponding to each feature in the image feature, that is, the more information contained in the image feature. The smaller the spatial scale of an image feature, the larger the pixel area corresponding to each feature in the image feature.

[0115] For example, if the original spatial scale information of the image to be inpainted is 512x512 pixels, the down-sampling scale can be the original scale information divided by 2 raised to the power of 1. After one down-sampling convolution processing of the first image feature, an image feature with a spatial scale of 256x256 pixels can be obtained. After four down-sampling convolution processing of the first image feature through the hierarchical convolutional neural network, a second image feature with a spatial scale of 16x16 pixels can be obtained. The second image feature can be referred to as a hidden feature corresponding to the target object in the image to be inpainted.

[0116] 204. The electronic device obtains a reference inpainting feature matched with the target object in the image to be inpainted.

[0117] The reference repair feature can be obtained from a vector quantization dictionary corresponding to the target object, the vector quantization dictionary including a large number of reference repair features corresponding to the target object and a feature matching algorithm. For example, the reference repair feature is extracted from a large number of real face high-quality image samples, and includes real face detail information. The reference repair feature can be matched with the second image feature, so as to repair the target object in the image to be repaired. In this embodiment, the reference repair feature can be a high-quality face image feature.

[0118] For example, the electronic device can obtain the vector quantization dictionary corresponding to the target object from a local device or from a server, and obtain the reference repair feature matched with the target object in the image to be repaired from the vector quantization dictionary.

[0119] 205. The electronic device determines the repair feature of the target object based on the second image feature and the reference repair feature.

[0120] The repair feature is a feature in the reference repair feature matched with the second image feature. In this embodiment, the reference repair feature can be a high-quality face image feature corresponding to the target object, and the repair feature can be a high-quality face image feature most similar to the second image feature extracted from the target object.

[0121] It should be noted that the smaller the feature of the face image is, the more difficult it is to accurately match the feature. Therefore, in order to more accurately match the feature of the face, the reference repair feature in the vector quantization dictionary can be set to a low-scale reference repair feature, and one feature in the reference repair feature can represent the feature of a small area of the face. Therefore, in this embodiment, the first image feature of the original scale corresponding to the image to be repaired is processed by hierarchical convolution to obtain the low-scale second image feature, and the low-scale second image feature is used to match the repair feature of the target object, so that the reference repair feature in the dictionary can be more easily matched, and the accuracy of the repair feature matching is improved.

[0122] Optionally, in an embodiment, the similarity between the reference repair feature and the second image feature can be calculated; and the second image feature is replaced by the reference repair feature satisfying a preset condition, to obtain the repair feature of the target object.

[0123] The similarity between the reference repair feature and the second image feature can be calculated by calculating a feature distance between the reference repair feature and a feature vector in the second image feature, wherein the feature distance can be an Euclidean distance, and determining the similarity between the reference repair feature and the feature vector in the second image feature according to the feature distance. In this embodiment, the smaller the feature distance between the reference repair feature and the feature vector in the second image feature, the higher the similarity between the reference repair feature and the feature vector in the second image feature. In this embodiment, a mapping relationship between the feature distance and the similarity between the reference repair feature and the feature vector in the second image feature can be set in advance, so that the similarity between the reference repair feature and the feature vector in the second image feature can be quickly determined according to the mapping relationship after the feature distance between the reference repair feature and the feature vector in the second image feature is calculated.

[0124] In an embodiment, the electronic device replaces the second image feature with the reference repair feature to obtain the repair feature of the target object based on the similarity between the reference repair feature and the second image feature satisfying a preset condition, and the previous operation can include: calculating a feature distance between the reference repair feature and the second image feature; and determining that the similarity between the reference repair feature and the second image feature satisfies the preset condition if the feature distance is less than a preset threshold.

[0125] The preset threshold can be set in advance, and the preset threshold can be a fixed value or a threshold value calculated according to all feature distances in a percentage, for example, if 100 feature distances are calculated between 100 reference repair features in the reference repair feature and an image feature A in the second image feature, the 100 feature distances can be sorted in ascending order, and the feature distances in the first 1% can be selected as the preset threshold, and if 1000 feature distances are calculated between 1000 reference repair features in the reference repair feature and an image feature A in the second image feature, the 1000 feature distances can be sorted in ascending order, and the feature distances in the first 0.1% can be selected as the preset threshold. If the calculated preset threshold is 3, the reference repair feature with a feature distance less than 3 is considered to satisfy the preset condition.

[0126] In an embodiment, the electronic device replaces the second image feature with the reference repair feature to obtain the repair feature of the target object based on the similarity between the reference repair feature and the second image feature satisfying a preset condition, and the previous operation can include: calculating a feature distance between the reference repair feature and the second image feature; and determining that the similarity between the reference repair feature and the second image feature satisfies the preset condition if the feature distance is less than a preset threshold.

[0127] For example, the electronic device calculates feature distances between the reference repair features A, B, C, and D and the second image feature X as 2, 4, 3, and 1 respectively, the feature distance between the reference repair feature D and the second image feature X is the smallest, it is determined that the similarity between the reference repair feature and the second image feature X meets the preset condition, the second image feature X is replaced based on the reference repair feature D, and the repair feature of the target object is obtained.

[0128] 206、The electronic device performs up-sampling convolution processing on the repair feature to obtain a multi-scale repair feature.

[0129] The up-sampling convolution processing can be processed by a repair feature decoding module in the electronic device, the repair feature can be a high-quality face image feature matched with the second image feature corresponding to the face in the image to be repaired in the vector quantization dictionary, and the multi-scale repair feature is a scaling of the repair feature in all scales corresponding to the image to be repaired.

[0130] It should be noted that since the first image feature and the second image feature are in a single spatial scale, and the repair feature obtained based on the first image feature or the second image feature in a single spatial scale is also a feature in a single spatial scale, the fusion repair feature based on the repair feature in a single spatial scale and the first image feature in a single spatial scale cannot accurately repair the target object in the image to be repaired. Therefore, in order to more accurately determine the fusion repair feature, the repair feature and the first image feature can be scaled in all scales to obtain multi-scale repair features and multi-scale first image features.

[0131] For example, the electronic device obtains a preset sampling scale, and performs up-sampling convolution processing on the repair feature according to the sampling scale to obtain a multi-scale repair feature. For example, if the preset sampling scale is the square of 2, the scale of the repair feature is 16x16 pixels, and the original scale of the image to be repaired is 512x512 pixels, then one up-sampling convolution processing is performed on the repair feature according to the preset sampling scale, and the repair feature in 32x32 pixel scale can be obtained. Multiple up-sampling convolution processing can be performed on the repair feature to obtain repair features with scales of 16x16, 32x32, 64x64, 128x128, 256x256, and 512x512 pixels.

[0132] 207、The electronic device performs down-sampling processing on the first image feature to obtain a multi-scale image feature.

[0133] It should be noted that, in order to reserve the available facial information related to the image to be repaired in the first image feature, when the first image feature is scaled at each scale, no convolution processing is performed, and only simple difference calculation is performed on the first image feature to scale the first image feature at all scales.

[0134] For example, a preset sampling scale is obtained, and the first image feature is down-sampled and difference-processed according to the sampling scale to obtain a multi-scale image feature. For example, if the preset sampling scale is 2 raised to the power of 1, the scale of the first image feature is 512x512 pixels, and the original scale of the image to be repaired is 512x512 pixels, then one time of down-sampling and difference-processing of the first image feature according to the preset sampling scale can obtain the first image feature at a scale of 256x256 pixels, and multiple times of down-sampling and convolution processing of the first image feature can obtain image features at scales of 16x16, 32x32, 64x64, 128x128, 256x256, and 512x512 pixels.

[0135] 208、The electronic device fuses the multi-scale repair feature and the multi-scale image feature to obtain a fused repair feature at a multi-scale.

[0136] The fusion processing can be feature deformation processing, such as deforming the multi-scale repair feature to the facial information corresponding to the multi-scale first image feature, so as to obtain the fused repair feature at a multi-scale.

[0137] In this embodiment, the repair feature has high-quality detailed features corresponding to the target object, but since the repair feature is obtained by similarity matching of the reference repair feature, the result contains detailed features of the target object, but has a certain degree of untrustworthiness, and the first image feature is directly extracted from the image to be repaired, and thus has trustworthiness. Therefore, the repair feature and the first image feature are fused to improve the quality and trustworthiness of the repaired image corresponding to the image to be repaired.

[0138] For example, the repair features at scales of 16x16, 32x32, 64x64, 128x128, 256x256, and 512x512 pixels are fused with the first image features at scales of 16x16, 32x32, 64x64, 128x128, 256x256, and 512x512 pixels, respectively, to obtain fused repair features at scales of 16x16, 32x32, 64x64, 128x128, 256x256, and 512x512 pixels.

[0139] Optionally, in an embodiment, the multi-scale repair feature and the multi-scale image feature are fused to obtain a multi-scale fused repair feature, and the method can further include:

[0140] S1, performing displacement convolution processing on the multi-scale repair feature and the multi-scale first image feature to determine a multi-scale displacement feature of the convolution sampling position.

[0141] For example, for a repair feature and a first image feature at a scale of 16x16, if the dimension of the repair feature is 256 and the dimension of the first image feature is 256, the repair feature and the first image feature are connected in parallel in the dimension channel to obtain a repair feature and a first image feature with a dimension of 256 and a spatial scale of 16x16. The repair feature and the first image feature are processed by displacement convolution to generate a displacement feature corresponding to the repair feature and the first image feature. Correspondingly, the above steps are repeated, and the repair feature and the first image feature at the multi-scale are processed by displacement convolution, and the multi-scale displacement feature can be generated.

[0142] S2, performing deformable convolution processing on the multi-scale repair feature according to the multi-scale displacement feature to obtain a multi-scale fused repair feature.

[0143] For example, the multi-scale displacement feature and the repair feature are input into the deformable convolution to obtain a multi-scale fused repair feature.

[0144] 209, the electronic device decodes the fused repair feature to repair the target object in the to-be-repaired image to obtain a repaired image.

[0145] The fused repair feature can be a multi-scale fused repair feature, and the decoding of the fused repair feature includes: obtaining an up-sampling scale; and performing up-sampling convolution processing on the fused repair feature based on the up-sampling scale.

[0146] In an embodiment, if the scales of the multi-scale fusion repair features are 16x16, 32x32, 64x64, 128x128, 256x256 and 512x512 pixels, and the scale of the image to be repaired is 512x512 pixels, the upsampling convolution processing of the fusion repair features based on the upsampling scale can include: performing the upsampling convolution processing on the fusion repair features of each scale in the order from small to large, for example, if the upsampling scale is 2, the fusion repair features of the scale of 16x16 pixels are processed once by upsampling convolution, and the fusion repair features of the scale of 32x32 pixels can be obtained, the fusion repair features of the scale of 32x32 pixels are processed again by upsampling convolution, and the fusion repair features of the scale of 64x64 pixels can be obtained, and the above steps are sequentially executed, and finally the fusion repair features of the scale of 512x512 pixels are obtained, and the fusion repair features of the scale are the target object after repair, and the image after repair can be determined according to the target object after repair.

[0147] In the technical scheme provided in the embodiment, the electronic device obtains an image to be repaired, and the image to be repaired contains a target object; performs repair information identification on the image to be repaired to extract first image features corresponding to the target object in the image to be repaired; determines repair features of the target object based on the first image features; performs fusion processing on the repair features and the first image features to obtain fusion repair features; and performs repair on the target object in the image to be repaired based on the fusion repair features to obtain an image after repair. In this way, the repair features determined based on the first image features of the target object in the defective image to be repaired can improve the high-definition detail effect of image restoration, and the fusion processing on the repair features and the first image features improves the credibility of image restoration, thereby achieving the effect of simultaneously improving the high-definition and credibility of the image to be repaired.

[0148] According to the method described in the above embodiment, the following examples will be further described in detail.

[0149] In the embodiment, the image repair device is specifically integrated in an electronic device, such as Figure 3 、 Figure 4 and Figure 5 as shown, Figure 3 the image repair processing module in the image repair device provided in the embodiment of the application, Figure 4 the schematic diagram of the vector quantization module in the image repair device provided in the embodiment of the application, Figure 5 the schematic diagram of the texture deformation module in the image repair device provided in the embodiment of the application; wherein the image repair device can be provided with a vector quantization module and a parallel decoder, the vector quantization module can include a vector quantization dictionary and a feature matching algorithm, and the parallel decoder can include a texture branch decoder and a main decoder, and the texture branch decoder and the main decoder adopt the texture deformation module.

[0150] In the embodiment, the vector quantization dictionary contains a reference repair feature corresponding to a target object in the image to be repaired, which can be a high-quality face image feature corresponding to the target object. The image processing device acquires an image to be repaired, which includes the target object, which can be a face. The image to be repaired is subjected to repair information recognition to extract a first image feature corresponding to the target object, which can be a low-quality face feature. The first image feature is subjected to down-sampling convolution processing to obtain a second image feature of the target object, which can be a hidden feature corresponding to the face image. Then, the feature matching algorithm is used to calculate the similarity between the reference repair feature and the second image feature corresponding to the target object. Based on the reference repair feature satisfying the preset condition, the second image feature is corrected to obtain the repair feature of the target object.

[0151] The texture branch decoder can provide a multi-scale high-quality face image feature corresponding to the target object. The texture branch decoder can be used to perform up-sampling convolution processing on the repair feature to obtain a multi-scale repair feature, and down-sampling processing on the first image feature to obtain a multi-scale image feature. At this time, the multi-scale repair feature provides a high-quality real face image feature, and the multi-scale image feature provides low-quality face information.

[0152] The texture deformation module can be used to fuse the multi-scale repair feature and the multi-scale image feature to obtain a fused repair feature under multi-scale. That is, the texture deformation module can deform the high-quality real face image feature to low-quality face information to obtain a deformed texture feature. The fused repair feature obtained in this way balances the texture reality and credibility of the face repair result.

[0153] The main decoder can be used to decode the fused repair feature under multi-scale to repair the target object in the image to be repaired to obtain a repaired image.

[0154] Specifically, the reference Figure 4 , Figure 4A schematic diagram of a vector quantization module in an image repairing apparatus provided by an embodiment of the present application is provided, wherein the vector quantization module can include a vector quantization dictionary and a feature matching algorithm, the vector quantization dictionary includes reference repairing features corresponding to target objects in the image to be repaired, and the feature matching algorithm is configured to calculate a feature distance between a second image feature and a reference repairing feature, and if the feature distance is less than a preset threshold and / or the feature distance is the minimum, it is determined that the similarity between the reference repairing feature and the second image feature meets a preset condition, and the reference repairing feature meeting the preset condition is replaced by the second image feature to obtain a repairing feature of the target object. In this way, the high-quality face image feature with the minimum feature distance to the low-quality face image feature is replaced by the low-quality face image feature, so that the repairing feature closest to the face image in the image to be repaired can be obtained.

[0155] Specifically, referring to Figure 5 , Figure 5 A schematic diagram of a texture deformation module in an image repairing apparatus provided by an embodiment of the present application is provided, wherein the texture deformation module can include a displacement convolution processing algorithm and a deformable convolution processing algorithm, the displacement convolution processing algorithm can perform displacement convolution processing on the repairing feature and the first image feature to determine a displacement feature of the convolution sampling position, and the deformable convolution processing algorithm can perform deformable convolution processing on the repairing feature based on the displacement feature to obtain a fused repairing feature.

[0156] In this embodiment, an image to be repaired is obtained, the image to be repaired includes a target object; repairing information of the image to be repaired is identified to extract a first image feature corresponding to the target object in the image to be repaired; based on the first image feature, a repairing feature of the target object is determined; the repairing feature and the first image feature are fused to obtain a fused repairing feature; and the target object in the image to be repaired is repaired based on the fused repairing feature to obtain a repaired image. In this way, the repairing feature determined based on the first image feature of the target object in the defective image to be repaired can improve the high-definition detail effect of image restoration, and the fusion processing of the repairing feature and the first image feature improves the credibility of image restoration, thereby achieving the effect of improving the high-definition and credibility of the image to be repaired at the same time.

[0157] In order to better implement the above method, an image repairing apparatus is further provided by an embodiment of the present application, which can be integrated in an electronic device, such as a server or a terminal device, and the terminal device can include a tablet computer, a smart television, a mobile phone, a notebook computer, a personal computer, and / or the like.

[0158] For example, as Figure 6 shown, the image repairing apparatus can include an acquisition unit 301, an extraction unit 302, a determination unit 303, a fusion unit 304, and a repairing unit 305, as follows:

[0159] The acquisition unit 301 is configured to acquire a to-be-repaired image, and the to-be-repaired image contains a target object.

[0160] The to-be-repaired image can be a blurred low-quality image, and the to-be-repaired image can be in a case of missing details or missing part components in the image. In this embodiment, the target object is taken as an example of a face, and the to-be-repaired image can be a low-quality face image.

[0161] For example, the electronic device can acquire the to-be-repaired image from local storage, from a server, or from a third-party electronic device.

[0162] The extraction unit 302 is configured to perform repair information identification on the to-be-repaired image to extract a first image feature corresponding to the target object in the to-be-repaired image.

[0163] For example, the electronic device can extract the first image feature corresponding to the target object in the to-be-repaired image through an image feature extraction algorithm. The image feature extraction algorithm can be a Histogram of Oriented Gradient (HOG) feature extraction algorithm, and the image feature extraction algorithm is not limited herein.

[0164] The determination unit 303 is configured to determine a repair feature of the target object based on the first image feature.

[0165] Optionally, the first image feature is subjected to down-sampling convolution processing to obtain a second image feature of the target object in the to-be-repaired image, reference repair features matching the target object in the to-be-repaired image are acquired, similarity between the reference repair features and the second image feature is calculated, and the second image feature is corrected based on the reference repair feature satisfying a preset condition to obtain the repair feature of the target object.

[0166] For example, the feature distance between the reference repair features A, B, C, and D and the second image feature X is calculated, if the feature distance between the reference repair feature D and the second image feature X is the smallest, it is determined that the similarity between the reference repair feature D and the second image feature X is the highest, the reference repair feature D is replaced by the second image feature X as the repair feature.

[0167] The fusion unit 304 is configured to perform fusion processing on the repair feature and the first image feature to obtain a fused repair feature.

[0168] The fusion processing can be feature deformation processing, such as deforming the repair feature to information corresponding to the first image feature, so as to obtain the fused repair feature.

[0169] In the embodiment, the repair feature has high-quality detail features corresponding to the target object, but the repair feature is obtained by similarity matching the reference repair feature, so the result contains the detail features of the target object, but has a certain degree of untrustworthiness, and the first image feature is directly extracted from the image to be repaired, so it has trustworthiness, and therefore the repair feature and the first image feature are fused to improve the quality and trustworthiness of the repaired image corresponding to the image to be repaired.

[0170] For example, the displacement convolution processing is performed according to the repair feature and the first image feature to determine the displacement feature of the convolution sampling position; and the deformable convolution processing is performed on the repair feature according to the displacement feature to obtain the fused repair feature.

[0171] The repair unit 305 is configured to repair the target object in the image to be repaired based on the fused repair feature to obtain a repaired image.

[0172] Optionally, the fused repair feature is decoded to repair the target object in the image to be repaired to obtain a repaired image.

[0173] The fused repair feature can be a multi-scale fused repair feature, and the decoding of the fused repair feature includes: obtaining an up-sampling scale; and performing up-sampling convolution processing on the fused repair feature based on the up-sampling scale.

[0174] For example, if the scales of the multi-scale fused repair feature are 16x16, 32x32, 64x64, 128x128, 256x256 and 512x512 pixels, and the scale of the image to be repaired is 512x512 pixels, the up-sampling convolution processing on the fused repair feature based on the up-sampling scale can include: performing up-sampling convolution processing on the fused repair feature of each scale in ascending order, for example, if the up-sampling scale is 2, the fused repair feature of 16x16 pixels is processed once to obtain the fused repair feature of 32x32 pixels, and the fused repair feature of 32x32 pixels is processed again to obtain the fused repair feature of 64x64 pixels, and the above steps are sequentially performed to finally obtain the fused repair feature of 512x512 pixels, which is the repaired target object, and the repaired image can be determined according to the repaired target object.

[0175] From the above, the acquisition unit 301 in this embodiment is configured to acquire a to-be-repaired image, the to-be-repaired image containing a target object; the extraction unit 302 is configured to perform repair information identification on the to-be-repaired image to extract a first image feature corresponding to the target object in the to-be-repaired image; the determination unit 303 is configured to determine a repair feature of the target object based on the first image feature; the fusion unit 304 is configured to perform fusion processing on the repair feature and the first image feature to obtain a fused repair feature; and the repair unit 305 is configured to perform repair on the target object in the to-be-repaired image based on the fused repair feature to obtain a repaired image. In this way, the repair feature determined based on the first image feature of the target object in the defective to-be-repaired image can improve the high-definition detail effect of image restoration, and the fusion processing on the repair feature and the first image feature improves the credibility of image restoration, thereby achieving the effect of simultaneously improving the high definition and credibility of the to-be-repaired image.

[0176] The embodiment of the present application also provides an electronic device, as shown in the figure, which shows a structural schematic diagram of the electronic device related to the embodiment of the present application, in particular: Figure 7

[0177] The electronic device can include a processor 401 with one or more processing cores, a memory 402 with one or more computer readable storage media, a power supply 403, an input unit 404, and the like. Those skilled in the art can understand that the structure of the electronic device shown in the figure does not constitute a limitation on the electronic device, and can include more or fewer components than shown, or combine certain components, or different component arrangements. Among them: Figure 7 The processor 401 is the control center of the electronic device, which connects all parts of the electronic device through various interfaces and lines, and performs various functions of the electronic device and processes data by running or executing software programs and / or modules stored in the memory 402 and calling data stored in the memory 402. Optionally, the processor 401 can include one or more processing cores; preferably, the processor 401 can integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface, and application programs, etc., and the modem processor mainly processes wireless communication. It can be understood that the above-mentioned modem processor can also not be integrated into the processor 401.

[0178]

[0179] ​​The memory 402 can be used to store software programs and modules, and the processor 401 can execute various function applications and data processing by running the software programs and modules stored in the memory 402. The memory 402 can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, application programs required by at least one function (such as a sound playing function, an image playing function, etc.), and the like; and the data storage area can store data created according to the use of the electronic device, etc. In addition, the memory 402 can include a high-speed random access memory, and can also include a non-volatile memory, such as at least one magnetic disk storage device, a flash memory device, or other volatile solid-state memory device. Accordingly, the memory 402 can also include a memory controller to provide the processor 401 with access to the memory 402.

[0180] The electronic device also includes a power supply 403 for powering the various components. Preferably, the power supply 403 can be logically connected to the processor 401 through a power management system, so as to realize functions such as management of charging, discharging, and power consumption management through the power management system. The power supply 403 can also include one or more direct current or alternating current power supplies, a recharging system, a power supply fault detection circuit, a power supply converter or inverter, a power supply status indicator, and the like.

[0181] The electronic device can also include an input unit 404, which can be used to receive input digital or character information, and generate keyboard, mouse, joystick, optical or trackball signal inputs related to user settings and function controls.

[0182] Although not shown, the electronic device can also include a display unit and the like, which will not be described here. Specifically, in the present embodiment, the processor 401 in the electronic device will load the executable file corresponding to the process of one or more application programs into the memory 402 according to the following instructions, and run the application programs stored in the memory 402 by the processor 401, so as to realize various functions, as follows:

[0183] An image to be repaired is obtained, the image to be repaired containing a target object; repair information identification is performed on the image to be repaired to extract first image features corresponding to the target object in the image to be repaired; based on the first image features, repair features of the target object are determined; the repair features and the first image features are fused to obtain fused repair features; and based on the fused repair features, the target object in the image to be repaired is repaired to obtain a repaired image.

[0184] The specific implementation of each operation can refer to the foregoing embodiments, which will not be described here.

[0185] From the above, the embodiment of the present application obtains a to-be-repaired image, the to-be-repaired image containing a target object; performs repair information identification on the to-be-repaired image to extract a first image feature corresponding to the target object in the to-be-repaired image; determines a repair feature of the target object based on the first image feature; performs fusion processing on the repair feature and the first image feature to obtain a fused repair feature; and performs repair on the target object in the to-be-repaired image based on the fused repair feature to obtain a repaired image. In this way, the repair feature determined based on the first image feature of the target object in the defective to-be-repaired image can improve the high-definition detail effect of image restoration, and the fusion processing on the repair feature and the first image feature improves the credibility of image restoration, thereby achieving the effect of simultaneously improving the high definition and credibility of the to-be-repaired image.

[0186] Those skilled in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructions or controlled by instructions related to hardware, which can be stored in a computer readable storage medium and loaded and executed by a processor.

[0187] To this end, the embodiment of the present application provides a computer readable storage medium, which stores a plurality of instructions capable of being loaded by a processor to execute the steps in any of the image repair methods provided by the embodiments of the present application. For example, the instructions can execute the following steps:

[0188] Obtain a to-be-repaired image, the to-be-repaired image containing a target object; perform repair information identification on the to-be-repaired image to extract a first image feature corresponding to the target object in the to-be-repaired image; determine a repair feature of the target object based on the first image feature; perform fusion processing on the repair feature and the first image feature to obtain a fused repair feature; and perform repair on the target object in the to-be-repaired image based on the fused repair feature to obtain a repaired image.

[0189] The specific implementation of each operation can be referred to the previous embodiments, which will not be described here.

[0190] The computer readable storage medium can include a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0191] Since the instructions stored in the computer readable storage medium can execute the steps in any of the image repair methods provided by the embodiments of the present application, the beneficial effects that can be achieved by any of the image repair methods provided by the embodiments of the present application can be achieved, which will be described in detail in the previous embodiments, and will not be described here.

[0192] According to an aspect of the present application, a computer program product or computer program is provided, which includes computer instructions stored in a computer readable storage medium. A processor of a computer device reads the computer instructions from the computer readable storage medium, and the processor executes the computer instructions, so that the computer device executes the method provided in the various optional implementations of the image inpainting aspect.

[0193] The above describes in detail the image inpainting method, device, computer device, computer readable storage medium, and computer program product provided by the embodiments of the present application. The principles and implementation manners of the present application are described by applying specific examples. The above description of the embodiments is only used to help understand the method of the present application and its core idea. Meanwhile, for those skilled in the art, the specific implementation manners and application ranges can be changed according to the idea of the present application. In summary, the content of the specification should not be understood as a limitation of the present application.

Claims

1. An image inpainting method characterized by, The method comprises the following steps: acquire a to-be-repaired image, the to-be-repaired image containing a target object; perform repair information identification on the to-be-repaired image to extract a first image feature corresponding to the target object in the to-be-repaired image; determine a repair feature of the target object based on the first image feature, wherein the determination comprises: performing down-sampling convolution processing on the first image feature to obtain a second image feature of the target object in the to-be-repaired image; and determining the repair feature of the target object based on the second image feature; the determination of the repair feature of the target object based on the second image feature comprises: acquire a reference repair feature matching the target object in the to-be-repaired image; determine the repair feature of the target object based on the second image feature and the reference repair feature; perform fusion processing on the repair feature and the first image feature to obtain a fused repair feature; perform repair on the target object in the to-be-repaired image based on the fused repair feature to obtain a repaired image.

2. The image inpainting method of claim 1, wherein, the fusion processing of the repair feature and the first image feature to obtain a fused repair feature comprises: perform up-sampling convolution processing on the repair feature to obtain a multi-scale repair feature; perform down-sampling processing on the first image feature to obtain a multi-scale image feature; perform fusion processing on the multi-scale repair feature and the multi-scale image feature to obtain a fused repair feature under multi-scale.

3. The image inpainting method of claim 1, wherein, the fusion processing of the repair feature and the first image feature to obtain a fused repair feature comprises: perform displacement convolution processing on the repair feature and the first image feature to determine a displacement feature of a convolution sampling position; perform deformable convolution processing on the repair feature based on the displacement feature to obtain the fused repair feature.

4. The image inpainting method of claim 1, wherein, the repair on the target object in the to-be-repaired image based on the fused repair feature to obtain a repaired image comprises: perform decoding processing on the fused repair feature to repair the target object in the to-be-repaired image to obtain a repaired image.

5. The image inpainting method of claim 4, wherein, the decoding processing on the fused repair feature comprises: acquire an up-sampling scale; perform up-sampling convolution processing on the fused repair feature based on the up-sampling scale.

6. The image inpainting method of claim 1, wherein, The repair information comprises at least one of a texture, an outline, and a position of the target object, and the repair information identification on the to-be-repaired image to extract a first image feature corresponding to the target object in the to-be-repaired image comprises: perform convolution operation on the to-be-repaired image to extract at least one of a texture, an outline, and a position of the target object in the to-be-repaired image; determine the first image feature corresponding to the target object based on the at least one of the texture, the outline, and the position.

7. The image inpainting method of claim 1, wherein, the determination of the repair feature of the target object based on the second image feature and the reference repair feature comprises: calculate a similarity between the reference repair feature and the second image feature; correct the second image feature based on the reference repair feature satisfying a preset condition to obtain the repair feature of the target object.

8. The image inpainting method of claim 7, wherein, The reference repair feature based on the similarity satisfying the preset condition is used to correct the second image feature, to obtain the repair feature of the target object, including: The reference repair feature with the highest similarity is used to replace the second image feature, to obtain the repair feature of the target object.

9. The image inpainting method of claim 7, wherein, Before the reference repair feature based on the similarity satisfying the preset condition is used to correct the second image feature, to obtain the repair feature of the target object, including: The feature distance between the reference repair feature and the second image feature is calculated; If the feature distance is less than a preset threshold, it is determined that the similarity between the reference repair feature and the second image feature satisfies the preset condition.

10. An image inpainting apparatus characterized by comprising: Including: An acquisition unit is configured to acquire a to-be-repaired image, the to-be-repaired image containing a target object; An extraction unit is configured to perform repair information identification on the to-be-repaired image, to extract a first image feature corresponding to the target object in the to-be-repaired image; A determination unit is configured to determine a repair feature of the target object based on the first image feature; wherein the determination unit is specifically configured to perform down-sampling convolution processing on the first image feature, to obtain a second image feature of the target object in the to-be-repaired image; and determine the repair feature of the target object based on the second image feature; The determination of the repair feature of the target object based on the second image feature includes: Obtaining a reference repair feature matching the target object in the to-be-repaired image; Determining the repair feature of the target object based on the second image feature and the reference repair feature; A fusion unit is configured to perform fusion processing on the repair feature and the first image feature, to obtain a fused repair feature; A repair unit is configured to perform repair on the target object in the to-be-repaired image based on the fused repair feature, to obtain a repaired image.

11. A computer device, comprising: A memory and a processor are included, the memory stores a computer program, and the processor is used to run the computer program in the memory to execute the image repair method in any one of claims 1 to 9.

12. A computer-readable storage medium, characterized in that, The computer readable storage medium is used to store a computer program, the computer program is loaded by the processor to execute the image repair method in any one of claims 1 to 9.

13. A computer program product comprising a computer program, characterized in that, The computer program is executed by the processor to implement the image repair method in any one of claims 1 to 9.

Images