Image restoration method and device and storage medium

Abstract

The invention relates to an image restoration method and device and a storage medium. The image restoration method comprises the following steps: determining a restoration area boundary of a to-be-restored image; determining a target pixel point on the boundary of the restoration area, wherein the target pixel point is the first pixel point to be restored on the boundary of the restoration area; determining a to-be-restored image block taking the target pixel point as a central point, and determining a target image block; replacing the pixel points in the to-be-restored image block with the pixel points in the target image block to obtain a restored image block; on the basis of the repaired image block, updating the boundary of the to-be-repaired area; and taking the updated to-be-restored region boundary as a restored region boundary, and repeatedly executing the process until the to-be-restored image does not have the restored region boundary. According to the invention, the definition, texture continuity and rationality of the restored image can be improved.

Description

Technical Field

[0001] The present disclosure relates to the field of image processing, and in particular to an image restoration method, device and storage medium. Background Art

[0002] The purpose of image restoration is to fill in the pixels of a missing area in a given image so that the image can be consistent in texture and structure, or semantically and visually credible. It has a wide range of applications, such as image restoration, image editing, and image denoising. The restoration process of image restoration itself is highly subjective, such as expert restoration of works of art.

[0003] In related technologies, most deep learning methods regard image restoration as a conditional generation problem, where the mapping from input to output is one-to-many, and an optimal result is generated in the end. However, the result does not have rich semantic content, and the content of the generated result may be different, but it still looks reasonable.

[0004] Traditional restoration methods can be divided into two categories: diffusion-based and sample-based image restoration methods. Compared with diffusion-based image restoration methods, which mainly focus on the restoration of small areas, the commonly used digital image restoration algorithm at this stage is the restoration method based on sample point matching. This method is mainly prone to erroneous restoration methods due to the rapid decay of the confidence term. In some cases, it is difficult to effectively distinguish between edge areas and texture areas, affecting clarity. This method will also limit the effect of damaged image restoration, resulting in texture discontinuity, and the restoration effect may be unreasonable. Summary of the invention

[0005] In order to overcome the problems existing in the related art, the present disclosure provides an image restoration method, device and storage medium.

[0006] According to a first aspect of an embodiment of the present disclosure, there is provided an image restoration method, comprising: determining a restoration area boundary of an image to be restored; determining a target pixel point on the restoration area boundary, the target pixel point being the first pixel point to be restored on the restoration area boundary; determining an image block to be restored with the target pixel point as the center point, and determining a target image block, the target image block being an image block in the image to be restored that has the greatest similarity with the image block to be restored and has the same size; replacing the pixels in the image block to be restored with the pixels in the target image block to obtain a restored image block; updating the restoration area boundary based on the restored image block; using the updated restoration area boundary as the restoration area boundary, and repeatedly executing the above image restoration method process until the restoration area boundary no longer exists in the image to be restored.

[0007] In one embodiment, determining the target pixel point on the boundary of the repair area includes: determining the brightness weight, gradient weight and effective information weight of the pixel point on the boundary of the repair area, wherein the larger the brightness channel value, the higher the corresponding brightness weight, the larger the gradient, the higher the corresponding gradient weight, and the more effective pixels around the pixel point, the higher the effective information weight; determining the priority of all pixel points on the boundary of the repair area according to the brightness weights, the gradient weights and the effective information weights of all pixel points on the boundary of the repair area; and determining the pixel point on the boundary of the repair area with the highest priority as the target pixel point.

[0008] In one embodiment, determining the target image block includes: determining a target image block search area in the image to be repaired, wherein a central pixel point of the target image block search area is a central point of the image block to be repaired, and a size of the target image block search area is larger than an area size of the image block to be repaired; and determining the target image block in the target image block search area.

[0009] In one embodiment, determining the target image block search area in the image to be repaired includes: determining the background image of the image to be repaired based on the brightness value of each pixel in the image to be repaired; and determining the target image block search area from the background image.

[0010] In one embodiment, the method further includes: recording a first target image block corresponding to a first repaired image block, the first target image block including pixels located in a credible area, the credible area being an image area in the image to be repaired that is outside a boundary of the repair area; determining the target image block in the target image block search area includes: if all the determined target image blocks are located in a non-credible area, determining the first target image block as the target image block of the current image block to be repaired.

[0011] In one implementation, the method further includes: if it is determined that the restoration area boundary does not exist in the image to be restored, performing edge filtering on the restoration area composed of the restored image blocks.

[0012] According to a second aspect of an embodiment of the present disclosure, there is provided an image restoration device, comprising: a detection module, for determining a restoration area boundary of an image to be restored; a processing module, for determining a target pixel point on the restoration area boundary, the target pixel point being the first pixel point to be restored on the restoration area boundary; determining an image block to be restored with the target pixel point as the center point, and determining a target image block, the target image block being an image block in the image to be restored that has the greatest similarity with the image block to be restored and has the same size; replacing the pixels in the image block to be restored with the pixels in the target image block to obtain a restored image block; updating the restoration area boundary based on the restored image block; using the updated restoration area boundary as the restoration area boundary, and repeatedly executing the above image restoration method process until the restoration area boundary does not exist in the image to be restored.

[0013] In one embodiment, the processing module determines the target pixel point on the boundary of the repair area in the following manner: determine the brightness weight, gradient weight and effective information weight of the pixel point on the boundary of the repair area, wherein the larger the brightness channel value, the higher the corresponding brightness weight, the larger the gradient, the higher the corresponding gradient weight, and the more effective pixels around the pixel point, the higher the effective information weight; determine the priority of all pixel points on the boundary of the repair area according to the brightness weight, the gradient weight and the effective information weight of all pixel points on the boundary of the repair area; determine the pixel point on the boundary of the repair area with the highest priority as the target pixel point.

[0014] In one embodiment, the processing module determines the target image block in the following manner: determining a target image block search area in the image to be repaired, wherein the central pixel point of the target image block search area is the central point of the image block to be repaired, and the size of the target image block search area is larger than the area size of the image block to be repaired; and determining the target image block in the target image block search area.

[0015] In one embodiment, the processing module determines the target image block search area in the image to be repaired in the following manner: based on the brightness value of each pixel in the image to be repaired, determining the background image of the image to be repaired; and determining the target image block search area from the background image.

[0016] In one embodiment, the processing module is further used to: record a first target image block corresponding to a first repaired image block, the first target image block including pixels located in a credible area, the credible area being an image area in the image to be repaired that is outside the boundary of the repair area; and determine the target image block in the target image block search area, including: if all the determined target image blocks are located in a non-credible area, determining the first target image block as the target image block of the current image block to be repaired.

[0017] In one implementation, the processing module is further configured to: if it is determined that the restoration region boundary does not exist in the image to be restored, perform edge filtering on the restoration region composed of the restored image blocks.

[0018] According to a third aspect of an embodiment of the present disclosure, there is provided an image restoration device, comprising: a processor; and a memory for storing instructions executable by the processor; wherein the processor is configured to: execute the image restoration method described in the first aspect or any one of the embodiments of the first aspect.

[0019] According to a fourth aspect of an embodiment of the present disclosure, a storage medium is provided, in which instructions are stored. When the instructions in the storage medium are executed by a processor of a terminal, the terminal is enabled to execute the image restoration method described in the first aspect or any one of the embodiments of the first aspect.

[0020] The technical solution provided by the embodiments of the present disclosure may include the following beneficial effects: by searching for target image blocks to fill the image blocks to be repaired in the repair area, the texture continuity of the image repair result is higher; after each repair of the image block to be repaired centered on the first target pixel point is completed, the repair area is updated to avoid repeated repairs, thereby improving the repair efficiency; the image repair method based on target block search makes the image texture continuity after the repair is better, the clarity is higher, and the repair result is more reasonable.

[0021] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0023] Figure 1 The figure is a flow chart of an image restoration method based on target block search according to an exemplary embodiment.

[0024] Figure 2aThe figure is a flow chart showing a method for determining a target pixel point according to an exemplary embodiment.

[0025] Figure 2b The figure is a schematic diagram showing a method of determining a target pixel point according to an exemplary embodiment.

[0026] Figure 3a The invention is a flowchart showing a method for determining a target image block in a target image block search area according to an exemplary embodiment.

[0027] Figure 3b The diagram is a schematic diagram showing a method of determining a target image block in a target image block search area according to an exemplary embodiment.

[0028] Figure 4 The present invention is a flowchart of determining a search area for a target image block in a background image according to an exemplary embodiment.

[0029] Figure 5a The present invention is a flowchart showing a method of determining a target image block in a trusted area according to an exemplary embodiment.

[0030] Figure 5b The diagram is a schematic diagram showing a method of determining a target image block in a trusted area according to an exemplary embodiment.

[0031] Figure 6 The figure is a flowchart of an image restoration method according to an exemplary embodiment.

[0032] Figure 7 The figure is a flowchart of an image restoration method according to an exemplary embodiment.

[0033] Figure 8 The figure is a schematic diagram of a process of repairing an image after image fusion according to an exemplary embodiment.

[0034] Fig. 9 The figure is a schematic diagram showing a comparison of image restoration effects according to an exemplary embodiment.

[0035] Fig.10 The figure is a block diagram of an image restoration device according to an exemplary embodiment.

[0036] Fig.11 The invention is a block diagram of a device for image restoration according to an exemplary embodiment. DETAILED DESCRIPTION

[0037] Here, exemplary embodiments will be described in detail, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present disclosure.

[0038] The image restoration method provided in the present disclosure is mainly used for processing various types of pictures that need to be restored in terminal devices. Among them, the terminal device in the present disclosure can be an electronic device that can acquire and process images, such as a smart terminal, a digital camera, and a video camera.

[0039] In the related art, the weight of the target pixel on the boundary of the image to be repaired is usually used to determine the repair priority of the target point, and then the target pixel is filled according to the information of the adjacent pixel points of the target point, such as the Fast Marching Method (FMM) and the interpolation algorithm based on the connected domain boundary. Among them, when the fast matching method selects the surrounding known information, it is caused by selecting all points in the neighborhood and calculating the corresponding direction factor, geometric distance factor, level set factor and then continuing the operation, which has a certain degree of blindness. And it will inevitably introduce more information that is very little or even irrelevant to the point to be repaired, which will cause blurred image details when repairing a large range. The interpolation algorithm based on the boundary of the connected domain will cause blurred image details when repairing a large range for similar reasons to the fast matching method. The image repair method based on deep learning depends on the data set, and the repair effect is also based on the generalization ability of the data set, and the robustness is relatively low.

[0040] In view of this, the embodiment of the present disclosure provides an image restoration method, which can repair the to-be-restored area of ​​various types of images through target block search, improve image restoration efficiency, and improve image restoration effect. Compared with the traditional sample-based point restoration method, the image restoration method provided by the embodiment of the present disclosure is a sample block-based restoration method, which makes the texture continuity of the image restoration area better and the clarity higher. Compared with the traditional sample block-based image restoration method, the image restoration method provided by the embodiment of the present disclosure uses different weights and target block search methods, so that the image restoration effect is more accurate and more reasonable. In one embodiment, in the image restoration method provided by the embodiment of the present disclosure, for the image to be restored, based on the boundary of the image area to be restored determined in the image to be restored, the first target point to be restored on the boundary is determined and the image block to be restored is established with the target point as the center, and the target image block is determined to fill the pixel points in the image block to be restored to complete the restoration of the image block to be restored. After the restoration of the image block to be restored is completed, the boundary of the area to be restored is updated to re-determine a new graphic block to be restored, and the above process is repeated until there is no area to be restored in the image to be restored, so as to improve the texture continuity and rationality of the restored image.

[0041] Figure 1 is a flowchart of an image restoration method based on target block search according to an exemplary embodiment. Figure 1 As shown, the image restoration method based on target block search is applied in the terminal, including the following steps:

[0042] In step S11, the boundary of the repair area of ​​the image to be repaired is determined.

[0043] The image restoration method in the disclosed embodiment is to perform restoration from the boundary of the restoration area to the interior of the area to be restored, so the boundary of the restoration area needs to be determined first as the starting position for restoring the image to be restored.

[0044] In step S12, a target pixel point is determined on the boundary of the repair area, where the target pixel point is the first pixel point to be repaired on the boundary of the repair area.

[0045] In the disclosed embodiment, there are one or more pixels on the border of the repair area, and the first pixel to be repaired is the one with the highest priority among the one or more pixels on the repair area. The position of the first pixel to be repaired on the border of the repair area is determined as the pixel position from which the image repair starts on the border of the repair area, that is, the position of the target pixel. The target pixel is the center point of the image block to be repaired that currently needs to be repaired, and has the highest priority among the one or more pixels on the border of the current repair area.

[0046] Among them, the position of the first pixel point to be repaired is the target pixel point position, which is used to determine the position of the current image block to be repaired centered on this pixel point, and start image repair from the boundary of the repair area according to the position of the current image block to be repaired.

[0047] In step S13, an image block to be repaired with the target pixel as the center point is determined, and a target image block is determined. The target image block is an image block in the image to be repaired that has the greatest similarity with the image block to be repaired and has the same size.

[0048] In the disclosed embodiment, the target pixel point is the first pixel point to be repaired on the boundary of the repair area, which is used to determine the position of the image block to be repaired with the target pixel point as the center, and the image block to be repaired is the first image block to be repaired in the current repair area.

[0049] In the disclosed embodiment, the image block to be repaired is the first image block to be repaired in the current repair area, and the target image block corresponding to the image block to be repaired is an image block located outside the repair area before the first repair and used to replace the image block to be repaired. Each image block to be repaired corresponds to a target image block, and the target image blocks corresponding to each image block to be repaired can be the same or different.

[0050] In the disclosed embodiment, the target image block is used to match the internal pixel points of the image block to be repaired, and is used to fill or replace the internal pixel points of the image block to be repaired. The target image block is the one in the image to be repaired that has the greatest similarity with the image block to be repaired and is the same size, so as to prevent omissions in the process of filling or replacing internal pixels, and to prevent the filling effect of the internal pixels in the image block to be repaired from being too different from the expected effect due to low similarity, resulting in a reduced image repair effect after repair.

[0051] In step S14, the pixels in the image block to be repaired are replaced with the pixels in the target image block to obtain a repaired image block.

[0052] In the disclosed embodiment, the pixels in the image block to be repaired are replaced with the pixels in the target image block, so as to modify the internal pixels of the image block to be repaired, thereby repairing the image block area to be repaired.

[0053] The repaired image block is the first repaired image block in the current repair area.

[0054] In step S15, based on the restored image block, the boundary of the area to be restored is updated.

[0055] In the disclosed embodiment, after the image block to be repaired is repaired, a repaired image block is obtained, and the repaired image block is used as the repaired area. The updated boundary of the area to be repaired is the boundary of the repair area that has not been repaired.

[0056] In step S16, the updated boundary of the region to be repaired is used as the boundary of the repair region, and the above image repair process is repeatedly performed until the boundary of the repair region does not exist in the image to be repaired.

[0057] In the disclosed embodiment, the updated boundary of the area to be repaired is used as the boundary of the repair area, that is, the repaired area is distinguished as a repaired area from an unrepaired repair area, and the current repair area boundary is re-determined as the starting position for the next repair.

[0058] The fact that there is no repair region boundary in the image to be repaired means that there is no repair region to be repaired in the image to be repaired, that is, all repair regions have been repaired.

[0059] In the disclosed embodiment, each repair process is to repair the first image block to be repaired in the current repair area, and the above repair process is repeated several times until all repair areas to be repaired are completely repaired, that is, there is no repair area to be repaired.

[0060] In the disclosed embodiment, each time the image restoration process in the above embodiment is performed, the restoration area boundary will be updated, and the pixel point information on the restoration area boundary will also change accordingly. Therefore, each time the restoration area boundary is updated in the image restoration process in the above embodiment, the target pixel point must be re-determined as the first pixel point to be restored.

[0061] In the disclosed embodiment, an image block to be repaired is determined, and a corresponding target image block is determined to replace the image block to be repaired so as to implement an image repair method based on target block search. The correlation between the image block to be repaired and the target block is higher than that between the point to be repaired and the target point used to fill the point to be repaired. Because the image block includes multiple pixel points, the large similarity indicates that the correlation between the multiple pixel points between the image blocks is high, and the repair effect after repair is also better. By repairing the image using the above-mentioned image repair method in the disclosed embodiment, the image after repair has better clarity, and higher texture continuity and rationality.

[0062] In one implementation, SSD (sum of squared differences) may be used to calculate the similarity between the current image block to be repaired and the target block. The SSD calculation formula is as follows:

[0063]

[0064] For Bayer array images, the SSD of the four channels of all pixels in the image block will be calculated separately, and the final SSD of the target image block and the image block to be repaired will be accumulated. The target image block will be selected to repair the image block to be repaired. The final SSD obtained by accumulating the SSD of the four channels of the Bayer array image is only used to obtain the SSD of the Bayer array image. If the SSD of the three channels is calculated for RGB (Red Green Blue, a color mode), the final SSD can be accumulated. The above method for obtaining the final SSD can be transformed accordingly according to the channel information of different types of images. Other methods can also be used to calculate the similarity between the current image block to be repaired and the target block.

[0065] In one implementation, the image to be repaired may be an image that has a region with missing valid information, an image that has a region with missing valid information due to high dynamic image fusion, or an image that has a region with missing valid information due to other image processing.

[0066] In one implementation, the image to be restored may be a Bayer (a color mode) image, an RGB (RedGreenBlue, a color mode) image, or an image in another format.

[0067] In one implementation, the area to be repaired may be an area where information is missing in the image after high-dynamic image fusion, an area disturbed by electric current during image capture, or an area to be repaired where effective information is missing due to other reasons.

[0068] In one implementation, the image to be repaired may not have a repair area, that is, no area with missing information. In this case, the image does not need to be repaired, and the image processing method in the embodiment of the present disclosure is skipped to directly output the image.

[0069] When the boundary of the area to be repaired is close to the edge of the image to be repaired or coincides with the edge of the image to be repaired, it may not be possible to determine the image block to be repaired in the image to be repaired with the target pixel point on the boundary of the repair area as the center point. At this time, part of the area in the image block to be repaired may be repaired or the image block to be repaired may not be determined with the target pixel point on the boundary of the repair area as the center point. For example, in one embodiment, the center point of the image block to be repaired may not be the target pixel point located on the boundary of the repair area. At this time, the position of the image block to be repaired may be confirmed and repaired at a position close to the edge of the image to be repaired according to the size of the image block to be repaired. At this time, the edge of the image block to be repaired may be close to the edge of the image or partially coincide with the edge of the image. For example, in one embodiment, the target pixel point may also be determined on the edge of the image to be repaired, and the image block to be repaired may be determined based on the target pixel point. Part of the area in the image block to be repaired may be outside the image to be repaired. At this time, the area to be repaired in the image block to be repaired within the image to be repaired may be repaired, and the area outside the image to be repaired in the image block to be repaired may not be repaired.

[0070] In the above embodiment, the image restoration process is based on the determination of the position of the block to be restored, and the position of the block to be restored depends on the target pixel point as the center point. The determination of the target pixel point determines the position of the block to be restored and also determines the starting position of the image restoration, thereby affecting the restoration effect of the restoration area of ​​the image to be restored. The process of determining the target pixel point is described below.

[0071] Figure 2a The figure is a flow chart showing a method for determining a target pixel point according to an exemplary embodiment.

[0072] In step S21, the brightness weight, gradient weight and effective information weight of the pixel points on the boundary of the repair area are determined, wherein the larger the brightness channel value, the higher the corresponding brightness weight, the larger the gradient, the higher the corresponding gradient weight, and the more effective pixels around the pixel point, the higher the effective information weight.

[0073] In the disclosed embodiment, the target pixel point is determined from the pixel points on the boundary of the repair area.

[0074] In the disclosed embodiment, the pixel points on the boundary are sorted according to the size of the brightness channel, and the pixel point with greater brightness has a higher brightness weight.

[0075] In the disclosed embodiment, the gradient weight reflects the edge characteristics of the image. For the edge portion of the image, the grayscale value changes greatly and the gradient value is also large; for the relatively smooth portion of the image, the grayscale value changes less and the corresponding gradient value is also small.

[0076] In the disclosed embodiment, the effective information weight represents the number of nearby effective information points. An effective information point is a pixel point outside the repair area boundary that does not need to be repaired. The more effective information points there are around a pixel point, the higher the effective information weight of the pixel point.

[0077] In step S22, the priority of all the pixels on the border of the repair area is determined according to the brightness weight, gradient weight and effective information weight of all the pixels on the border of the repair area.

[0078] In the embodiment of the present disclosure, the priority of all pixels on the boundary of the repair area is determined to sort all the pixels and determine the pixel with the highest priority.

[0079] In step S23, a pixel point on the boundary of the repair area with the highest priority is determined as a target pixel point.

[0080] In the disclosed embodiment, the priority can be obtained by the priority calculation formula in one of the above-mentioned implementation modes. Among them, the larger the gradient value, that is, the larger the gradient weight of the pixel point, the more obvious the outline of the image. The brightness weight of the pixel point is reflected in the visual effect of the human eye. The higher the brightness, the higher the recognition of the human eye, that is, the higher the brightness weight. The higher the effective information weight of the pixel point, the higher the selectivity of the nearby area for filling the pixel point, and the image block to be repaired centered on the target pixel point can find a target image block with a large similarity for pixel point replacement in the target block search area.

[0081] In one embodiment, when there are multiple pixels with the same weight on the border of the repair area, one of them can be randomly selected as the target pixel, that is, the center position of the first image block to be repaired at the beginning of the repair, to determine the position of the current image block to be repaired at the beginning of the image repair. That is, one image block to be repaired is determined from multiple image blocks to be repaired with the same center point weight as the image block to be repaired that currently needs to be repaired. For example, when the border of the repair area is circular, there may be multiple pixels with the same priority on the border of the area to be repaired, and one of the pixels can be randomly selected as the target pixel to determine the image block to be repaired that needs to be repaired.

[0082] In the disclosed embodiment, the position of the target pixel is determined according to the texture change of the image, that is, the pixel on the boundary of the repair area with the highest priority is determined as the target pixel, that is, the center point of the image block to be repaired, so that the texture clarity of the repaired image is higher. At the same time, because of the effective information weight and brightness weight, more effective information is referenced, so that the rationality of the repaired image is also higher.

[0083] Figure 2b The figure is a schematic diagram showing a method of determining a target pixel point according to an exemplary embodiment.

[0084] like Figure 2b As shown, where Ω 0 is the boundary of the repair area before repairing the image to be repaired, Ω 1 is the boundary of the repair area after multiple repairs. P0 is the boundary Ω 0 The target pixel on the 1 The target pixel on the image block Ψ P0 and P1 are the image blocks to be repaired centered at pixel points P0 and P1, and the target image block search area W 0 and W 1 They are image blocks Ψ P0 and P1 The corresponding target image block search area.

[0085] In the disclosed embodiment, the priority-based image repair algorithm enables the repair process to proceed from the outside to the inside, such as Figure 2b As shown, Ω 0 is the boundary of the repair area before the first repair, Ω 1 is the new restoration area boundary after multiple image restoration processes. 0 The size of the region in is greater than Ω 1 The area in which Ω 1 The area in Ω 0 inside the area in .

[0086] In one implementation, the weight in the above pixel priority calculation formula may not exist. For example, when repairing points within the repair area boundary, there may be no valid information points around the target pixel due to location reasons. In this case, the effective information weight of the pixel may not exist. Figure 2b As shown, the image block to be repaired Ψ P1 The center point is the repair area Ω after multiple repairs 1 Points on the boundary, at this time repair the image block Ψ P1 There are only repaired areas and unrepaired repaired areas near the center point of , and there are no valid information points, that is, there is no valid information weight. At this time, Wmask is 0. Similarly, brightness weight and gradient weight may also be non-existent due to other reasons.

[0087] In one implementation, the brightness channel value can be obtained by calculating the four-channel information of the Bayer array image. The brightness channel value obtained by calculating the four-channel information of the Bayer array image is only used to obtain the brightness channel value of the Bayer image. If the brightness channel value of the RGB image can be obtained by calculating three channels, the brightness channel value obtaining method can be transformed accordingly according to the channel information of different types of images.

[0088] In one implementation, the gradient information of the image to be repaired may be determined in the following manner:

[0089] The Sobel operator is a gradient detection operator, which consists of two convolution kernels Gx and Gy as follows:

[0090]

[0091] The horizontal gradient and vertical gradient information of the image to be repaired can be obtained by performing sliding window convolution on the image to be repaired using Gx and Gy in the Sobel operator. Other methods other than the Sobel operator can also be used to determine the gradient information of the image to be repaired.

[0092] In one implementation of the present disclosure, the repair priority of each pixel on the boundary can be obtained by the following pixel priority calculation formula:

[0093] priority=w bright *c bright +w mask *c mask +w grad *c grad

[0094] Among them, peiority is the priority, W bright is the brightness weight, W mask is the effective information weight, W grad is the gradient weight, C bright , C mask , C grad It is a parameter that can be adjusted manually according to the actual repair effect required.

[0095] In the above embodiment, the priority of the pixel points is determined according to the brightness weight, the gradient weight and the effective information weight, and the point with the highest priority is used as the target pixel point, that is, the center point of the image block to be repaired. The determination of the target pixel point determines that the repair process can be started from the bright area to the dark area, from the outside to the inside and along the area with high gradient, so that the clarity and rationality of the repair process are higher.

[0096] Figure 3a The invention is a flowchart showing a method for determining a target image block in a target image block search area according to an exemplary embodiment.

[0097] In step S31, a target image block search area is determined in the image to be repaired, the central pixel point of the target image block search area is the central point of the image block to be repaired, and the size of the target image block search area is larger than the area size of the image block to be repaired.

[0098] In the disclosed embodiment, the target image block search area is larger than the image block area to be repaired, so that the target block search area includes an area outside the image block to be repaired for determining the target image block.

[0099] In step S32, a target image block is determined in the target image block search area.

[0100] In the disclosed embodiment, a target image block is determined in the target image block search area so that the target image block has a greater correlation with the image block to be repaired.

[0101] In the disclosed embodiment, by determining the target image block search area, the target image block is determined in the target image block search area, so that the determination of the target image block will not be due to the distance from the image block to be repaired being too far. For example, when the target image block has a great similarity with the image block to be repaired, but has no reference because of the distance being too far, and it is necessary to search all areas of the image to be repaired, the amount of calculation is increased, and the efficiency of image repair is reduced. Therefore, by determining the target image block search area and determining the target image block in the target image block search area, the efficiency of image repair is improved.

[0102] Figure 3b The diagram is a schematic diagram showing a method of determining a target image block in a target image block search area according to an exemplary embodiment.

[0103] like Figure 3b As shown, where Ω 0 is the boundary of the repair area to be repaired, and the image block Ψ P0 To repair the region boundary Ω 0 The image block to be repaired is centered on the target pixel on the image block Ψ 0 is the image block Ψ P0 The corresponding target image block. Among them, the target image block search area W 0 is the image block Ψ P0 The corresponding target image block search area, image block Ψ 0 Located in Area W 0 middle.

[0104] In one embodiment, if Figure 3b As shown, the image block Ψ P0is the image block to be repaired, which is determined by taking the target pixel point on the boundary of the repair area as the center point. 0 The search area is the target image block centered on the current target pixel, and the area size is larger than the area size of the image block to be repaired. P0 When it is a 3×3 image block, W 0 It can be a 5×5 area, then W 0 Greater than Ψ P0 Region. The above 3×3 and 5×5 can be set to other region sizes, which are not fixed sizes. They can be set to other values ​​when the search region size of the target image block is larger than the region size of the image block to be repaired.

[0105] In the above embodiment, the efficiency of image restoration is improved by determining the target image block in the target image block search area. In order to further improve the effect of image restoration, in one implementation of the disclosed embodiment, the background image of the image to be restored is determined by image segmentation, and the target image block search area is determined in the background image. The process of determining the target image block search area in the background image is described below.

[0106] Figure 4 The present invention is a flowchart of determining a search area for a target image block in a background image according to an exemplary embodiment.

[0107] In step S41, a background image of the image to be restored is determined based on the brightness value of each pixel in the image to be restored.

[0108] In one implementation of the disclosed embodiment, OTSU (Otsu's adaptive threshold segmentation) can be used to obtain the foreground image and background image of the image, wherein the background image is used for target block search.

[0109] In step S42, a target image block search area is determined from the background image.

[0110] In the disclosed embodiment, the above-mentioned repair method is used to repair the background image, which is manifested in the high-dynamic imaging process as the foreground image incorrectly occluding the background image, resulting in information missing areas in the background image, such as the ghost image caused by the shaking of the hand of the person being photographed when taking a photo. The ghost image is the area that needs to be repaired, that is, the area with missing information. Among them, the ghost image can be other types of areas with missing information. Among them, the search for the target image block is to search from the valid area in the background image, so the target image block search area should also be in the background image.

[0111] In the disclosed embodiment, by segmenting the background image and the foreground image, the target image block search area is determined in the background image, thereby removing the influence of the image foreground on the image restoration. For example, near the junction of the foreground image and the background image, the image block to be restored may match the target image block of the foreground image, which will further blur the background image content and reduce the restoration effect of the image to be restored. Therefore, by determining the target image block search area in the background image, it is helpful to improve the reliability of the image restoration process.

[0112] In the above embodiment, a target image block search area is determined in the background image. The center point of the target image block search area is the center point of the image block to be repaired. When the image repair is performed inside the repair area and there is no valid information area in the target image block search area, the target image block currently being repaired for the image in the area to be repaired is determined from the first determined target image block. The following describes the process of determining the target image block when there is no valid information area in the target image block search area.

[0113] Figure 5a The present invention is a flowchart showing a method of determining a target image block in a trusted area according to an exemplary embodiment.

[0114] In step S51, a first target image block corresponding to a first restored image block is recorded, wherein the first target image block includes pixels located in a credible area, and the credible area is an image area outside a restoration area boundary in the image to be restored.

[0115] In the disclosed embodiment, each time the image is repaired and the boundaries of the repaired area are updated, the current target image block is recorded, wherein the recorded target image block contains valid information, i.e., the target image block may contain a partially repaired area. All recorded target image blocks have valid information to prevent the subsequent image repair effect from being reduced due to the lack of valid information, i.e., all pixels are repaired or not repaired, thereby preventing the iteration of information loss from causing the effect of subsequent image repair to be reduced. The recorded target image blocks are multiple target image blocks.

[0116] In step S52, if all the determined target image blocks are located in the untrusted area, the first target image block is determined as the target image block of the current image block to be repaired.

[0117] In the embodiment of the present disclosure, all the determined target image blocks are located in the untrusted area, that is, all the target image blocks in the search area are located in the untrusted area. The first target image block is one of the target image blocks in the recorded image blocks.

[0118] In one implementation, all pixels in the target image block are from the image.

[0119] In the embodiment of the present disclosure, when there is no target image block in the target image block search area, the target image block is determined from the first recorded target image block for repairing the image block to be repaired. The method for determining the target image block in the embodiment of the present disclosure avoids the unreasonable image repair result caused by the determination of the target image block in the untrusted area during the image repair process and the iteration of the repair information during the image repair process. In addition, the image repair method provided by the embodiment of the present disclosure will not cause image blurring due to the use of all image blocks in the untrusted area.

[0120] Figure 5b The diagram is a schematic diagram showing a method of determining a target image block in a trusted area according to an exemplary embodiment.

[0121] like Figure 5b As shown, where Ω 0 is the boundary of the repair area before repairing the image to be repaired, Ω m is the boundary of the repair area after m repairs, Ω n is the boundary of the repair area after n repairs. Where m<n, so Ω n The area size is less than Ω m Similarly, 0<m, so Ω m The area size is less than Ω 0 .

[0122] Image block Ψ P0 ,Ψ P1 and P2 The repair area boundaries Ω are 0 ,Ω m and Ω n The image block to be repaired is centered on the target pixel point on the image block to be repaired. When repairing the same image to be repaired, P0 ,Ψ P1 and P2 Same size.

[0123] Target image block search area W 0 , W 1 and W 2 They are image blocks Ψ P0 ,Ψ P1 and P2 The corresponding target image block search area. Image block Ψ 0 ,Ψ 1 and 2 They are image blocks Ψ P0 ,Ψ P1 and P2 The corresponding target image block. Among them, the image block Ψ 0 and 1 Located in area W 0 and W1 In the image block Ψ 2 Located outside area W2.

[0124] Region Ω' is the region that has been repaired when performing the mth image repair. Region Ω" and region Ω' are the regions that have been repaired when performing the nth repair.

[0125] In one embodiment, if Figure 5b As shown in the figure, when performing the mth restoration, ΨP1 is the current image block to be restored, W1 is the current target image block search area, and at this time W1 has a credible area, and Ψ1 is the target image block with the greatest similarity to ΨP1 among the target image blocks in the credible area. Ψ1 is used as the target image block of ΨP1, ΨP1 is restored, and the image block Ψ1 is recorded.

[0126] In one implementation, the recorded target image block is a plurality of target image blocks. Figure 5b Both the target image blocks Ψ0 and Ψ1 in will be recorded.

[0127] In one embodiment, if Figure 5b As shown, when the nth repair is performed, Ψ P2 is the current image block to be repaired, W 2 is the search area of ​​the current target image block, at this time W 2 There is no trusted region, Ψ 2 is the target image block with Ψ P2 The target image block with the largest similarity. 2 As a P2 The target image block is P2 Make repairs.

[0128] The image restoration method provided by the disclosed embodiment repairs the image area to be restored by replacing the image block to be restored with the target block multiple times. The restoration sequence from outside to inside, from bright to dark, and from high gradient to low gradient in the image restoration process is determined by determining the weight of the center point of the image block to be restored, so that the restored image has higher clarity and higher rationality. The pixel points of the image block to be restored are replaced by the pixel points of the target image block to realize the restoration method based on sample block matching, so that the image texture continuity is better. When the restoration is within the restoration area, the target image block is determined in the area outside the restoration boundary, so that the restoration image texture is more reasonable, and the restoration effect will not be reduced due to the iteration of the restoration information.

[0129] Figure 6 The figure is a flowchart of an image restoration method according to an exemplary embodiment.

[0130] In step S61, the boundary of the repair area of ​​the image to be repaired is determined.

[0131] In the disclosed embodiment, the boundary of the repair area of ​​the image to be repaired serves as the starting position of the above-mentioned image repair process and also serves as a condition for performing the above-mentioned image repair process.

[0132] In step S62, if it is determined that there is no restoration area boundary in the image to be restored, edge filtering is performed on the restoration area consisting of the restored image blocks.

[0133] In the disclosed embodiment, if it is determined that there is no repair area boundary in the image to be repaired, the repair of the entire repair area is completed. The repair area composed of the repaired image blocks is edge filtered to eliminate the discontinuity of the repaired image. The repaired image and the image to be repaired are then fused. In one implementation, the repaired area can be filtered after a morphological operation, so that the edge of the repaired area is smoothed.

[0134] Figure 7 The figure is a flowchart of an image restoration method according to an exemplary embodiment.

[0135] like Figure 7 As shown, after the image to be repaired is obtained, image repair will be started on the image to be repaired, and the repaired image will be output after the image repair is completed.

[0136] In one embodiment, the image to be repaired is repaired, the image to be repaired and the area to be repaired are determined, and the image to be repaired is segmented into a background image to obtain a background image after background segmentation. The repair area boundary is determined according to the area to be repaired. It is determined whether there is a repair area boundary in the image to be repaired. When there is a repair area boundary in the image to be repaired, the pixel points on the repair area boundary are prioritized and the pixel points with the highest priority are determined as the target pixel points. The current image block to be repaired is determined with the target pixel point as the center point. The target image block is searched in the background image to determine the target image block corresponding to the current image block to be repaired. Among them, the target image block search area is the valid information area in the background image. The pixel points in the image block to be repaired are filled and replaced according to the pixel points in the target image block to repair the current image block to be repaired. After repairing the current image block to be repaired, the area to be repaired is updated and the repair area boundary is re-determined, and it is re-determined whether there is a repair area boundary. When it is determined that there is still a repair area boundary, the above image repair process is repeated to update and determine the repair area boundary. After re-determining the repair area boundary for multiple times, when it is determined that the repair area boundary does not exist in the image to be repaired, the image to be repaired is edge filtered to obtain a repaired image, and the repaired image is output.

[0137] In one implementation, the image to be repaired is repaired initially, and if it is determined that the image to be repaired does not have a boundary of a region to be repaired, that is, there is no region to be repaired, the image repair process in the above embodiment is not performed, and the image is directly output.

[0138] In one implementation, after filtering the repaired area, the repaired image and the image to be repaired can be fused using the following formula:

[0139] Dst=(1-maskValue)*Src1+maskValue*Src2

[0140] Src1 and Src2 are the images before and after repair, Dst is the output image, and maskValue is the repaired area.

[0141] The image restoration method in the disclosed embodiment can restore the image to be restored after image fusion. The process of restoring the image to be restored after image fusion is described below.

[0142] Figure 8 The figure is a schematic diagram of a process of repairing an image after image fusion according to an exemplary embodiment.

[0143] like Figure 8 As shown, the image restoration mask is the missing information area, that is, the area to be restored. Among them, the restored image in the Bayer image restoration is a high dynamic range raw (image storage format corresponding to the Bayer image) image with the image restoration mask, that is, a high dynamic range Bayer image with the area to be restored.

[0144] In one embodiment, if Figure 8As shown in , when the user takes a photo, the camera can obtain raw images with different exposures through photoelectric conversion imaging. After image registration and synthesis, multiple raw images with different exposures can be fused into a high dynamic range raw image. At this time, due to the existence of motion, the high dynamic range raw image may be missing information, that is, there is a corresponding image restoration mask. The image restoration mask is the area to be restored. After determining that the high dynamic raw image and the corresponding image restoration mask are obtained, the Bayer image restoration is performed on the high dynamic raw image. That is, through the image restoration method in the disclosed embodiment, the high dynamic raw image is background segmented to obtain the background image of the high dynamic raw image. And the point with the highest priority is determined as the target pixel point on the restoration area boundary of the image restoration mask, and the current image block to be restored is determined with the target pixel point as the center. In the credible area (the area outside the image restoration mask) in the background image of the high dynamic raw image, the target image block corresponding to the current image block to be restored is determined. The current image block to be restored is restored according to the target image block, and the pixels in the current image block to be restored are restored and replaced. After repairing the current image block to be repaired, update the repair area boundary (the boundary of the image repair mask). Repair the current image block to be repaired multiple times and update the repair area boundary until all image repair masks are repaired. After all image repair masks are repaired, a complete Bayer image is obtained. The complete Bayer image is then subjected to a series of post-processing to obtain an RGB image, and the image output is written into the album for the user.

[0145] In one implementation, the post-processing may be performed in an ISP (Image Signal Processing) manner.

[0146] In one implementation, a user can use the image restoration method provided in the disclosed embodiment to restore an RGB image, a YUV (a color encoding method) image, or other type of image that has an invalid information region. By restoring the invalid information region in the image, a restored image is obtained.

[0147] In one embodiment, a user can use the image restoration method provided in the disclosed embodiment to restore an image with invalid information areas due to human image modification, stored image damage or other reasons. By restoring the invalid information areas in the image, a restored image is obtained. Fig. 9 The figure is a schematic diagram showing a comparison of image restoration effects according to an exemplary embodiment.

[0148] like Fig. 9As shown, the image restoration method in the disclosed embodiment restores the image in the image, and there is no obvious restoration trace at the junction transition part of the girl's hand and the cloud in the image, the texture continuity of the image is better and more natural, and the cloud is restored, and the restoration effect is more reasonable. However, the image restoration method based on the connected domain boundary is not as good as the image restoration method provided in the disclosed embodiment in terms of texture continuity and rationality, and fails to effectively process the junction transition part of the girl's hand and the cloud, with obvious restoration traces, and the cloud is not restored.

[0149] Based on the same concept, the embodiment of the present disclosure also provides an image restoration device.

[0150] It is understandable that in order to realize the above functions, the image restoration device provided by the embodiment of the present disclosure includes hardware structures and / or software modules corresponding to the execution of each function. In combination with the units and algorithm steps of each example disclosed in the embodiment of the present disclosure, the embodiment of the present disclosure can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art may use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the technical solution of the embodiment of the present disclosure.

[0151] Fig.10 is a block diagram of an image restoration device according to an exemplary embodiment. Fig.10 The device includes a detection module 101 and a processing module 102.

[0152] The detection module 101 is used to determine the boundary of the repair area of ​​the image to be repaired;

[0153] The processing module 102 is used to determine a target pixel point on the boundary of the repair area, where the target pixel point is the first pixel point to be repaired on the boundary of the repair area; determine an image block to be repaired with the target pixel point as the center point, and determine a target image block, where the target image block is an image block in the image to be repaired that has the greatest similarity with the image block to be repaired and has the same size; replace the pixels in the image block to be repaired with the pixels in the target image block to obtain a repaired image block; based on the repaired image block, update the boundary of the area to be repaired; use the updated boundary of the area to be repaired as the boundary of the repair area, and repeat the above image repair method process until the boundary of the repair area does not exist in the image to be repaired.

[0154] In one implementation provided by an embodiment of the present disclosure, the processing module 102 determines the target pixel point on the boundary of the repair area in the following manner: determine the brightness weight, gradient weight and effective information weight of the pixel point on the boundary of the repair area, wherein the larger the brightness channel value, the higher the corresponding brightness weight, the larger the gradient, the higher the corresponding gradient weight, and the more effective pixels around the pixel point, the higher the effective information weight; determine the priority of all pixel points on the boundary of the repair area according to the brightness weight, gradient weight and effective information weight of all pixel points on the boundary of the repair area; determine the pixel point on the boundary of the repair area with the highest priority as the target pixel point.

[0155] In an implementation manner provided by an embodiment of the present disclosure, the processing module 102 determines the target image block in the following manner: determining a target image block search area in the image to be repaired, wherein the central pixel point of the target image block search area is the central point of the image block to be repaired, and the size of the target image block search area is larger than the area size of the image block to be repaired; determining the target image block in the target image block search area.

[0156] In one implementation provided by an embodiment of the present disclosure, the processing module 102 determines the target image block search area in the image to be repaired in the following manner: based on the brightness value of each pixel in the image to be repaired, determine the background image of the image to be repaired; and from the background image, determine the target image block search area.

[0157] In an implementation manner provided by an embodiment of the present disclosure, the processing module 102 is further used to: record a first target image block corresponding to a first repaired image block, the first target image block including pixels located in a credible area, the credible area being an image area outside a boundary of a repair area in the image to be repaired; and determine a target image block in a target image block search area, including: if all the determined target image blocks are located in a non-credible area, determining the first target image block as the target image block of the current image block to be repaired.

[0158] In an implementation provided by an embodiment of the present disclosure, the processing module 102 is further configured to: if it is determined that there is no restoration area boundary in the image to be restored, perform edge filtering on the restoration area composed of the restored image blocks.

[0159] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0160] Fig.112 is a block diagram of an apparatus 200 for image restoration according to an exemplary embodiment. For example, the apparatus 200 may be a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, etc.

[0161] Reference Fig.11 , the device 200 may include one or more of the following components: a processing component 202 , a memory 204 , a power component 206 , a multimedia component 208 , an audio component 210 , an input / output (I / O) interface 212 , a sensor component 214 , and a communication component 216 .

[0162] The processing component 202 generally controls the overall operation of the device 200, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 202 may include one or more processors 220 to execute instructions to perform all or part of the steps of the above-described method. In addition, the processing component 202 may include one or more modules to facilitate interaction between the processing component 202 and other components. For example, the processing component 202 may include a multimedia module to facilitate interaction between the multimedia component 208 and the processing component 202.

[0163] The memory 204 is configured to store various types of data to support operations on the device 200. Examples of such data include instructions for any application or method operating on the device 200, contact data, phone book data, messages, pictures, videos, etc. The memory 204 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0164] The power component 206 provides power to the various components of the device 200. The power component 206 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power for the device 200.

[0165] The multimedia component 208 includes a screen that provides an output interface between the device 200 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 208 includes a front camera and / or a rear camera. When the device 200 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera may receive external multimedia data. Each front camera and rear camera may be a fixed optical lens system or have a focal length and optical zoom capability.

[0166] The audio component 210 is configured to output and / or input audio signals. For example, the audio component 210 includes a microphone (MIC), and when the device 200 is in an operation mode, such as a call mode, a recording mode, and a speech recognition mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in the memory 204 or sent via the communication component 216. In some embodiments, the audio component 210 also includes a speaker for outputting audio signals.

[0167] I / O interface 212 provides an interface between processing component 202 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include but are not limited to: a home button, a volume button, a start button, and a lock button.

[0168] The sensor assembly 214 includes one or more sensors for providing various aspects of the status assessment of the device 200. For example, the sensor assembly 214 can detect the open / closed state of the device 200, the relative positioning of components, such as the display and keypad of the device 200, the sensor assembly 214 can also detect the position change of the device 200 or a component of the device 200, the presence or absence of user contact with the device 200, the orientation or acceleration / deceleration of the device 200 and the temperature change of the device 200. The sensor assembly 214 may include a proximity sensor configured to detect the presence of a nearby object without any physical contact. The sensor assembly 214 may also include an optical sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 214 may also include an accelerometer, a gyroscope sensor, a magnetic sensor, a pressure sensor or a temperature sensor.

[0169] The communication component 216 is configured to facilitate wired or wireless communication between the device 200 and other devices. The device 200 can access a wireless network based on a communication standard, such as WiFi, 2G or 3G, or a combination thereof. In an exemplary embodiment, the communication component 216 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 216 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0170] In an exemplary embodiment, the apparatus 200 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors or other electronic components to perform the above method.

[0171] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 204 including instructions, and the instructions can be executed by the processor 220 of the device 200 to perform the above method. For example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0172] It is to be understood that in the present disclosure, "plurality" refers to two or more than two, and other quantifiers are similar. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the associated objects before and after are in an "or" relationship. The singular forms "a", "the" and "the" are also intended to include plural forms, unless the context clearly indicates other meanings.

[0173] It is further understood that the terms "first", "second", etc. are used to describe various information, but such information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other, and do not indicate a specific order or degree of importance. In fact, the expressions "first", "second", etc. can be used interchangeably. For example, without departing from the scope of the present disclosure, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. It is further understood that, unless otherwise specified, "connection" includes a direct connection between the two without the presence of other components, and also includes an indirect connection between the two with other elements.

[0174] It can be further understood that, unless otherwise specified, “connection” includes a direct connection without other components between the two, and also includes an indirect connection with other components between the two.

[0175] It is further understood that, although the operations are described in a specific order in the drawings in the embodiments of the present disclosure, it should not be understood as requiring the operations to be performed in the specific order shown or in a serial order, or requiring the execution of all the operations shown to obtain the desired results. In certain environments, multitasking and parallel processing may be advantageous.

[0176] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. This application is intended to cover any modifications, uses or adaptations of the present disclosure, which follow the general principles of the present disclosure and include common knowledge or customary technical means in the art that are not disclosed in the present disclosure.

[0177] It should be understood that the present disclosure is not limited to the precise structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the scope of the appended claims.

Claims

1. An image restoration method, characterized in that: include: Determine the restoration area boundary of the image to be restored; Determine a target pixel point on the boundary of the repair area, where the target pixel point is the first pixel point to be repaired on the boundary of the repair area; Determine an image block to be repaired with the target pixel point as the center point, and determine a target image block, wherein the target image block is an image block in the image to be repaired that has the greatest similarity with the image block to be repaired and has the same size; Replacing the pixels in the image block to be repaired with the pixels in the target image block to obtain a repaired image block; Based on the repaired image block, updating the boundary of the area to be repaired; The updated boundary of the region to be repaired is used as the boundary of the repair region, and the above image repair process is repeatedly performed until the boundary of the repair region does not exist in the image to be repaired.

2. The image restoration method according to claim 1, characterized in that: The step of determining a target pixel point on a boundary of the repair area includes: Determine the brightness weight, gradient weight and effective information weight of the pixel points on the boundary of the repair area, wherein the larger the brightness channel value, the higher the corresponding brightness weight, the larger the gradient, the higher the corresponding gradient weight, and the more effective pixels around the pixel point, the higher the effective information weight; Determining the priority of all pixel points on the boundary of the repair area according to the brightness weight of all pixel points on the boundary of the repair area, the gradient weight and the effective information weight; A pixel point on the boundary of the repair area with the highest priority is determined as the target pixel point.

3. The image restoration method according to claim 1, characterized in that: The determining of the target image block comprises: Determine a target image block search area in the image to be repaired, wherein a central pixel point of the target image block search area is a central point of the image block to be repaired, and a size of the target image block search area is larger than a size of the image block to be repaired; In the target image block search area, the target image block is determined.

4. The image restoration method according to claim 3, characterized in that: The step of determining a target image block search area in the image to be repaired comprises: Determining a background image of the image to be restored based on the brightness value of each pixel in the image to be restored; The target image block search area is determined from the background image.

5. The image restoration method according to claim 3 or 4, characterized in that: The method further comprises: Recording a first target image block corresponding to a first repaired image block, wherein the first target image block includes pixels located in a credible area, and the credible area is an image area in the image to be repaired that is outside a boundary of the repair area; Determining the target image block in the target image block search area includes: If all the determined target image blocks are located in the untrusted area, the first target image block is determined as the target image block of the current image block to be repaired.

6. The image restoration method according to claim 1, characterized in that: The method further comprises: If it is determined that the restoration area boundary does not exist in the image to be restored, edge filtering is performed on the restoration area composed of the restored image blocks.

7. An image restoration device, characterized in that: include: A detection module, used to determine the boundary of the repair area of ​​the image to be repaired; A processing module is used to determine a target pixel point on the boundary of the repair area, wherein the target pixel point is the first pixel point to be repaired on the boundary of the repair area; determine an image block to be repaired with the target pixel point as the center point, and determine a target image block, wherein the target image block is an image block in the image to be repaired that has the greatest similarity with the image block to be repaired and has the same size; The pixels in the image block to be repaired are replaced with the pixels in the target image block to obtain a repaired image block; based on the repaired image block, the boundary of the area to be repaired is updated; the updated boundary of the area to be repaired is used as the boundary of the repair area, and the above image repair method process is repeatedly executed until the boundary of the repair area does not exist in the image to be repaired.

8. The image restoration device according to claim 7, characterized in that: The processing module determines the target pixel point on the boundary of the repair area in the following manner: Determine the brightness weight, gradient weight and effective information weight of the pixel points on the boundary of the repair area, wherein the larger the brightness channel value, the higher the corresponding brightness weight, the larger the gradient, the higher the corresponding gradient weight, and the more effective pixels around the pixel point, the higher the effective information weight; Determining the priority of all pixel points on the boundary of the repair area according to the brightness weight of all pixel points on the boundary of the repair area, the gradient weight and the effective information weight; A pixel point on the boundary of the repair area with the highest priority is determined as the target pixel point.

9. The image restoration device according to claim 7, characterized in that: The processing module determines the target image block in the following manner: Determine a target image block search area in the image to be repaired, wherein a central pixel point of the target image block search area is a central point of the image block to be repaired, and a size of the target image block search area is larger than a size of the image block to be repaired; In the target image block search area, the target image block is determined.

10. The image restoration device according to claim 9, characterized in that: The processing module determines the target image block search area in the image to be repaired in the following manner: Determining a background image of the image to be restored based on the brightness value of each pixel in the image to be restored; The target image block search area is determined from the background image.

11. The image restoration device according to claim 9 or 10, characterized in that: The processing module is also used for: Recording a first target image block corresponding to a first repaired image block, wherein the first target image block includes pixels located in a credible area, and the credible area is an image area in the image to be repaired that is outside a boundary of the repair area; In the target image block search area, the processing module determines the target image block in the following manner: If all the determined target image blocks are located in the untrusted area, the first target image block is determined as the target image block of the current image block to be repaired.

12. The image restoration device according to claim 7, characterized in that: The processing module is also used for: If it is determined that the restoration area boundary does not exist in the image to be restored, edge filtering is performed on the restoration area composed of the restored image blocks.

13. An image restoration device, characterized in that: include: processor; a memory for storing processor-executable instructions; Wherein, the processor is configured to: execute the image restoration method described in any one of claims 1 to 6.

14. A storage medium, characterized in that: The storage medium stores instructions, and when the instructions in the storage medium are executed by a processor of the terminal, the terminal is enabled to execute the image restoration method described in any one of claims 1 to 6.

Images