Image restoration method and device, equipment and storage medium
By dividing the image into blocks and adopting image enhancement and mask fusion strategies, the problems of low mask efficiency and visual quality degradation in the prior art are solved, the image repair efficiency and quality are improved, and the color target box in the degraded image is achieved Accurate removal.
Patent Information
- Application Number
- CN202410961710.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-05-15
- Publication Date
- 2024-11-08
AI Technical Summary
The prior art has problems in image repair with low mask efficiency, poor accuracy and inability to effectively deal with visual quality degradation, resulting in low repair efficiency and quality.
By dividing the image into image blocks, extracting the area to be repaired in the target box to generate a binarized mask image, performing image enhancement and downsampling, combining the mask fusion strategy to generate repair image blocks, and finally stitching into a natural transitional repair image.
Improve image repair efficiency and quality, remove artifacts and noise, reduce repair area to retain the effective area of the original image, and achieve accurate removal of colored target boxes in degraded images.
Smart Images

Figure CN118918045A_ABST
Abstract
Description
[0001] This case is a divisional application based on the invention patent with application date of May 15, 2024, application number 202410598181.6, and name “Image restoration method, device, equipment and storage medium” as the parent case. Technical Field
[0002] The present invention relates to the technical field of image restoration, and in particular to an image restoration method, device, equipment and storage medium. Background Art
[0003] Existing monitoring systems are often used in conjunction with intelligent analysis systems. For abnormal images or videos, they are saved and an abnormal target detection box (usually a red rectangle) is superimposed. Therefore, the saved images or videos have local interference and occlusion, which is not conducive to direct use in subsequent deep learning model training, nor is it conducive to big data statistical analysis. Considering the scarcity and application value of data, an image restoration method that removes the colored target box is urgently needed.
[0004] Limited by the shooting scene and image quality, people's demand for image restoration and image editing is increasing. With the rapid development of AIGC, some AI image restoration methods and editing tools based on deep learning have emerged. Compared with professional tools such as PS, this type of AI tool shows good ease of use and amazing restoration effects.
[0005] Traditional image restoration tools, such as the stamp tool in Photoshop, fill and repair local areas in a structural propagation manner based on the principle of image self-similarity.
[0006] Generally, AI repair tools consist of a front-end page and a back-end model inference framework. Users upload images to the front-end page, use a brush to draw the target area to be repaired or removed, and generate a mask corresponding to the original image position. The back-end inference model receives both the original image and the mask as input, and outputs the repaired image end-to-end.
[0007] To improve the repair effect, the most advanced AI repair tool introduces the large-scale language model LLM and the image segmentation model SAM, allowing users to roughly prompt the repair content and expected goals through text input to guide the deep learning model to complete the repair. The SAM segmentation model generates the segmented area of the target object according to the user's click position, and then assists in generating the mask layer.
[0008] The existing technical solutions can better repair the specified mask area, but in practical applications, there are still limitations in terms of mask efficiency, mask accuracy and repair of images with degraded visual quality. For example:
[0009] (1) Manual or semi-automatic mask extraction is inefficient: Most existing technical methods require users to manually participate in the workflow in actual application scenarios, which is not conducive to large-scale automated deployment and results in low repair efficiency.
[0010] (2) Low accuracy of masked areas: The repaired areas based on LLM and SAM hints are not accurate enough and may be missed. The areas marked manually have poor consistency and large errors. The mask may cover key information and cause repair failure.
[0011] (3) Unable to cope with visual quality degradation: Image restoration methods are extremely dependent on the quality of neighborhood information of the area to be restored. Due to image visual quality degradation, such as JPEG lossy compression, artifacts, noise, low resolution, etc., the neighborhood signal-to-noise ratio is reduced, which leads to image restoration failure or restoration errors. Summary of the invention
[0012] The technical problem to be solved by the present invention is to provide an image restoration method, device, equipment and storage medium, which can remove the color target frame existing in the degraded image and improve the image restoration efficiency and restoration quality.
[0013] In a first aspect, the present invention provides an image restoration method, comprising:
[0014] Extract the area to be repaired where the target frame exists in the image to be repaired, and generate a binary mask image;
[0015] According to a preset block size, the image to be repaired and the mask image are respectively cropped to obtain a plurality of image blocks and mask blocks, wherein each image block corresponds to each mask block one by one;
[0016] The mask block containing the area to be repaired is used as a valid mask block, and the image block corresponding to the valid mask block is enhanced to obtain a super-resolution image block upsampled N times, and the super-resolution image block is downsampled 1 / N times to obtain an enhanced image block, where N is a preset multiple;
[0017] According to the enhanced image block and the effective mask block, a repair content is generated for the repair area to obtain a repaired image block;
[0018] According to the effective mask block, the image to be repaired and the repair image block are merged to obtain a merged image block, and the image block corresponding to the effective mask block is replaced with the merged image block;
[0019] Based on each image block, a final output image is generated.
[0020] In a second aspect, the present invention further provides an image restoration device, comprising:
[0021] The first generation module is used to extract the area to be repaired where the target frame exists in the image to be repaired, and generate a binary mask image;
[0022] A cropping module, used for cropping the image to be repaired and the mask image according to a preset block size, to obtain a plurality of image blocks and mask blocks, wherein each image block corresponds to each mask block one by one;
[0023] an enhancement module, configured to take the mask block containing the area to be repaired as a valid mask block, perform image enhancement on the image block corresponding to the valid mask block to obtain a super-resolution image block upsampled N times, and perform 1 / N times downsampling on the super-resolution image block to obtain an enhanced image block, wherein N is a preset multiple;
[0024] A repair module, used for generating repair content for the area to be repaired according to the enhanced image block and the effective mask block, to obtain a repaired image block;
[0025] A fusion module, configured to fuse the image to be repaired and the repair image block according to the effective mask block to obtain a fused image block, and replace the image block corresponding to the effective mask block with the fused image block;
[0026] The second generating module is used to generate a final output image according to each image block.
[0027] In a third aspect, the present invention further provides an electronic device, the electronic device comprising:
[0028] one or more processors;
[0029] A storage device for storing one or more programs;
[0030] When the one or more programs are executed by the one or more processors, the one or more processors implement the image restoration method provided in the first aspect.
[0031] In a fourth aspect, the present invention further provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the image restoration method provided in the first aspect.
[0032] The beneficial effects of the present invention are as follows: by dividing an image into image blocks, only the image blocks with target frames are subsequently repaired, thereby improving the efficiency of image repair; by performing image enhancement, artifacts and noise can be removed to improve image quality; by performing downsampling, artifacts and noise can be further compressed and the image block size can be restored; by adopting a mask-based image fusion strategy, only the repair area in the repaired image block is retained, and by the fusion strategy, the subsequent splicing of image blocks produces obvious transition differences at the splicing edge, thereby generating a naturally transitioned repaired image, and minimizing the repair area as much as possible to retain the effective area of the original image. The present invention can remove the color target frame existing in the degraded image in the actual scene, and improve the repair efficiency and repair quality of the image or video data. BRIEF DESCRIPTION OF THE DRAWINGS
[0033] Figure 1 A flowchart of an image restoration method provided by the present invention;
[0034] Figure 2 This is a flow chart of an image restoration method according to the first embodiment of the present invention;
[0035] Figure 3 This is a flow chart of an image restoration method according to Embodiment 1 of the present invention;
[0036] Figure 4 is a schematic diagram of an image to be restored according to the first embodiment of the present invention;
[0037] Figure 5 is a schematic diagram of a mask image according to the first embodiment of the present invention;
[0038] Figure 6 is a schematic diagram of an image block and a mask block according to a first embodiment of the present invention;
[0039] Figure 7 Schematic diagram of an enhanced image block and a repaired image block according to the first embodiment of the present invention;
[0040] Figure 8 The basis of the first embodiment of the present invention Figure 6 Schematic diagram of the weights generated by the mask block in (b) and the fused image block;
[0041] Fig. 9 is a schematic diagram of a final output image according to the first embodiment of the present invention;
[0042] Fig.10 This is a flow chart of a target frame mask extraction method according to a second embodiment of the present invention;
[0043] Fig.11 It is a flowchart of a target frame mask extraction method according to a second embodiment of the present invention;
[0044] Fig.12A schematic structural diagram of an image restoration device provided by the present invention;
[0045] Fig.13 The present invention provides a schematic structural diagram of an electronic device. DETAILED DESCRIPTION
[0046] The present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the present invention, rather than to limit the present invention. It should also be noted that, for ease of description, only parts related to the present invention, rather than all structures, are shown in the accompanying drawings.
[0047] Before discussing the exemplary embodiments in more detail, it should be mentioned that some exemplary embodiments are described as processes or methods depicted as flow charts. Although the flow charts describe the steps as sequential processes, many of the steps therein can be implemented in parallel, concurrently or simultaneously. In addition, the order of the steps can be rearranged. The process can be terminated when its operation is completed, but can also have additional steps not included in the accompanying drawings. The process can correspond to a method, function, procedure, subroutine, sub-computer program, etc.
[0048] In addition, the terms "first", "second", etc. can be used in this article to describe various directions, actions, steps or elements, but these directions, actions, steps or elements are not limited by these terms. These terms are only used to distinguish a first direction, action, step or element from another direction, action, step or element. For example, without departing from the scope of the present application, the first information can be referred to as the second information, and similarly, the second information can be referred to as the first information. Both the first information and the second information are information, but they are not the same information. The terms "first", "second", etc. cannot be understood as indicating or implying relative importance or implicitly indicating the number of indicated technical features. Thus, the features defined as "first" and "second" can expressly or implicitly include one or more of the features. In the description of the present invention, the meaning of "multiple" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.
[0049] like Figure 1 As shown, an image restoration method comprises:
[0050] Extract the area to be repaired where the target frame exists in the image to be repaired, and generate a binary mask image;
[0051] According to a preset block size, the image to be repaired and the mask image are respectively cropped to obtain a plurality of image blocks and mask blocks, wherein each image block corresponds to each mask block one by one;
[0052] The mask block containing the area to be repaired is used as a valid mask block, and the image block corresponding to the valid mask block is enhanced to obtain a super-resolution image block upsampled N times, and the super-resolution image block is downsampled 1 / N times to obtain an enhanced image block, where N is a preset multiple;
[0053] According to the enhanced image block and the effective mask block, a repair content is generated for the repair area to obtain a repaired image block;
[0054] According to the effective mask block, the image to be repaired and the repair image block are merged to obtain a merged image block, and the image block corresponding to the effective mask block is replaced with the merged image block;
[0055] Based on each image block, a final output image is generated.
[0056] From the above description, it can be seen that the color target frame existing in the degraded image in the actual scene can be removed, thereby improving the restoration efficiency and restoration quality of the image or video data.
[0057] In an optional embodiment, the step of extracting the to-be-repaired area having the target frame in the to-be-repaired image and generating a binary mask image is specifically as follows:
[0058] Performing color space conversion on the image to be repaired to obtain an RGB color gamut image and an HSV color gamut image, and performing color threshold segmentation on the RGB color gamut image and the HSV color gamut image according to the color of the target frame to be removed, to obtain a first binary image corresponding to the RGB color gamut and a second binary image corresponding to the HSV color gamut;
[0059] According to the unit color vector of the target frame color, two basis vectors with a large angle difference with the target frame color are determined from the basis vectors of the RGB color domain, and Schmidt orthogonal decomposition is performed to obtain a unit orthogonal basis, and according to the unit orthogonal basis, the RGB color domain image is coordinate transformed to obtain a color representation image based on the target frame color;
[0060] According to the color representation image, the minimum channel difference is calculated to obtain a difference image;
[0061] De-noising the difference image by using an edge-preserving filtering algorithm, and performing edge detection on the de-noised difference image to obtain a third binary image;
[0062] performing closing operation and opening operation of morphological filtering on the first binarized image, the second binarized image and the third binarized image respectively;
[0063] According to the geometric properties of the target frame, respectively, geometric feature filtering is performed on the first binary image, the second binary image, and the third binary image after morphological filtering;
[0064] Performing gated fusion on the first binary image, the second binary image, and the third binary image after geometric feature filtering to obtain a binary fused image;
[0065] Morphological filtering is performed on the binary fusion image to generate a mask image corresponding to the area to be repaired.
[0066] From the above description, it can be seen that according to the geometric features, color differences and spatial gradients of the target frame, it is possible to robustly, accurately and efficiently generate a mask map of the area to be repaired under complex background interference and similar color interference.
[0067] In an optional embodiment, according to the color of the target frame to be removed, the RGB color gamut image and the HSV color gamut image are respectively subjected to color threshold segmentation to obtain a first binary image corresponding to the RGB color gamut and a second binary image corresponding to the HSV color gamut, specifically:
[0068] Determine a first segmentation threshold according to the color of the target frame to be removed, and perform binarization processing on the RGB color gamut image according to the first segmentation threshold to obtain a first binarized image corresponding to the RGB color gamut;
[0069] A second segmentation threshold is determined according to the color and saturation of the target frame to be removed, and the HSV color gamut image is binarized according to the second segmentation threshold to obtain a second binarized image corresponding to the HSV color gamut.
[0070] From the above description, it can be seen that the binarization processing based on the absolute color information of the target frame is implemented.
[0071] In an optional embodiment, according to the unit color vector of the target frame color, two basis vectors with a large angle difference with the target frame color are determined from the basis vectors of the RGB color domain, and Schmidt orthogonal decomposition is performed to obtain a unit orthogonal basis, and according to the unit orthogonal basis, the RGB color domain image is coordinate transformed to obtain a color representation image based on the target frame color, specifically:
[0072] The color vector mean of the target frame color is obtained by sampling, and then normalized to obtain the unit color vector;
[0073] Calculate the cosine similarity between each basis vector of the RGB color domain and the unit color vector respectively, and replace the basis vector corresponding to the minimum cosine similarity with the unit color vector to obtain an orthogonalized basis vector mark;
[0074] According to the unit color vector, the other two basis vectors in the basis vector mark are subjected to Schmidt orthogonal decomposition to obtain a unit orthogonal basis;
[0075] According to the coordinate conversion formula, the RGB color gamut image is converted to obtain a color representation image based on the target frame color. The coordinate conversion formula is [x] c =C -1 [x] rgb , where [x] c represents the coordinates of the color representation image, C represents the unit orthogonal basis, [x] rgb Represents the coordinates of an RGB color space image.
[0076] From the above description, it can be seen that the binary segmentation of the target frame of any color is achieved based on the binarization processing of the target frame relative to the color gradient.
[0077] In an optional embodiment, according to the geometric attributes of the target frame, geometric feature filtering is performed on the first binarized image, the second binarized image, and the third binarized image after morphological filtering, respectively, specifically:
[0078] Obtaining external closed contours in a binary image, and respectively calculating the contour convex hull area and the internal solidity rate of each external closed contour, wherein the binary image includes a first binary image, a second binary image, and a third binary image;
[0079] If the convex hull area of an external closed contour is less than or equal to a preset area threshold, the area where the external closed contour is located is set as a non-repairing area;
[0080] If the convex hull area of an external closed contour is greater than a preset area threshold and the internal solid ratio of the external closed contour is greater than a preset solid ratio threshold, polygon fitting is performed on the external closed contour, and the number of vertices of the polygon obtained by fitting and the average of the absolute values of the cosines of its internal angles are counted;
[0081] If the number of vertices of the polygon is different from the number of vertices of the target frame and the mean of the absolute values of the cosine is greater than a preset mean threshold, the area where the external closed contour is located is set as a non-repairing area.
[0082] From the above description, it can be seen that by performing geometric feature filtering, the wrong segmentation area can be eliminated according to the geometric attributes of the target frame.
[0083] In an optional embodiment, before the image to be repaired and the mask image are cropped according to a preset block size to obtain a plurality of image blocks and mask blocks, the method further comprises:
[0084] The edge filling is performed on the image to be repaired and the mask image respectively, and the sizes of the image to be repaired and the mask image after the edge filling are integer multiples of the preset block size.
[0085] It can be seen from the above description that by filling the edges of the image to be repaired and the mask image so that their sizes are integer multiples of the preset block size, the sizes of each image block and each mask block obtained by subsequent cropping are consistent.
[0086] In an optional embodiment, generating a final output image according to each image block is specifically:
[0087] Each image block is spliced to obtain a spliced image, and edge filling pixels in the spliced image are removed according to the size of the image to be repaired to obtain a final output image.
[0088] From the above description, it can be seen that by removing the pixels filled at the edge, the size of the final output image is made consistent with the original image to be repaired.
[0089] In an optional embodiment, generating the repair content for the area to be repaired according to the enhanced image block and the effective mask block to obtain the repaired image block is specifically:
[0090] According to the background information of the enhanced image block and the neighborhood information of the to-be-repaired area in the effective mask block, a restoration content is generated by a preset image restoration algorithm to obtain a restored image block.
[0091] In an optional embodiment, the image to be repaired and the repair image block are fused according to the effective mask block to obtain a fused image block, specifically:
[0092] The effective mask block is normalized, inverted, expanded and blurred in sequence to obtain a weight;
[0093] According to the weighted fusion formula, the image to be repaired and the repaired image block corresponding to the effective mask block are weightedly fused to obtain a fused image block. The weighted fusion formula is F p =I p ·(1-W p )+R p ·W p , where F p represents the fused image block, I p Represents the image to be repaired, R p represents the repaired image block, W p Indicates the weight corresponding to the valid mask block.
[0094] From the above description, it can be seen that the mask-based image fusion strategy is adopted to retain only the repaired area in the repaired image block. The weighted fusion strategy is used to alleviate the edge transition difference, generate a naturally transitioned repaired image, and reduce the repair area as much as possible to retain the effective area of the original image.
[0095] The present invention also provides an image restoration device, comprising:
[0096] The first generation module is used to extract the area to be repaired where the target frame exists in the image to be repaired, and generate a binary mask image;
[0097] A cropping module, used for cropping the image to be repaired and the mask image according to a preset block size, to obtain a plurality of image blocks and mask blocks, wherein each image block corresponds to each mask block one by one;
[0098] an enhancement module, configured to take the mask block containing the area to be repaired as a valid mask block, perform image enhancement on the image block corresponding to the valid mask block to obtain a super-resolution image block upsampled N times, and perform 1 / N times downsampling on the super-resolution image block to obtain an enhanced image block, wherein N is a preset multiple;
[0099] A repair module, used for generating repair content for the area to be repaired according to the enhanced image block and the effective mask block, to obtain a repaired image block;
[0100] A fusion module, configured to fuse the image to be repaired and the repair image block according to the effective mask block to obtain a fused image block, and replace the image block corresponding to the effective mask block with the fused image block;
[0101] The second generating module is used to generate a final output image according to each image block.
[0102] The present invention further provides an electronic device, comprising:
[0103] one or more processors;
[0104] A storage device for storing one or more programs;
[0105] When the one or more programs are executed by the one or more processors, the one or more processors implement the image restoration method as described above.
[0106] The present invention also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the image restoration method as described above is implemented.
[0107] Embodiment 1
[0108] Please refer to Figure 2-9 , Embodiment 1 of the present invention is: an image restoration method, which can be applied to the scenario where a monitoring system is used in combination with an intelligent analysis system, to automatically, efficiently and accurately remove the color target frame in the actual degraded image, and generate and fill in appropriate content based on background information and neighborhood pixels, thereby improving the restoration efficiency and restoration quality of large-scale image or video data.
[0109] like Figure 2 and Figure 3 As shown, the following steps are included:
[0110] S1: Extract the area to be repaired where the target box exists in the image to be repaired, and generate a binary mask image.
[0111] Specifically, read the input image or video frame image as the image to be repaired I, such as Figure 4 As shown, then, the area to be repaired in the target frame in the image to be repaired is extracted to generate a binary mask (ie, mask) image M, as shown in FIG. Figure 5 The specific meaning of the pixel value of the mask image M depends on the actual setting. In this embodiment, a pixel value of 0 indicates that the image needs to be repaired, and a pixel value greater than 0 indicates that the original image is retained without repair, that is, the pixel value of the pixel point in the area to be repaired in the mask image is 0.
[0112] S2: edge-filling the image to be repaired and the mask image respectively, wherein the sizes of the image to be repaired and the mask image after edge-filling are integer multiples of a preset block size.
[0113] Specifically, the surrounding pixels of the image to be repaired I and the mask image M are padded (Padding), and the image size after padding is an integer multiple of the preset block size. In this embodiment, since the subsequent image repair algorithm adopts the MAT (Mask-Aware Transformer) model, and the MAT model is a ViT structure, its input size is an integer multiple of 256 or 512. Therefore, the preset block size is 256 or 512 to take into account the input requirements, operating efficiency and repair effect.
[0114] In order to minimize the impact of filled pixels on subsequent image processing, the image to be repaired I is filled with pixel value 0, and the mask image M is filled with pixel value 255 (indicating that the area is not repaired). That is, the pixel value of the pixel points filled in the image to be repaired I is 0, and the pixel value of the pixel points filled in the mask image M is 255.
[0115] S3: According to the preset block size, the image to be repaired and the mask image are respectively cut to obtain multiple image blocks and mask blocks, and each image block corresponds to each mask block. Among them, there are mask blocks of the area to be repaired and their corresponding image blocks such as Figure 6 As shown in (a) and (b) in .
[0116] In this embodiment, since the image to be restored and the mask image have been padded to an integral multiple of the block size in step S2, the sizes of each image block and each mask block obtained by cutting are consistent.
[0117] Furthermore, according to the position of each image block in the image to be repaired and the position of each mask block in the mask image, the row and column coordinate index of each image block and the row and column coordinate index of each mask block are generated, and the image blocks and mask blocks are stored in a list according to the (row, column) coordinate index.
[0118] In another preferred embodiment, the cropping area is automatically adjusted according to the position of the area to be repaired. On the one hand, as many repair areas as possible are covered to improve the efficiency of a single repair; on the other hand, the neighborhood information of the area to be repaired is retained as much as possible to ensure the repair effect.
[0119] S4: traverse each image block, sequentially obtain an image block and its corresponding mask block as the current image block and the current mask block, that is, read the image block I in pairs according to the (row, column) coordinate index. p and mask block M p .
[0120] S5: Determine whether the current mask block has an area to be repaired. If so, that is, the current mask block is a valid mask block, then execute step S6. If not, continue to sequentially obtain the next image block and its corresponding mask block as the current image block and current mask block, that is, execute step S4.
[0121] S6: Perform image enhancement on the current image block to obtain a super-resolution image block that is upsampled N times, and perform 1 / N times downsampling on the super-resolution image block to obtain an enhanced image block, such as Figure 7 As shown in (a) in .
[0122] Specifically, in order to improve the image quality and solve the problem that the image restoration algorithm is difficult to handle the actual degraded image, the Real-ESRGAN blind super-resolution model is used to enhance the current image block I p , remove artifacts and noise, obtain a super-resolution image block that is upsampled N times, and then downsample 1 / N times to further compress the noise and artifacts and restore the image block size to obtain the enhanced image block E of the original resolution p In this embodiment, N is 2 or 4.
[0123] Among them, Real-ESRGAN uses a second-order degradation process to realistically simulate actual image degradation and generates paired simulation data sets based on a large amount of high-quality data. The model trained on this basis covers a wider data distribution and has stronger generalization ability. It can robustly remove image noise and artifacts, enhance edges and colors, and improve image resolution.
[0124] S7: Generate repair content for the repaired area according to the enhanced image block and the current mask block to obtain a repaired image block, such as Figure 7 As shown in (b) in .
[0125] In this embodiment, based on the enhanced image block E p Background information and current mask block M p The surrounding neighborhood information of the area to be repaired in the image is used to generate reasonable repair content. The MAT (Mask-Aware Transformer) model is used as the image repair algorithm. The incomplete enhanced image block E after masking is obtained by element-by-element multiplication. p M =E p ·M p , and M p and E p M The four-channel tensor is used as input, so that the MAT model can avoid the interference of residual information in the area to be repaired, and focus on the area to be repaired indicated by the mask image to obtain a better repaired image block R p .
[0126] The MAT model first uses a 4-layer convolutional network to extract shallow local features as input tokens for the subsequent Transformer block, and quickly downsamples the feature map by 1 / 8 to reduce the amount of subsequent calculations. MAT uses a Transformer block based on a multi-head contextual attention mechanism MCA to optimize the repair effect of large-area masks. MCA can use long-range information based on the non-local prior of the image to create reasonable structures and textures in the missing area. MCA is based on the mask M p The attention relationship is established only in the effective pixel area, and the mask dynamic update strategy is used to gradually expand the area of the effective area to reduce the color deviation and blur of the repaired area. In addition, MAT also provides a style control interface to achieve diversified image repair. In practical applications, in order to ensure the consistency of the repair style, this input is set to the default constant. The features repaired by Transformer are upsampled by another convolutional neural network and reconstructed into a repaired image with the same size as the input. Then, another UNet-like network is used to improve the local details to obtain the final repaired image block R p .
[0127] S8: According to the current mask block, the image to be repaired and the repair image block are fused to obtain a fused image block, and the current image block is replaced with the fused image block.
[0128] Since the repaired image block R pAs a result of the dual effects of image enhancement and image restoration, its visual effect is significantly different from that of the unenhanced area. Therefore, directly replacing and splicing image blocks will produce obvious transition differences at the splicing edges. Therefore, this step adopts a mask-based image fusion strategy, retaining only the repaired area in the repaired image block, and using a weighted fusion strategy to alleviate the edge transition difference, generate a naturally transitioned repaired image, and minimize the repair area as much as possible to retain the effective area of the original image.
[0129] Specifically, according to the current mask block M p Generate weight W p To fuse the image to be repaired I p and the repaired image patch R p First, for the current mask block M p Normalize it so that its representation range is between [0,1] to facilitate subsequent fusion; then, normalize the current mask block M p Negate or invert the current mask block M so that the pixel value is 1 to indicate the repaired area and the pixel value is 0 to indicate the retained area, so as to facilitate subsequent morphological operations and weighted fusion. p Morphological dilation is performed to appropriately expand the repair area and cover the neighboring pixels that may have artifacts. Then, the edge of the dilation result is blurred using a box filter (Box Filter Blur) to smooth the transition area and alleviate edge differences. Finally, according to the weighted fusion formula F p =I p ·(1-W p )+R p ·W p , get the fused image block F p , and use it to fuse the image block F p Replace the current image block I with the corresponding (row, column) coordinate index p .
[0130] Among them, the weight is Figure 8 As shown in (a) in the figure, the fused image block is Figure 8 As shown in (b) in .
[0131] S9: Determine whether all image blocks have been traversed. If so, execute step S10. If not, continue to sequentially obtain the next image block and its corresponding mask block as the current image block and current mask block, that is, execute step S4.
[0132] S10: Generate a final output image based on each image block, such as Fig. 9 shown.
[0133] Specifically, the image blocks are spliced according to the (row, column) coordinate index to obtain a complete spliced image, and the edge filling pixels in the spliced image are removed according to the size of the image to be repaired to obtain the final output image F.
[0134] In this embodiment, each time a mask block with a to-be-repaired area is traversed, the corresponding image block is repaired, that is, this embodiment adopts a block-by-block serial processing method.
[0135] In another embodiment, after all mask blocks with areas to be repaired are determined, these mask blocks and their corresponding image blocks may be aggregated for parallel batch processing, thereby further improving the repair efficiency.
[0136] In view of the high failure rate of existing methods in repairing low-quality degraded images, this embodiment introduces a blind super-resolution model as a pre-processing step, which can remove JPEG compression effects, ringing and overshoot artifacts, potential noise, etc., enhance image quality, improve the repair effect, reduce the failure rate and error rate, and effectively solve the repair failure problems caused by various quality degradations in actual monitoring images. At the same time, in view of the high computational cost of deep learning models for high-resolution image processing in existing methods, this embodiment divides high-resolution images into image blocks and only repairs image blocks with target frames, greatly improving operating efficiency and reducing video memory usage.
[0137] Embodiment 2
[0138] Please refer to Figure 10-11 This embodiment is a specific implementation of step S1 in embodiment 1. Fig.10 and Fig.11 As shown, the following steps are included:
[0139] S101: Performing color space conversion on the image to be repaired to obtain an RGB color gamut image and an HSV color gamut image, and performing color threshold segmentation on the RGB color gamut image and the HSV color gamut image according to the color of the target frame to be removed to obtain a first binary image corresponding to the RGB color gamut and a second binary image corresponding to the HSV color gamut.
[0140] Specifically, the image to be restored I is first converted into the HSV color space and the RGB color space, respectively. Then, according to the known target frame color prior, color threshold segmentation is performed in different color spaces to obtain the first binary image S rgb and the second binarized image S hsv .
[0141] This step is to perform binarization based on the absolute color information of the target frame. When the color of the target frame is known, the traditional color segmentation can efficiently extract the segmentation area of a simple sample (without obvious color interference). To this end, the segmentation threshold can be set directly in the RGB color domain according to the known color of the target frame to perform color segmentation and obtain the first binary image S rgb .
[0142] For samples with similar color interference, the color segmentation threshold can be set according to the saturation in the HSV color domain, because the color of the target frame is always the maximum saturation when drawn. Therefore, the image to be repaired I is converted to the HSV color domain, and the segmentation threshold is set according to the prior information (i.e. the color and saturation of the target frame) to perform color segmentation, and the second binary image S is obtained. hsv .
[0143] In this embodiment, for color segmentation of the RGB color gamut, the segmentation threshold may include an interval upper threshold and an interval lower threshold, that is, a valid interval is determined according to the color of the target frame, such as [a, b]. Since the pixel value is generally represented by a three-dimensional vector, such as (x[0], x[1], x[2]), the segmentation threshold is also three-dimensional, and the valid pixels need to be within the valid interval in each dimension, that is, a[0]≤x[0]≤b[0]&a[1]≤x[1]≤b[1]&a[2]≤x[2]≤b[2]. The pixel value within the valid interval is determined as a valid pixel value, and the pixel value not within the valid interval is determined as an invalid pixel value.
[0144] For example, if the color of the target frame is red, that is, the pixel value is (255, 0, 0), then the effective intervals of R, G, and B three-dimensional separation can be set to [180, 255], [0, 50], and [0, 50], respectively.
[0145] The color segmentation of HSV color gamut is the same as that of RGB color gamut.
[0146] S102: According to the unit color vector of the target frame color, two basis vectors with a large angle difference with the target frame color are determined from the basis vectors of the RGB color domain, and Schmidt orthogonal decomposition is performed to obtain a unit orthogonal basis, and according to the unit orthogonal basis, the RGB color domain image is coordinate transformed to obtain a color representation image based on the target frame color.
[0147] This step is to perform binarization based on the target relative color gradient.
[0148] In practical applications, due to image quality degradation and complex and diverse shooting scenes, the data set to be processed often contains both simple samples and difficult samples (with artifacts, color casts, interference, etc.). Using absolute color thresholds for segmentation cannot take into account both precision and recall, and requires targeted parameter adjustment, which does not meet the original design intention of efficient and automated. To this end, consider introducing relative color information for binary segmentation and filtering based on the target box geometry prior to take into account both segmentation accuracy and recall. Color gradient difference requires that the segmented color is exactly a basis for orthogonal color representation, while the RGB color gamut is only suitable for the segmentation of red, green and blue colors.
[0149] To achieve binary segmentation of any color target frame, first, the three-dimensional representation of the RGB color domain is transformed. Specifically, the actual sample is sampled to obtain the color vector mean of the actual target frame, and then normalized to obtain the unit color vector [V] rgb ∈R 3 .
[0150] Then, for the basis vector of the RGB color space, that is, the 3x3 unit matrix, calculate its basis vector and unit color vector [V] respectively. rgb The cosine similarity of , and the unit color vector [V] rgb Replace the basis vector corresponding to the minimum cosine similarity to obtain the orthogonalized basis vector symbol C'=[V, e1, e2], where e1 and e2 represent the unit color vector [V] rgb The two basis vectors with the largest cosine similarity.
[0151] Next, the unit color vector [V] in C' is labeled with the basis vector rgb As the initial value, perform Schmidt orthogonal decomposition on the other two basis vectors to obtain the unit orthogonal basis C.
[0152] Finally, according to the coordinate transformation formula [x] c =C -1 [x] rgb , for RGB color gamut image I rgb Perform coordinate transformation to obtain the color representation image I based on the target frame prior color V c . Where [x] c Represents color representation image I c The coordinates of [x] rgb Represents RGB color gamut image I rgb The coordinates of .
[0153] Furthermore, if the target frame color itself is one of red, green, and blue, there is no need to perform basis transformation.
[0154] S103: Calculate the minimum channel difference according to the color representation image to obtain a difference image.
[0155] Specifically, the single-channel difference image I diff =min(I c 1 -I c 2 , I c 1 -I c 3 ), where I c 1 ,I c 2 and I c 3 Respectively represent the color representation image I c The 1st, 2nd, and 3rd channels of , the 1st channel corresponds to the basis vector of the target box color vector, and the 2nd and 3rd channels correspond to the other two orthogonal basis vectors.
[0156] S104: De-noising the difference image by using an edge-preserving filtering algorithm, and performing edge detection on the de-noised difference image to obtain a third binary image.
[0157] Specifically, in order to avoid the potential noise of the degraded image affecting the edge extraction, the visible image noise situation uses an edge-preserving filtering algorithm (this embodiment uses a bilateral filtering algorithm) to process the difference image, remove the noise of non-main gradients, and reduce false edge detection. diff The significant gradient in the target box is taken as the edge of the target box, and the denoised difference image I diff Perform edge detection (Canny edge detection algorithm is used in this embodiment) to obtain the third binary image S diff .
[0158] S105: performing closing operation and opening operation of morphological filtering on the first binarized image, the second binarized image and the third binarized image respectively.
[0159] Specifically, for the first binary image S rgb , the second binary image S hsv and the third binarized image S diff , and perform closing and opening operations in morphological filtering in sequence to eliminate internal gaps and noise and connect virtual areas.
[0160] S106: According to the geometric attributes of the target frame, geometric feature filtering is performed on the first binarized image, the second binarized image and the third binarized image after morphological filtering respectively.
[0161] Specifically, it is known that the target box is hollow, and a solid segmentation area greater than the threshold can be achieved according to the solid ratio. First, the external closed contours in the binary image are extracted, and each external closed contour is traversed to calculate the contour convex hull area A and the internal solid ratio r%, where the calculation formula of the internal solid ratio is r% = the area of the area to be repaired with non-zero pixel values in A / A×100%.
[0162] After calculating the contour convex hull area A and the internal solid rate r%, the noise area can be removed according to the geometric prior of the target frame. Specifically, for contours whose contour convex hull area A is less than or equal to the preset area threshold (625 in this embodiment), the area where it is located is set as a non-repairing area to remove tiny areas that cannot be the target frame; for contours whose contour convex hull area A is greater than the preset area threshold and whose internal solid rate r% is less than or equal to the preset solid rate threshold (75% in this embodiment), they are retained; for contours whose contour convex hull area A is greater than the preset area threshold and whose internal solid rate r% is greater than the preset solid rate threshold, further checks are required to avoid erroneously removing moments that are completely filled due to morphological operations. The specific operation is as follows: first, perform contour polygon fitting, and then count the number of vertices of the polygon obtained by fitting and the mean of the absolute value of the cosine of each internal angle. According to the geometric prior of the target frame (generally a rectangle, that is, the number of vertices is 4), the contours with 4 vertices and the mean of the absolute value of the cosine less than or equal to the preset mean threshold (0.1 in this embodiment) are retained; for the contours with not 4 vertices and the mean of the absolute value of the cosine greater than the preset mean threshold, the area where they are located is set as a non-repairing area to eliminate the contour area that obviously does not conform to the shape of the target frame and retain possible overlapping figures of multiple rectangular frames.
[0163] The three binary images after morphological filtering are filtered by geometric features to eliminate the wrong segmentation areas according to the geometric properties of the target box.
[0164] S107: performing gated fusion on the first binary image, the second binary image and the third binary image after geometric feature filtering to obtain a binary fused image.
[0165] Specifically, the first binary image S' after geometric feature filtering rgb , the second binary image S' hsv And the third binary image S' diff Gated fusion is performed to achieve complementary advantages of multiple segmentation and extraction methods. The redundant detection mechanism can improve the recall rate. The binary fusion image S f =S' rgb ∩S' hsv ∩S' diff If one of the binary images cannot provide effective information, the gating mechanism can be used to close the calculation path to improve the calculation efficiency.
[0166] S108: Perform morphological filtering on the binary fusion image to generate a mask image corresponding to the area to be repaired.
[0167] Specifically, in order to avoid the potential artifacts and noise in the neighborhood of the to-be-repaired area from affecting the restoration result (the compression artifacts generated by JPEG generally appear in the 8×8 neighborhood of the edge of the target frame), the binary fusion image S f Morphological filtering (dilation) is performed to cover the interference area and generate a mask image M of the area to be repaired.
[0168] Furthermore, according to actual application needs, the binary pixel values in the mask image can be inverted, that is, the pixel value of the pixel point whose original pixel value is 1 is modified to 0, and the pixel value of the pixel point whose original pixel value is 0 is modified to 1.
[0169] Furthermore, in another optional embodiment, the three binary images after geometric feature filtering may be firstly dilated by morphological filtering, and then the three binary images after dilation processing may be fused.
[0170] In view of the problem that the existing technology requires users to manually participate in the workflow in actual applications, which is not conducive to large-scale automated deployment, this embodiment can realize the automatic extraction of the target frame mask, thereby realizing the full process automated deployment. In view of the problem that the existing technology prompts that the location of the repair area is too rough and there are omissions, this embodiment combines the traditional color threshold segmentation method to improve it, and at the same time uses the target frame color difference, spatial gradient, and geometric features to obtain an accurate, consistent, and robust target frame mask image with a gated post-fusion strategy.
[0171] Embodiment 3
[0172] Please refer to Fig.12 , Embodiment 3 of the present invention is: an image restoration device, which can execute the image restoration method provided by Embodiment 1 of the present invention, and has the corresponding functional modules and beneficial effects of the execution method. The device can be implemented by software / or hardware, and specifically includes:
[0173] The first generating module 201 is used to extract the to-be-repaired area with the target frame in the to-be-repaired image and generate a binary mask image;
[0174] A cropping module 202, used to crop the image to be repaired and the mask image according to a preset block size, to obtain a plurality of image blocks and mask blocks, wherein each image block corresponds to each mask block one by one;
[0175] The enhancement module 203 is used to take the mask block with the area to be repaired as a valid mask block, perform image enhancement on the image block corresponding to the valid mask block to obtain a super-resolution image block upsampled N times, and perform 1 / N times downsampling on the super-resolution image block to obtain an enhanced image block, where N is a preset multiple;
[0176] A repair module 204, configured to generate repair content for the area to be repaired according to the enhanced image block and the effective mask block, and obtain a repaired image block;
[0177] A fusion module 205 is used to fuse the image to be repaired and the repair image block according to the effective mask block to obtain a fused image block, and replace the image block corresponding to the effective mask block with the fused image block;
[0178] The second generating module 206 is used to generate a final output image according to each image block.
[0179] Embodiment 4
[0180] Please refer to Fig.13 , Embodiment 4 of the present invention is: an electronic device, the electronic device comprising:
[0181] One or more processors 301;
[0182] Storage device 302, used to store one or more programs;
[0183] When the one or more programs are executed by the one or more processors 301, the one or more processors 301 implement the various processes in the image restoration method embodiment as described above, and can achieve the same technical effect. To avoid repetition, they are not repeated here.
[0184] Embodiment 5
[0185] Embodiment 5 of the present invention provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the various processes in the above-mentioned image restoration method embodiment are implemented and the same technical effect can be achieved. To avoid repetition, it will not be described here.
[0186] In summary, the image restoration method, device, equipment and storage medium provided by the present invention can robustly and accurately extract the area to be restored in the image to be restored according to the color prior of the target frame, and provide a mask area for subsequent steps, so as to realize automatic processing of large-scale data and improve the restoration efficiency and accuracy; by dividing the image into image blocks, only the image blocks with target frames are subsequently restored, which can improve the image restoration efficiency; by performing image enhancement, artifacts and noise can be removed, and the image quality can be improved; by performing downsampling, artifacts and noise can be further compressed and the image block size can be restored; by adopting a mask-based image fusion strategy, only the restoration area in the restoration image block is retained, and a weighted fusion strategy is used to alleviate the edge transition difference, a restoration image with a natural transition is generated, and the restoration area is minimized as much as possible to retain the effective area of the original image. The present invention can remove the color target frame existing in the degraded image in the actual scene, and improve the restoration efficiency and restoration quality of large-scale image or video data.
[0187] Through the above description of the implementation methods, the technicians in the relevant field can clearly understand that the present invention can be implemented by means of software and necessary general hardware, and of course it can also be implemented by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as a computer floppy disk, a read-only memory (ROM), a random access memory (RAM), a flash memory (FLASH), a hard disk or an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present invention.
[0188] It is worth noting that in the embodiment of the above-mentioned device, the various units and modules included are only divided according to functional logic, but are not limited to the above-mentioned division, as long as the corresponding functions can be achieved; in addition, the specific names of the functional units are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of the present invention.
[0189] The above descriptions are merely embodiments of the present invention and are not intended to limit the patent scope of the present invention. Any equivalent transformations made using the contents of the present invention's specification and drawings, or directly or indirectly applied in related technical fields, are also included in the patent protection scope of the present invention.
Claims
1. An image restoration method, characterized in that: include: Extract the area to be repaired where the target frame exists in the image to be repaired, and generate a binary mask image; According to a preset block size, the image to be repaired and the mask image are respectively cropped to obtain a plurality of image blocks and mask blocks, wherein each image block corresponds to each mask block one by one; The mask block containing the area to be repaired is used as a valid mask block, and the image block corresponding to the valid mask block is enhanced to obtain a super-resolution image block upsampled N times, and the super-resolution image block is downsampled 1 / N times to obtain an enhanced image block, where N is a preset multiple; Generate repair content for the repair area according to the enhanced image block and the effective mask block to obtain a repaired image block; According to the effective mask block, the image to be repaired and the repair image block are merged to obtain a fused image block, and the image block corresponding to the effective mask block is replaced with the fused image block; According to each image block, a final output image is generated; The step of extracting the area to be repaired where the target frame exists in the image to be repaired and generating a binary mask image is as follows: Performing color space conversion on the image to be repaired to obtain an RGB color gamut image and an HSV color gamut image, and performing color threshold segmentation on the RGB color gamut image and the HSV color gamut image according to the color of the target frame to be removed, to obtain a first binary image corresponding to the RGB color gamut and a second binary image corresponding to the HSV color gamut; According to the unit color vector of the target frame color, two basis vectors with a large angle difference with the target frame color are determined from the basis vectors of the RGB color domain to perform Schmidt orthogonal decomposition to obtain a unit orthogonal basis, and according to the unit orthogonal basis, the RGB color domain image is coordinate transformed to obtain a color representation image based on the target frame color; According to the color representation image, the minimum channel difference is calculated to obtain a difference image; Performing edge detection on the differential image to obtain a third binary image; performing closing operation and opening operation of morphological filtering on the first binarized image, the second binarized image and the third binarized image respectively; According to the geometric properties of the target frame, respectively, geometric feature filtering is performed on the first binary image, the second binary image, and the third binary image after morphological filtering; The first binary image, the second binary image and the third binary image after geometric feature filtering are fused to obtain a mask image corresponding to the area to be repaired; The first binary image, the second binary image and the third binary image after the geometric feature filtering are fused to obtain a mask image corresponding to the area to be repaired, specifically: Performing gated fusion on the first binary image, the second binary image and the third binary image after geometric feature filtering to obtain a binary fused image; performing morphological filtering on the binary fused image to generate a mask image corresponding to the area to be repaired; Alternatively, the first binary image, the second binary image and the third binary image after geometric feature filtering are respectively subjected to morphological filtering dilation processing; the first binary image, the second binary image and the third binary image after dilation processing are gated and fused to generate a mask image corresponding to the area to be repaired.
2. The image restoration method according to claim 1, characterized in that: According to the color of the target frame to be removed, the RGB color gamut image and the HSV color gamut image are respectively subjected to color threshold segmentation to obtain a first binary image corresponding to the RGB color gamut and a second binary image corresponding to the HSV color gamut, specifically: Determine a first segmentation threshold according to the color of the target frame to be removed, and perform binarization processing on the RGB color gamut image according to the first segmentation threshold to obtain a first binarized image corresponding to the RGB color gamut; A second segmentation threshold is determined according to the color and saturation of the target frame to be removed, and the HSV color gamut image is binarized according to the second segmentation threshold to obtain a second binarized image corresponding to the HSV color gamut.
3. The image restoration method according to claim 1, characterized in that: According to the unit color vector of the target frame color, two basis vectors with a large angle difference are determined from the basis vectors of the RGB color domain to perform Schmidt orthogonal decomposition to obtain a unit orthogonal basis, and according to the unit orthogonal basis, the RGB color domain image is coordinate transformed to obtain a color representation image based on the target frame color, specifically: The color vector mean of the target frame color is obtained by sampling, and then normalized to obtain the unit color vector; Calculate the cosine similarity between each basis vector of the RGB color domain and the unit color vector respectively, and replace the basis vector corresponding to the minimum cosine similarity with the unit color vector to obtain an orthogonalized basis vector mark; According to the unit color vector, the other two basis vectors in the basis vector mark are subjected to Schmidt orthogonal decomposition to obtain a unit orthogonal basis; According to the coordinate conversion formula, the RGB color gamut image is converted to obtain a color representation image based on the target frame color. The coordinate conversion formula is [x] c =C -1 [x] rgb , where [x] c represents the coordinates of the color representation image, C represents the unit orthogonal basis, [x] rgb Represents the coordinates of an RGB color space image.
4. The image restoration method according to claim 1, characterized in that: According to the geometric attributes of the target frame, the first binary image, the second binary image and the third binary image after morphological filtering are respectively subjected to geometric feature filtering, specifically: Obtaining external closed contours in a binary image, and respectively calculating the contour convex hull area and the internal solidity rate of each external closed contour, wherein the binary image includes a first binary image, a second binary image, and a third binary image; If the convex hull area of an external closed contour is less than or equal to a preset area threshold, the area where the external closed contour is located is set as a non-repairing area; If the convex hull area of an external closed contour is greater than a preset area threshold and the internal solid ratio of the external closed contour is greater than a preset solid ratio threshold, polygon fitting is performed on the external closed contour, and the number of vertices of the polygon obtained by fitting and the average of the absolute values of the cosines of its internal angles are counted; If the number of vertices of the polygon is different from the number of vertices of the target frame and the mean of the absolute values of the cosine is greater than a preset mean threshold, the area where the external closed contour is located is set as a non-repairing area.
5. The image restoration method according to claim 1, characterized in that: Before the image to be repaired and the mask image are respectively cut according to the preset block size to obtain a plurality of image blocks and mask blocks, the method further comprises: Performing edge filling on the image to be repaired and the mask image respectively, wherein the sizes of the image to be repaired and the mask image after the edge filling are integer multiples of a preset block size; The method of generating a final output image according to each image block is specifically as follows: Each image block is spliced to obtain a spliced image, and edge filling pixels in the spliced image are removed according to the size of the image to be repaired to obtain a final output image.
6. The image restoration method according to claim 1, characterized in that: The generating of the repair content for the repaired area according to the enhanced image block and the effective mask block to obtain the repaired image block is specifically as follows: According to the background information of the enhanced image block and the neighborhood information of the to-be-repaired area in the effective mask block, a restoration content is generated by a preset image restoration algorithm to obtain a restored image block.
7. The image restoration method according to claim 1, characterized in that: The step of fusing the image to be repaired and the repaired image block according to the effective mask block to obtain a fused image block is specifically as follows: The effective mask block is normalized, inverted, expanded and blurred in sequence to obtain a weight; According to the weighted fusion formula, the image to be repaired and the repaired image block corresponding to the effective mask block are weightedly fused to obtain a fused image block. The weighted fusion formula is F p =I p ·(1-W p )+R p ·W p , where F p represents the fused image block, I p Represents the image to be repaired, R p represents the repaired image block, W p Indicates the weight corresponding to the valid mask block.
8. An image restoration device, characterized in that: include: The first generation module is used to extract the area to be repaired where the target frame exists in the image to be repaired, and generate a binary mask image; A cropping module, used for cropping the image to be repaired and the mask image according to a preset block size, to obtain a plurality of image blocks and mask blocks, wherein each image block corresponds to each mask block one by one; an enhancement module, configured to take the mask block containing the area to be repaired as a valid mask block, perform image enhancement on the image block corresponding to the valid mask block to obtain a super-resolution image block upsampled N times, and perform 1 / N times downsampling on the super-resolution image block to obtain an enhanced image block, wherein N is a preset multiple; A repair module, used for generating repair content for the area to be repaired according to the enhanced image block and the effective mask block, to obtain a repaired image block; A fusion module, configured to fuse the image to be repaired and the repair image block according to the effective mask block to obtain a fused image block, and replace the image block corresponding to the effective mask block with the fused image block; A second generating module, used for generating a final output image according to each image block; The step of extracting the area to be repaired where the target frame exists in the image to be repaired and generating a binary mask image is as follows: Performing color space conversion on the image to be repaired to obtain an RGB color gamut image and an HSV color gamut image, and performing color threshold segmentation on the RGB color gamut image and the HSV color gamut image according to the color of the target frame to be removed, to obtain a first binary image corresponding to the RGB color gamut and a second binary image corresponding to the HSV color gamut; According to the unit color vector of the target frame color, two basis vectors with a large angle difference with the target frame color are determined from the basis vectors of the RGB color domain to perform Schmidt orthogonal decomposition to obtain a unit orthogonal basis, and according to the unit orthogonal basis, the RGB color domain image is coordinate transformed to obtain a color representation image based on the target frame color; According to the color representation image, the minimum channel difference is calculated to obtain a difference image; Performing edge detection on the differential image to obtain a third binary image; performing closing operation and opening operation of morphological filtering on the first binarized image, the second binarized image and the third binarized image respectively; According to the geometric properties of the target frame, respectively, geometric feature filtering is performed on the first binary image, the second binary image, and the third binary image after morphological filtering; The first binary image, the second binary image and the third binary image after geometric feature filtering are fused to obtain a mask image corresponding to the area to be repaired; The first binary image, the second binary image and the third binary image after the geometric feature filtering are fused to obtain a mask image corresponding to the area to be repaired, specifically: Performing gated fusion on the first binary image, the second binary image and the third binary image after geometric feature filtering to obtain a binary fused image; performing morphological filtering on the binary fused image to generate a mask image corresponding to the area to be repaired; Alternatively, the first binary image, the second binary image and the third binary image after geometric feature filtering are respectively subjected to morphological filtering dilation processing; the first binary image, the second binary image and the third binary image after dilation processing are gated and fused to generate a mask image corresponding to the area to be repaired.
9. An electronic device, characterized in that: The electronic device comprises: one or more processors; A storage device for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the image restoration method as described in any one of claims 1-7.
10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the image restoration method according to any one of claims 1 to 7 are implemented.
