Image restoration method and device, equipment, storage medium and program product

Abstract

The invention provides an image restoration method and device, equipment, a storage medium and a program product, and relates to the technical field of image processing. The image restoration method comprises the following steps: segmenting at least two regions of interest with different degradation scenes from a to-be-restored image; based on a preset experience knowledge base, determining an image inpainting scheme corresponding to each region of interest, the image inpainting scheme comprising at least one image inpainting operation and an execution sequence of each image inpainting operation; restoring each region of interest based on the corresponding image restoration scheme to obtain a corresponding image restoration region; and based on the image restoration areas, fusing to generate a target restoration image.

Description

Technical Field

[0001] This specification relates to the field of image processing technology, and in particular to an image restoration method, apparatus, device, storage medium, and computer program product. Background Technology

[0002] An image restoration system is an automated or semi-automated system that processes images with poor or damaged quality to restore or improve their visual quality. It can perform restoration processes such as noise reduction, deblurring, super-resolution, scratch repair, and color correction, thereby improving the display effect of the image.

[0003] Current image inpainting techniques have made significant progress in single tasks such as denoising and deblurring, with various deep learning models (such as Restormer, HAT, and DehazeFormer) demonstrating near-human visual restoration capabilities on standard datasets. However, existing technologies still face challenges when dealing with complex degradation commonly found in real images (such as the simultaneous presence of haze, noise, motion blur, and low resolution). Degradation in real images often exhibits strong regional differences. Existing image inpainting systems generally employ a globally uniform processing flow that cannot adapt to local degradation characteristics, ignoring the spatial heterogeneity of degradation distribution within the image. This leads to a mismatch between inpainting strategies and local needs, resulting in insufficient inpainting in some areas (such as incomplete haze removal) while other areas are overprocessed, introducing artifacts (such as texture smoothing and edge ringing), thus reducing overall visual quality.

[0004] In view of this, embodiments of this specification provide an image restoration method, apparatus, computer device, computer-readable storage medium, and computer program product, which aim to improve the quality of image restoration by introducing a region-aware and region-level task topology reconstruction mechanism and using a preset experience knowledge base to achieve adaptive and refined restoration of composite degraded images. Summary of the Invention

[0005] This specification provides one or more embodiments of an image restoration method, the method comprising: segmenting at least two regions of interest with different degradation scenes from an image to be restored; determining an image restoration scheme corresponding to each of the regions of interest based on a preset empirical knowledge base, the image restoration scheme including at least one image restoration operation and the execution order of each image restoration operation; restoring each region of interest based on the corresponding image restoration scheme to obtain a corresponding image restoration region; and fusing the image restoration regions to generate a target restored image.

[0006] In some embodiments, segmenting at least two regions of interest with different degradation scenarios from the image to be repaired includes: obtaining degradation analysis results and semantic segmentation results of the image to be repaired; segmenting the regions of interest based on the degradation analysis results and semantic segmentation results; wherein the degradation analysis results include degradation index data and degradation distribution of different degradation types in the image to be repaired, and the semantic segmentation results include location information of different semantic regions in the image to be repaired.

[0007] In some embodiments, regions of interest are segmented based on degradation analysis results and semantic segmentation results, including: obtaining feature vectors of each pixel in the image to be repaired based on degradation analysis results and semantic segmentation results; clustering each pixel according to the feature vectors of each pixel, and obtaining each region of interest and the degradation scene of each region of interest according to the clustering results; wherein, the degradation scene includes the semantic category, degradation type and degradation degree corresponding to each region of interest.

[0008] In some embodiments, based on the degradation analysis results and semantic segmentation results, the feature vector of each pixel in the image to be repaired is obtained, including: based on the degradation analysis results, obtaining the degradation features corresponding to each pixel; based on the semantic segmentation results, obtaining the semantic features corresponding to each pixel; and fusing the degradation features and semantic features corresponding to each pixel to generate the feature vector of each pixel.

[0009] In some embodiments, obtaining the degradation analysis results and semantic segmentation results of the image to be repaired includes: processing the image to be repaired using a preset visual language model to obtain the degradation type and semantic segmentation results of the image to be repaired, and generating a degradation heatmap that can characterize the degradation distribution of the corresponding degradation type; and using a preset image quality assessment network to obtain degradation index data of different degradation types in the image to be repaired based on the degradation heatmap corresponding to each degradation type.

[0010] In some embodiments, the experience knowledge base is generated based on historical restoration records of sample images; the experience knowledge base includes multiple image restoration strategies, which include sample degradation scenarios and image restoration schemes and image restoration effects corresponding to the sample degradation scenarios.

[0011] In some embodiments, based on a preset empirical knowledge base, determining an image restoration scheme corresponding to each region of interest includes: matching sample degradation scenes with a similarity higher than a preset threshold from the empirical knowledge base based on the degradation scenes of each region of interest, and using the image restoration scheme corresponding to the matched sample degradation scenes as candidate image restoration schemes for the corresponding region of interest; selecting the image restoration scheme with the highest restoration success rate and / or the best image restoration effect from the candidate image restoration schemes for the corresponding region of interest as the image restoration scheme for the corresponding region of interest.

[0012] In some embodiments, the experience knowledge base further includes image restoration operation execution order rules and / or image restoration operation warning rules; based on the preset experience knowledge base, determining the image restoration scheme corresponding to each region of interest, further includes: verifying the image restoration scheme corresponding to each region of interest based on the image restoration operation execution order rules; and / or providing corresponding warning information when there is an image restoration operation that triggers a warning in the image restoration scheme corresponding to each region of interest based on the image restoration operation warning rules.

[0013] In some embodiments, each image restoration operation includes processing steps executed in a preset order; restoration of each region of interest based on a corresponding image restoration scheme includes: performing each image restoration operation to restore each region of interest according to the corresponding execution order based on the corresponding image restoration scheme; evaluating the restoration quality of the intermediate restored image obtained by the monitored current processing step during the execution of at least one image restoration operation; and executing the next processing step in the corresponding image restoration operation according to a preset order when the restoration quality evaluation is passed.

[0014] In some embodiments, repairing each region of interest based on the corresponding image restoration scheme further includes: canceling the execution of the current processing step and obtaining the previous valid intermediate restored image obtained from the previous processing step when the restoration quality assessment fails; updating the corresponding image restoration scheme based on the previous valid intermediate restored image and the experience knowledge base; and repairing the previous valid intermediate restored image based on the image restoration operations and execution order included in the updated image restoration scheme.

[0015] In some embodiments, a target restored image is generated by fusing images based on each image restoration region, including: establishing a transition zone at the boundary of each image restoration region; stitching together each image restoration region; and performing weighted fusion processing on image blocks that overlap between adjacent image restoration regions within each transition zone to generate the target restored image.

[0016] In some embodiments, weighted fusion processing is performed on image blocks that overlap in adjacent image restoration regions within each transition zone, including: using Laplacian pyramid decomposition to decompose the overlapping image blocks within each transition zone into different frequency layers, and using different fusion weights at different frequency layers to perform weighted fusion of the overlapping image blocks.

[0017] In some embodiments, for frequency layers with frequencies higher than a preset frequency threshold, the fusion weights are assigned to the corresponding image blocks based on the image gradient magnitude; for frequency layers with frequencies lower than the preset frequency threshold, the fusion weights are assigned to the corresponding image blocks based on the semantic segmentation results.

[0018] According to one or more embodiments of this specification, an image restoration apparatus is also provided. The apparatus includes: a region perception module for segmenting at least two regions of interest with different degradation scenes from an image to be restored; a scheduling module for determining an image restoration scheme corresponding to each region of interest based on a preset experience knowledge base, wherein the image restoration scheme includes at least one image restoration operation and the execution order of each image restoration operation; a restoration module for restoring each region of interest based on the corresponding image restoration scheme to obtain a corresponding image restoration region; and a fusion module for fusing the image restoration regions to generate a target restored image.

[0019] One or more embodiments of this specification also provide a computer device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it is able to implement the image restoration methods of some embodiments of this specification.

[0020] This specification also provides a computer-readable storage medium storing computer instructions that, when executed by a processor, enable the image restoration methods of some embodiments of this specification.

[0021] One or more embodiments of this specification also provide a computer program product, including a computer program that, when at least a portion of the computer program is executed by a processor, enables the implementation of the image restoration methods of some embodiments of this specification.

[0022] The beneficial effects that the embodiments of this specification may bring include, but are not limited to: by segmenting different regions of interest (ROIs) from the image to be repaired, the spatial heterogeneity of degradation is identified; based on a preset knowledge base, a customized image restoration scheme is dynamically determined and restored for each ROI, and then the restoration results are fused to generate the target restored image, thereby reducing image restoration from the "whole image level" to the "region level." The most suitable image restoration scheme is independently assigned to each ROI, achieving precise restoration with a "one-region-one-policy" approach. This fundamentally solves the problem of insufficient restoration in some areas and over-processing in others when dealing with complex degradation using traditional "one-size-fits-all" restoration strategies, greatly improving the restoration effect on complex degradation scenes in real images. It should be noted that different embodiments may produce different beneficial effects. In different embodiments, the beneficial effects may be any one or a combination of the above, or any other possible beneficial effects. Attached Figure Description

[0023] This specification will be further described by way of exemplary embodiments, which will be described in detail with reference to the accompanying drawings. The same numbers in the drawings denote the same structures or steps.

[0024] Figure 1 This is a schematic diagram illustrating an application scenario of an image restoration method according to some embodiments of this specification.

[0025] Figure 2 This is an exemplary flowchart of an image restoration method according to some embodiments of this specification.

[0026] Figure 3 This is an exemplary flowchart illustrating a region of interest segmentation according to some embodiments of this specification.

[0027] Figure 4 This is an exemplary flowchart illustrating an image inpainting scheme for determining a region of interest, according to some embodiments of this specification.

[0028] Figure 5 This is an exemplary structural diagram of an image restoration apparatus according to some embodiments of this specification.

[0029] Figure 6 This is a schematic diagram of the hardware architecture of a computer device according to some embodiments of this specification. Detailed Implementation

[0030] To more clearly illustrate the technical solutions of the embodiments in this specification, the embodiments will be described in detail below with reference to the accompanying drawings. Obviously, the content described below are some examples or embodiments of this specification. For those skilled in the art, without creative effort, the technical solutions or means disclosed in this specification can be applied to other scenarios based on this technical content.

[0031] It should be understood that the terms "system," "device," "unit," and / or "module" used in this specification are a method of distinguishing different components, elements, parts, sections, or assemblies at different levels. However, if other terms can achieve the same purpose, they may be replaced by other expressions.

[0032] Unless otherwise specified, the technical terms used to describe components, elements, etc. in this specification are not singular but may include plural. Generally speaking, terms such as "comprising" or "including" only indicate that explicitly identified steps, elements, or components are included, and these steps, elements, and components do not constitute an exclusive list, as the described method or apparatus may also include other steps or components.

[0033] This specification uses flowcharts to illustrate the operational steps performed by the apparatus or system of related embodiments. However, unless otherwise specified, the order in which these steps are described should not be construed as a limitation on the order of execution. Those skilled in the art can adjust the order of these steps based on the knowledge and information conveyed by the embodiments in this specification. Such adjustments include, but are not limited to, reversing the order of steps, merging multiple steps, and splitting a step.

[0034] Current image inpainting techniques have made significant progress in single tasks such as denoising and deblurring, with various deep learning models (such as Restormer, HAT, and DehazeFormer) demonstrating near-human visual restoration capabilities on standard datasets. However, existing technologies still face challenges when dealing with complex degradation commonly found in real images (such as the simultaneous presence of haze, noise, motion blur, and low resolution).

[0035] In some related embodiments, the spatial non-uniformity of degradation distribution is not effectively modeled. Specifically, degradation in real images often exhibits strong regional differences: for example, in aerial footage, degradation in the sky is mainly due to atmospheric scattering (haze), while degradation in the ground building area includes sensor noise, motion blur, and loss of detail. Image inpainting systems generally adopt a globally uniform processing flow—that is, performing the same inpainting scheme on the entire image. This "one-size-fits-all" strategy cannot adapt to local degradation characteristics, easily leading to under-inpainting in some areas (such as incomplete removal of haze), while other areas are over-processed, introducing artifacts (such as smoothed textures and ringing edges). Although attention mechanisms or adaptive weights can be introduced for optimization, they are still limited to a single model and cannot achieve cross-tool, cross-task regional strategy customization.

[0036] In some related embodiments, the restoration process lacks dynamic intervention capabilities during execution. Specifically, mainstream systems often adopt an open-loop "plan-execute" architecture: a static restoration operation sequence is first generated by a scheduling module (such as rule-based or large language model-based systems), and then executed sequentially. This model is based on the premise that each operation can be completed as expected, ignoring the uncertainty and potential failure risk of image restoration tools under complex inputs. For example, a de-raining model may produce halos or structural breaks in areas with high-density rain and fog, but the system can only detect the anomaly through quality assessment after the tool has been fully executed. At this point, not only have a large amount of computing resources been wasted, but the original image information may also be permanently lost due to irreversible operations, and even if a rollback mechanism is subsequently introduced, the damaged intermediate features cannot be recovered.

[0037] In other related embodiments, multi-tool collaboration lacks topology awareness and experience-based guidance. Specifically, while pluggable toolchain architectures improve system flexibility, the topological dependencies between tools (e.g., "denoising should be performed before deblurring") and coupling effects (e.g., "super-resolution followed by denoising amplifies artifacts") have not been systematically modeled. Common scheduling strategies either rely on manual experience rules or semantic reasoning from large language models, lacking quantitative understanding of tool behavior under real-world degenerate combinations. This leads to the system potentially generating logically conflicting or inefficient operation sequences when faced with unseen degenerate combinations, resulting in highly unstable repair outcomes.

[0038] The aforementioned image restoration techniques all have significant shortcomings in terms of spatial adaptability, process controllability, and topological intelligence, making them ill-equipped to handle complex, heterogeneous, and dynamic degradation scenarios in the real world. Therefore, a new solution capable of achieving region perception, process monitoring, and dynamic reconstruction is urgently needed to overcome these performance bottlenecks.

[0039] In view of this, the embodiments of this specification provide a method to support image restoration, and design a region-level task topology reconstruction scheme: First, by segmenting different Regions of Interest (ROIs) of the degradation scene from the image to be restored, then, based on a preset experience knowledge base, dynamically determine a customized image restoration scheme for each ROI and perform restoration, and finally, fuse the restoration results to generate the target restored image, thereby sinking the task topology reconstruction from the "whole image level" to the "region level", dynamically generating the optimal image restoration scheme for each ROI, and achieving precise restoration with "one policy for each region". Secondly, an interruptible and predictable execution mechanism is constructed: by evaluating the direction of the tool (image restoration operation) in real time during execution, it is supported to stop execution and jump paths in advance based on the evaluation results. In addition, by introducing an "experience knowledge base" based on historical data and containing topological rules and quantified effect knowledge, the operation collaboration is transformed from relying on instantaneous and abstract reasoning to decision-making based on systematic and quantified historical experience. Even when faced with unseen degradation combinations, it can generate a logically reasonable and predictably stable operation sequence through similarity matching and rule constraints.

[0040] Figure 1 This is a schematic diagram illustrating an application scenario of an image restoration method according to some embodiments of this specification. In some embodiments, such as... Figure 1As shown, application scenario 100 may include server 110, network 120, and client 130, wherein server 110 can connect to client 130 through network 120. In some embodiments, client 130 can acquire the image to be repaired and segment at least two regions of interest with different degradation scenes from the image to be repaired; client 130 can upload the segmentation results to server 110 through network 120, and determine the image repair scheme corresponding to each region of interest through a preset experience knowledge base in server 110; client 130 can perform image repair processing on each region of interest according to the image repair scheme returned by server 110 and then fuse them to realize the entire image repair process, which is not limited here.

[0041] In some embodiments, server 110 can be a computer device with high computing performance. In some embodiments, server 110 can be a single computer device or a computing cluster composed of multiple computer devices, thereby providing powerful computing power and efficient response for image restoration. In some embodiments, server 110 can be a server, which can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Distribution Network), and big data and artificial intelligence platforms.

[0042] In some embodiments, network 120 can be any form of wired or wireless network, or any combination thereof. By way of example only, network 120 can be one or more combinations of wired networks, fiber optic networks, telecommunications networks, internal networks, the Internet, local area networks (LANs), wide area networks (WANs), wireless local area networks (WLANs), metropolitan area networks (MANs), public switched telephone networks (PSTNs), Bluetooth networks, etc. Network 120 can have multiple access points, through which server 110 and client 130 can access network 120.

[0043] In some embodiments, the client 130 may include, but is not limited to, a desktop computer, smartphone, laptop, VR (Virtual Reality) device, and tablet computer. In some embodiments, the user can use the client 130 to acquire the image to be repaired, perform repair processing, and present the final repair effect; this is not limited thereto. In some embodiments, the client 130 can provide the ROI segmentation result of the image to be repaired to the server 110 and receive the image repair scheme fed back by the server 110; this is not limited thereto.

[0044] It should be noted that, Figure 1 The illustrated application scenario diagram of the image restoration method is merely an example. The application scenarios described in the embodiments of this specification are intended to more clearly illustrate the technical solutions of the embodiments of this specification and do not constitute a limitation on the technical solutions provided in the embodiments of this specification. For example, Figure 1 The number of server 110 and client 130 in this specification is merely illustrative and is not intended to limit the scope of patent protection. Depending on the actual situation, there can be any number of server 110 and client 130. As those skilled in the art will understand, with the development of image restoration technology and the emergence of new business scenarios, for example, client 130 can store a preset experience knowledge base, enabling local restoration of the image to be restored without server 110 and network 120; or, for example, client 130 can directly provide the image to be restored to server 110 via network 120, whereby server 110 completes ROI segmentation, restoration scheme determination, image restoration, region fusion, and feedback of the image restoration results, while client 130 only presents the final image restoration result.

[0045] Figure 2 This is an exemplary flowchart of an image restoration method according to some embodiments of this specification. In some embodiments, Figure 2 The process 200 shown can be executed by a processing device, for example, by a... Figure 1 The server 110 shown is executed; or, Figure 2 The process 200 shown can be executed by a terminal device, for example, by a terminal device such as... Figure 1 The client 130 shown is executed; or, Figure 2 The process 200 shown can be executed jointly by a processing device and a terminal device, for example, it can be executed by a processing device and a terminal device. Figure 1 The server 110 and client 130 shown are executed together. In some embodiments, process 200 may be implemented by an image restoration device 500 deployed on a processing device and / or a terminal device.

[0046] In some embodiments, such as Figure 2As shown, process 200 may include the following steps.

[0047] Step 210: Segment at least two regions of interest with different degradation scenes from the image to be repaired. In some embodiments, step 210 can be implemented by the region-aware module 510.

[0048] In some embodiments, the image to be repaired may be an image with poor or damaged image quality, requiring targeted image restoration operations to restore or improve its visual quality. In some embodiments, the image to be repaired contains compound degradation, that is, multiple degradations coexist. For example, in aerial images, the degradation in the sky area is mainly due to haze, while the degradation in the ground building area may simultaneously include sensor noise, motion blur, and loss of detail. In some embodiments, the region of interest refers to a region in the image to be repaired that has a specific degradation scene (including specific degradation features and semantic attributes), such as a "high haze sky area" or a "high noise texture building area," and is the basic unit for regional restoration.

[0049] For ease of understanding, the following example image is taken with a resolution of 1920×1080. In this example image, there is high intensity of haze in the sky area and moderate rain streaks and Gaussian noise in the building area.

[0050] Figure 3 This is an exemplary flowchart illustrating the segmentation of regions of interest according to some embodiments of this specification. In some embodiments, Figure 3 The illustrated process 300 can be implemented by the area-aware module 510. In some embodiments, such as Figure 3 As shown, process 300 may include the following steps.

[0051] Step 310: Obtain the degradation analysis results and semantic segmentation results of the image to be repaired.

[0052] In some embodiments, degradation analysis results may include degradation index data and degradation distribution of different degradation types in the image to be restored. Degradation type refers to the defined category of image damage areas that cause a decrease in visual quality during image acquisition, transmission, or processing. In some embodiments, different degradation types may be caused by defects in the image imaging system, or by external interference or environmental factors during image acquisition, without limitation. In some embodiments, different degradation types may include at least two of image noise, image rain patterns, image haze, image scratches, image blur, low image resolution, image color deviation, and low light in the image. Image noise refers to random pixel interference in the image to be restored, which is usually caused by sensor defects or magnification under low light conditions during image acquisition; image rain patterns refer to linear or patchy distortion in the image to be restored due to occlusion by environmental factors such as rain during acquisition; image haze refers to image blur and / or color distortion in the image to be restored due to atmospheric scattering during acquisition, and image haze and image rain patterns often occur simultaneously; image scratches refer to... The degradation analysis results refer to the presence of local linear defects in the image to be repaired due to physical damage or image transmission errors; image blurring refers to the softening of the edges of objects in the image to be repaired, which is usually caused by lens defocusing during the capture of moving objects or during image acquisition; low image resolution refers to the loss of image details or pixelation in the overall or partial image to be repaired; image color deviation refers to the deviation of pixel values ​​in some areas of the image to be repaired from the normal range, which is usually caused by white balance errors or image sensor aging during image acquisition; low image brightness refers to the decrease in contrast in the overall or partial image to be repaired due to insufficient imaging illumination. In some embodiments, the degradation analysis results support the analysis and evaluation of whether the image to be repaired has the above-listed degradation types. Those skilled in the art can also adaptively extend the above-listed degradation types according to actual needs, which is not limited here.

[0053] In some embodiments, degradation index data can refer to a quantitative measure of the degree of damage to an image damage area corresponding to a certain degradation type. For example, for a certain degradation type present in the image to be repaired, the higher the degradation index data, the greater the deviation between the image damage area corresponding to that degradation type in the whole or part of the image to be repaired and the image content that should be displayed normally. The degree of repair and adjustment required in subsequent image repair operations is also greater, such as noise decibel data. In some embodiments, degradation index data for a certain degradation type in the image to be repaired can also be determined according to a pre-set degradation grading standard. Specifically, it can be quantitatively represented by grading standards such as "low", "medium", "high", and "extremely high". In some embodiments, degradation index data can also include the proportion / statistical distribution model of the image damage area corresponding to a certain degradation type in the image to be repaired. Taking image blurring as an example, it can be distributed in the entire imaging area of ​​the image to be repaired, or it can be distributed only in a certain local imaging area of ​​the image to be repaired. Its coverage can be statistically analyzed as an indicator so that subsequent image repair operations can accurately repair the image damage area. In some embodiments, taking image noise as an example, the degradation index data may include a statistical distribution model, such as a Gaussian distribution or a Poisson distribution. Those skilled in the art can also set their own quantitative measurement standards for the degradation index data according to actual needs, which are not limited here.

[0054] In some embodiments, the degradation distribution refers to the specific location of the image damage area corresponding to a certain degradation type in the image to be repaired. In some embodiments, a degradation heatmap is used to characterize the degradation distribution corresponding to the degradation type. In some embodiments, the resolution of the degradation heatmap is 1 / 4 of the original image, and the pixel value range is [0,1], which is used for subsequent region weighted clustering. Those skilled in the art can also set the relevant parameters of the degradation heatmap according to actual needs, which is not limited here.

[0055] In some embodiments, semantic segmentation can refer to dividing and labeling different object regions in an image to be restored according to semantic categories. The semantic segmentation result includes the location information of different semantic regions in the image to be restored, wherein different semantic regions have different semantic categories. In some embodiments, semantic categories include the main object regions that may appear in the image. For example, in natural scene or aerial image restoration tasks, semantic categories may include, but are not limited to: sky, buildings, roads, vehicles, vegetation, water bodies, faces, etc. Those skilled in the art can adaptively define and extend semantic categories according to actual needs, and no limitation is made here.

[0056] In some embodiments, the degradation analysis results and semantic segmentation results of the image to be restored can be obtained by analyzing the image to be restored. In some embodiments, a preset Vision Language Model (VLM) can be used first to process the image to be restored. The VLM uses an improved Mask2Former network as its core architecture and implements pixel-level degradation classification and multi-semantic label segmentation based on a cross attention mechanism to obtain the degradation type and semantic segmentation results of the image to be restored, and generates a degradation heatmap corresponding to the degradation type. The degradation heatmap can characterize the degradation distribution of the corresponding degradation type. Then, a preset image quality assessment network is used to obtain degradation index data of different degradation types in the image to be restored based on the degradation heatmaps corresponding to each degradation type, and generates a corresponding structured diagnostic report by combining the semantic analysis results.

[0057] In some embodiments, taking the example image of the image to be repaired as having high-intensity haze in the sky area and moderate rain streaks and Gaussian noise in the building area, the visual language model can identify the image to be repaired through a pre-trained Mask2Former network and obtain the semantic segmentation results (i.e., the location information of the sky area and building area) and degradation classification results (i.e., the presence of three degradation types: haze, rain streaks, and noise) in the image to be repaired, while generating a degradation heatmap corresponding to each degradation type. Then, the image quality assessment network can evaluate based on the degradation heatmaps that the haze coverage area of ​​the image to be repaired accounts for 82% of the entire image area, and the haze level is high; the statistical distribution model of the image noise can be Gaussian noise, with an average Gaussian noise intensity of 25dB in the image to be repaired; the image rain streak density is 45 streaks / 100px², and the rain streak level is medium. The image quality assessment network can also combine the semantic segmentation results to identify the bounding box positions of different semantic regions and the main degradation types present, for example, the sky area mainly has haze, and the building area mainly has rain streaks and noise. Finally, the image quality assessment network combines its own and VLM's recognition and assessment results to generate a structured diagnostic report, the format of which is not limited here. In some embodiments, the structured diagnostic report may simultaneously include multiple degradation types globally present in the image to be repaired, degradation index data for each degradation type in the image to be repaired, and the file address of the corresponding degradation heatmap, which can accurately reflect the overall image damage in the image to be repaired from a global perspective. In addition, the report may also include the location and degradation types of different semantic regions. This multi-level report structure, which combines a "global degradation overview" with "semantic region local degradation analysis," provides complete and structured factual evidence for subsequent precise repair based on "region awareness" and "one policy per region."

[0058] Step 320: Based on the obtained degradation analysis results and semantic segmentation results, segment the region of interest.

[0059] In some embodiments, the image restoration apparatus can segment the region of interest (ROI) based on the degradation analysis results and the semantic segmentation results. First, feature vectors of each pixel in the image to be restored can be obtained based on the degradation analysis results and the semantic segmentation results. Then, each pixel can be clustered according to its feature vector, and each ROI and its degradation scenario can be obtained based on the clustering results. The degradation scenario includes the semantic category, degradation type, and degradation degree corresponding to each ROI. In some embodiments, when obtaining the feature vectors of each pixel in the image to be restored, the image restoration apparatus can obtain the degradation features corresponding to each pixel based on the degradation analysis results obtained in step 310. Specifically, this is done by reading the degradation heatmap files corresponding to different degradation types in the structured report and obtaining the intensity of the influence of the corresponding degradation type on each pixel as the degradation feature corresponding to each pixel. Then, semantic features corresponding to each pixel can be obtained based on the semantic segmentation results. For example, the semantic category of each pixel can be read from the semantic segmentation results as the semantic feature corresponding to each pixel. Finally, the degradation features and semantic features corresponding to each pixel are fused to generate the feature vector of each pixel. For example, assuming the image to be repaired contains haze, rain streaks, and noise, the final generated feature vector can be represented as [haze_score, noise_score, rain_score, semantic_id], where haze_score represents the intensity of the haze effect on the corresponding pixel, noise_score represents the intensity of the noise effect on the corresponding pixel, rain_score represents the intensity of the rain streaks effect on the corresponding pixel, and semantic_id represents the semantic category of the corresponding pixel. In some embodiments, a weighted DBSCAN clustering algorithm can be used to cluster pixels based on their feature vectors. During clustering, the algorithm searches for spatially adjacent and feature-similar pixels based on a preset neighborhood radius parameter and groups pixels that meet the condition into the same cluster. Simultaneously, the algorithm is configured with a minimum cluster size constraint parameter (e.g., the minimum cluster size is not less than 10,000 pixels) to filter out small-scale pixel sets generated by sporadic noise points or over-segmentation, ensuring that each pixel cluster generated ultimately represents a region in the image with significant physical size and visual significance. After clustering, each pixel cluster output by the algorithm—that is, a set of spatially connected pixels that satisfies the minimum size constraint—is considered an independent region of interest (ROI). Simultaneously, based on the aforementioned structured report, the degradation scenario of each ROI is described, including semantic category, degradation type, and degradation degree. For example, when the image to be repaired is the aforementioned example image, clustering automatically segments two ROIs: ROI_A and ROI_B.The degradation scenario description for ROI_A is "sky,haze-H", indicating that ROI_A is a sky area with high-intensity haze; the degradation scenario description for ROI_B is "building,rain-M+noise-H", indicating that ROI_B is a building area with medium-intensity rain streaks and high-intensity noise.

[0060] In some embodiments, different regions of interest (ROIs) and corresponding degradation scenes of the image to be repaired can be provided along with the image to be repaired. For example, in some application scenarios, when a user provides an image to be repaired for image repair, the ROIs and corresponding degradation scene descriptions in the image to be repaired can be provided together and used directly as the ROI segmentation results.

[0061] Step 220: Based on a preset experience knowledge base, determine the image restoration scheme corresponding to each region of interest. In some embodiments, step 220 can be implemented by the restoration scheme determination module 520.

[0062] In some embodiments, an image restoration scheme can be a preferred restoration plan based on the degradation scene of the region of interest (ROI), which includes at least one image restoration operation and the execution order of each image restoration operation. In some embodiments, an image restoration operation can correspond to one or more degradation types of the ROI and is used to restore the image damage areas corresponding to one or more degradation types. In some embodiments, a single image restoration operation in an image restoration scheme can include an image restoration tool used to perform the image restoration operation and restoration configuration parameters of the image restoration tool. The restoration configuration parameters of the image restoration tool can be determined according to the degradation severity of the degradation type it is restoring, and are not limited here. For example, for an image noise degradation type with a degradation severity of "high", the image restoration tool Restormer-V3 can be selected to perform the image restoration operation, and the denoising intensity of the image restoration tool Restormer-V3 can be configured to level 5 to meet the restoration requirements of the image noise degradation type with a degradation severity of "high". In some embodiments, the execution order of each image restoration operation can refer to the order in which the image restoration operations are executed when the image restoration scheme includes two or more image restoration operations. It is understandable that when a region of interest contains two or more degradation types, two or more image inpainting operations may be required for targeted repair. Each image inpainting operation may, during its execution, cause further image quality degradation in parts of the region of interest. In some embodiments, taking a region of interest containing image noise, image haze, and image rain streaks as an example, if a haze removal operation is performed on the region of interest first, this operation may abnormally amplify noise particles in the image noise into artifacts, further degrading the image quality. If a denoising operation is performed on the region of interest first, this operation may easily misjudge normal rain streak structures in the image as high-frequency image noise and mistakenly eliminate them, also further degrading the image quality. Therefore, after determining the image inpainting operations included in the image inpainting scheme, it is also crucial to reasonably sort the execution order of the image inpainting operations. The following will further explain the method for determining the image inpainting scheme with reference to embodiments.

[0063] In some embodiments, the experience knowledge base can be generated based on historical restoration records of one or more sample images. In some embodiments, the experience knowledge base can utilize an automated testing platform to perform restoration tests on each sample image using different combinations of image restoration operations to generate a massive amount of historical restoration records. Based on the historical restoration records of the sample images, experience is summarized to generate multiple image restoration strategies. Each image restoration strategy can include a sample degradation scenario, an image restoration scheme corresponding to the sample degradation scenario, and an image restoration effect. In some embodiments, a sample degradation scenario includes the semantic category, degradation type, and degradation severity of the sample image. In some embodiments, the data structure describing the sample degradation scenario can be consistent with or correspond to the data structure describing the degradation scenario of the region of interest obtained in the foregoing embodiments, facilitating subsequent retrieval of similar sample degradation scenarios. In some embodiments, an image restoration scheme includes at least one image restoration operation and the execution order of each image restoration operation. Each image restoration scheme corresponds to a sample degradation scenario. When the degradation scenario of the region of interest matches a certain sample degradation scenario, the image restoration scheme corresponding to the matching sample degradation scenario can be selected as the image restoration scheme for that region of interest. In some embodiments, the image restoration effect includes the restoration success rate of the corresponding image restoration scheme and the image presentation effect after restoration of the sample image based on the corresponding image restoration scheme. A sample degradation scene may correspond to one or more image restoration schemes. Based on the degradation scene of the actual region of interest, the image restoration effect can be used to select and determine the image restoration scheme that meets the user's needs. In a specific example, taking a sample degradation scene that simultaneously contains image haze and image noise as an example, the corresponding image restoration scheme may be to first use the Restormer-v3 image restoration tool for image restoration and noise reduction, and then use the DehazeFormer-v2 image restoration tool for image restoration and deblurring. The corresponding image restoration effect may be that the restoration success rate of this image restoration scheme is 86%, the restoration time or restoration latency is 75ms, the peak signal-to-noise ratio gain of this image restoration scheme is 8.2 compared to other image restoration schemes, and the residual degradation of image noise in the restored image presentation effect is less than 5dB, and the residual degradation rate of image haze is less than 20%. The above data are all relevant empirical data in specific examples and do not constitute a limitation on the embodiments of this specification.

[0064] Figure 4 This is an exemplary flowchart illustrating an image inpainting scheme for determining a region of interest according to some embodiments of this specification. In some embodiments, Figure 4 The illustrated process 400 can be implemented by the repair scheme determination module 520. In some embodiments, such as Figure 4 As shown, process 400 may include the following steps.

[0065] Step 410: Based on the degradation scenes of each region of interest, match sample degradation scenes with a similarity higher than a preset threshold from the empirical knowledge base, and use the image restoration schemes corresponding to the matched sample degradation scenes as candidate image restoration schemes for the corresponding regions of interest. In some embodiments, a Large Language Model (LLM) can be applied to semantically match the degradation scenes of the acquired regions of interest with the sample degradation scenes in the empirical knowledge base through a semantic matching engine to obtain the similarity between the degradation scenes of each region of interest and each sample degradation scene; if the degradation scene of a region of interest has a high similarity to a certain sample degradation scene, it indicates that the image degradation problem faced by the region is highly similar to the image degradation problem corresponding to the sample degradation scene, and it is suitable to use the image restoration scheme corresponding to the sample degradation scene to perform image restoration processing on the region of interest. In some embodiments, considering that the degradation scene of the region of interest may be more similar to multiple sample degradation scenes in the experience knowledge base than a preset threshold (for example, the preset threshold can be set to 90%, 95%, 80%, etc. according to actual needs, and is not limited here), the image restoration schemes corresponding to these sample degradation scenes can be screened out as candidate image restoration schemes, and the image restoration scheme corresponding to the region of interest can be further determined from multiple candidate image restoration schemes through the subsequent steps of process 400.

[0066] Step 420: From the candidate image restoration schemes for the corresponding region of interest, select the image restoration scheme with the highest restoration success rate and / or the best image restoration effect as the image restoration scheme for the corresponding region of interest. In some embodiments, the image restoration scheme with the highest restoration success rate among the candidate image restoration schemes can be selected as the determined image restoration scheme, which can make the determined image restoration scheme have a higher execution success rate and stability in subsequent execution, avoiding possible repeated adjustments to the image restoration scheme later. In some embodiments, in some scenarios with high requirements for image restoration, the image restoration scheme with the best image restoration effect among the candidate image restoration schemes can also be selected as the determined image restoration scheme, which can obtain theoretically better image restoration results. In some embodiments, those skilled in the art can also adjust the selection weights of restoration success rate and image restoration effect according to actual needs, and select the image restoration scheme with both high restoration success rate and good image restoration effect as the determined image restoration scheme, which is not limited here.

[0067] In some embodiments, the experience knowledge base may further include image restoration operation execution order rules and / or image restoration operation warning rules. The image restoration operation execution order rules can be a series of dependencies between image restoration operations derived from summarizing historical restoration records of sample images. These dependencies can prevent negative impacts on the image restoration quality of the relevant areas during the specific execution of image restoration operations. For example, in some specific instances, there may be incompatibility issues between specific image denoising restoration tools and image deblurring restoration tools. Applying both of these tools simultaneously in the same image restoration scheme may result in the image not achieving the expected restoration effect, or even further degrading the image quality. In such cases, image restoration operation execution order rules can be generated to record conflicts between different image restoration operations and serve as a dependency that the image restoration scheme must adhere to. In other specific instances, based on historical restoration records, it can be observed that in scenarios requiring simultaneous image denoising and deblurring operations, better image restoration results are often achieved when denoising is performed before deblurring. Conversely, when denoising occurs after deblurring, the restoration scheme involving both operations may fail. In such cases, an image restoration execution order rule can be generated, establishing "denoising precedes deblurring" as a mandatory dependency for the restoration scheme. In some embodiments, this rule may include the order of execution of multiple image restoration operations and / or incompatibility rules between them, which are not limited here. In some embodiments, the execution order rule can be obtained by traversing and summarizing failed historical restoration records based on a pre-defined machine learning model. Those skilled in the art can also choose other suitable techniques to summarize historical restoration records to obtain the required execution order rule, which is not limited here.

[0068] In some embodiments, when the experience knowledge base includes image restoration operation execution order rules, the image restoration scheme corresponding to the region of interest can be verified based on these rules when determining the image restoration scheme for the region of interest. Specifically, the verification process may include determining whether the execution order of each image restoration operation in the scheme conforms to the image restoration operation execution order rules in the experience knowledge base, which is not limited here. In some embodiments, if the image restoration scheme conforms to the image restoration operation execution order rules in the experience knowledge base, it means the image restoration scheme has passed verification and can proceed with the subsequent specific image restoration process. In some embodiments, if the image restoration scheme does not conform to at least one image restoration operation execution order rule in the experience knowledge base, it means the image restoration scheme has failed verification. If this image restoration scheme is used for restoration, the probability of subsequent image restoration failure or further reduction in the image quality of the corresponding region of interest is high, and it is necessary to re-determine the image restoration scheme based on the experience knowledge base.

[0069] In some embodiments, image restoration operation early warning rules can be a series of early warning information derived from summarizing historical restoration records of sample images. This early warning information can provide advance warning of potential image quality changes caused by specific image restoration operations when the image restoration scheme includes such operations. For example, in some specific instances, based on historical restoration records, it can be found that when using the Restormer-V3 image restoration tool and configuring its denoising intensity to level 5 or greater, high-frequency detail loss is likely to occur after image restoration. In this case, an image restoration operation early warning rule can be generated, using "the denoising intensity level of the Restormer-V3 tool is greater than or equal to 5" as the trigger condition for the early warning rule, and configuring a preset early warning message for "high-frequency detail loss". In some embodiments, image restoration operation early warning rules can be obtained by traversing and summarizing manually marked image restoration operation risks in historical restoration records based on a preset machine learning model. Those skilled in the art can also choose other suitable technical means to summarize historical restoration records to obtain the required image restoration operation early warning rules, which are not limited here.

[0070] In some embodiments, when the experience knowledge base includes image restoration operation warning rules, when determining the image restoration scheme for each region of interest, based on the image restoration operation warning rules, if there is an image restoration operation that triggers a warning in the image restoration scheme corresponding to the region of interest, corresponding warning information can be provided to remind the user or for the user to refer to. For example, in some application scenarios, when performing image restoration on an image to be restored, the user pays special attention to and needs the high-frequency details in the restored image to be well preserved. In this case, if the generated image restoration scheme contains an image restoration operation with "the noise reduction intensity level of the image restoration tool Restormer-V3 is 5", the user can be provided with warning information that "high-frequency detail loss" may occur. If the user does not want high-frequency detail loss to occur during the image restoration process, they can choose to abandon the execution of the current image restoration scheme according to the warning information and re-determine the image restoration scheme based on the experience knowledge base, which is not limited here.

[0071] In some embodiments, for the aforementioned regions of interest ROI_A and ROI_B, step 220 can be used to obtain an image restoration scheme for ROI_A, which includes a dehazing operation (e.g., using the DehazeFormer_v2 tool with a high processing intensity parameter) and a super-resolution operation (e.g., using the HAT_x4 tool with a resolution enhancement factor of 4); and an image restoration scheme for ROI_B, which includes a denoising operation (e.g., using the Restormer_v3 tool with a noise intensity of 6), a deraining operation (e.g., using the HAT_x4 tool with a rain pattern morphology parameter of "stripes") and a super-resolution operation (e.g., using the HAT_x4 tool with a resolution enhancement factor of 4). This is only used as an example and is not intended to be limiting.

[0072] Step 230: Repair each region of interest based on the corresponding image inpainting scheme. In some embodiments, step 230 can be implemented by the inpainting module 530.

[0073] In some embodiments, based on the corresponding image restoration scheme provided in the foregoing embodiments, each image restoration operation can be performed sequentially according to the execution order to achieve restoration processing of each region of interest. The restoration processing of each region of interest is executed independently and synchronously, provided that computing resources permit.

[0074] In some embodiments, each image restoration operation includes several processing steps executed in a preset order. Considering that the image restoration scheme is determined by selecting highly similar sample degradation scenes from an empirical knowledge base based on the degradation scenes of each region of interest, there may be certain differences between the degradation scenes of the regions of interest and the sample degradation scenes. Furthermore, differences between different regions of interest can lead to different restoration effects, meaning that the image restoration operation cannot guarantee successful restoration of the corresponding region of interest, and restoration failure may also occur. If anomalies are only detected through quality assessment after each image operation has been fully executed, not only will a large amount of computational resources be wasted, but the original image information may also be permanently lost due to irreversible operations. Even if a rollback mechanism is subsequently introduced, the damaged intermediate features cannot be recovered. Therefore, during the execution of the corresponding image restoration operation, a restoration quality assessment can be performed on the intermediate restored image obtained by the monitored current processing step.

[0075] In some embodiments, during the execution of at least one image restoration operation, the restoration quality of the intermediate restored image obtained by the monitored current processing step can be evaluated. In some embodiments, the intermediate restored image refers to the intermediate image obtained during the image restoration process of each region of interest. By evaluating the restoration quality of the intermediate restored images generated during the execution of the image restoration operation, the restoration status of the corresponding processing steps of the image restoration operation can be verified in real time, and the processing steps with unsatisfactory restoration status can be accurately located. The image restoration scheme can then be adjusted in a timely manner to avoid restoration failures after the corresponding image restoration operation is completed. The specific implementation of the restoration quality evaluation will be further explained below with reference to embodiments.

[0076] In some embodiments, a monitor hook can be embedded in advance after the processing steps of the image restoration operation that needs to be monitored. For example, when performing the RainRemoval_v5 operation on the aforementioned region of interest ROI_B, a monitor hook can be embedded after the 2nd, 4th, and 6th Transformer blocks, without limitation. After the monitored processing step is completed, the subsequent monitor hooks are activated and collect information such as the feature tensor, L1 loss, gradient norm, and / or frequency domain energy spectrum of the intermediate restored image obtained after the corresponding processing step is completed. This information is then input into a preset anomaly predictor for analysis, so as to perform preliminary risk prediction. In some embodiments, the core task of the anomaly predictor is to predict the success probability that the output result of the current image restoration operation will meet the quality requirements if it continues to be executed to completion. When the calculated success probability is less than a preset probability threshold, the image quality assessment network is invoked to perform restoration quality assessment on the intermediate restored image obtained by the current processing step; otherwise, the next processing step in the corresponding image restoration operation is executed according to a preset order. In some embodiments, the anomaly predictor is a pre-trained lightweight model, such as a one-dimensional convolutional neural network (1D-CNN) with fewer than 50K parameters.

[0077] In some embodiments, the restoration quality assessment performed by the image quality assessment network may include an evaluation of the execution status of the current processing step and / or an overall image quality assessment of the intermediate restored images. The evaluation of the execution status of the current processing step assesses the restoration effect of the current processing step; the better the restoration effect of the current processing step, the better the elimination of the corresponding image damage in the intermediate restored images. The overall image quality assessment of the intermediate restored images assesses the overall image quality of the intermediate restored images; a good overall image quality of the intermediate restored images indicates that the current processing step has not introduced new or additional image damage to the corresponding region of interest. By evaluating both the execution status of the current processing step and the quality of the intermediate restored images, it is possible to accurately reflect whether the current processing step has passed the restoration quality assessment. In some embodiments, the restoration quality assessment may fail if either the execution status assessment or the overall image quality assessment fails to meet a preset indicator. In some embodiments, the restoration quality assessment may also fail if neither the execution status assessment nor the overall image quality assessment meets the preset indicator. Those skilled in the art can choose the passing standard for the restoration quality assessment according to actual needs, and no limitation is made here.

[0078] In some embodiments, when the restoration quality assessment passes, the next processing step in the corresponding image restoration operation can be executed based on a preset order. In some embodiments, a passing restoration quality assessment indicates that the current processing step can achieve the expected image processing effect without reducing the overall image quality of the intermediate restored image, and the current processing step has successfully restored at least a portion of the image damage in the image to be restored. Subsequent processing steps can then be executed based on a predetermined execution order in the corresponding image restoration operation.

[0079] In some embodiments, if the restoration quality assessment of the preprocessing step fails, the execution of the current processing step is canceled, and the previous valid intermediate restored image obtained from the previous processing step is acquired. Then, based on the previous valid intermediate restored image and the experience knowledge base, the corresponding image restoration scheme is updated. Finally, based on the image restoration operations and execution order included in the updated image restoration scheme, the previous valid intermediate restored image is restored. This process is performed because: a failure to pass the restoration quality assessment indicates that the current processing step cannot achieve the expected image processing effect, or it will reduce the overall image quality of the intermediate restored image. The current processing step has failed to achieve the expected restoration effect on the corresponding region of interest, so the execution of the current processing step can be canceled, and the execution result of the previous processing step can be rolled back to select other image restoration operations for execution.

[0080] In some embodiments, the image restoration scheme can be updated based on historical intermediate restored images and an experience knowledge base. In some embodiments, the image restoration scheme can be updated by retrieving similar degradation scenarios from the experience knowledge base based on historical intermediate restored images and / or degradation scenarios of historical intermediate restored images, which will not be elaborated upon here. In some embodiments, the experience knowledge base also stores various historical restoration records containing cases of failed restoration operations. The image restoration scheme can be updated based on the similarity between these historical restoration records and the current image restoration scheme and the failed image restoration operations, which is not limited here.

[0081] In some embodiments, updating an image restoration scheme includes one or more of the following methods: adjusting the execution order of image restoration operations that were not performed in the image restoration scheme; replacing at least one image restoration operation that was not performed in the image restoration scheme; canceling at least one image restoration operation that was not performed in the image restoration scheme; or adding at least one image restoration operation to the image restoration scheme.

[0082] In some embodiments, when repairing the previous valid intermediate repair image, for each image repair operation in the updated image repair scheme, the obtained intermediate repair image also needs to be repaired for quality evaluation during its execution. If the repair quality evaluation fails again, the current processing step needs to be canceled and the image repair scheme updated again until all image repair operations in the image repair scheme are completed and pass the corresponding repair quality evaluation, so as to achieve a complete image repair process for the image to be repaired.

[0083] In some embodiments, when a processing step in an image restoration scheme fails the restoration quality assessment, in addition to updating the image restoration scheme, the relevant circumstances of the image restoration failure can be recorded so that the experience information in the experience knowledge base can be adaptively updated based on the image restoration failure.

[0084] Step 240: Based on each image restoration region, fuse them to generate the target restoration image. In some embodiments, step 240 can be implemented by the fusion module 540.

[0085] In some embodiments, the image restoration apparatus can establish transition bands at the boundaries of each image restoration region; then, it stitches the image restoration regions together and performs weighted fusion processing on the overlapping image blocks of adjacent image restoration regions within each transition band to generate a target restored image. In some embodiments, a pixel transition band can be constructed by symmetrically extending a preset width of pixels from the boundaries of each image restoration region to both sides of the boundary, for example, with a width of 8 pixels. The purpose of establishing the transition band is to provide a smooth fusion buffer area for the subsequent regions on both sides of the boundary, avoiding abrupt direct stitching. The overlapping image blocks of adjacent image restoration regions within the transition band can be denoted as Image_Patch_A and Image_Patch_B. The target restored image is generated by performing weighted fusion processing on Image_Patch_A and Image_Patch_B.

[0086] In some embodiments, during weighted fusion processing, Laplacian pyramid decomposition is used to decompose overlapping image patches within each transition zone into different frequency layers. Different fusion weights are applied to these overlapping image patches at different frequency layers to achieve high-quality, seamless, adaptive fusion. In some embodiments, a 5-layer Laplacian pyramid can be used. For frequency layers with frequencies higher than a preset frequency threshold (i.e., lower layers, such as layers 0-1), fusion weights are assigned to the corresponding image patches based on the image gradient magnitude. The gradient magnitude directly reflects the sharpness of edges and textures. In transition zones, image patches with larger gradient magnitudes (indicating sharper edges / textures at that location) will receive higher fusion weights, ensuring that the detailed parts of the fused image originate from a higher-quality source. For frequency layers with frequencies lower than the preset frequency threshold (i.e., lower layers, such as layers 2-4), fusion weights are assigned to the corresponding image patches based on the semantic segmentation results. In other words, weights are assigned based on semantic consistency. For example, in pixel regions with the semantic category of "sky," Image_Patch_A (e.g., from the repaired region of interest ROI_A) is given a higher fusion weight; in pixel regions with the semantic category of "building," Image_Patch_B (e.g., from the repaired region of interest ROI_B) is given a higher fusion weight, achieving a smooth transition at semantic boundaries. This ensures that the overall tone and lighting of the fused image conform to the semantic logic of the scene, preventing color misalignment. Then, the final fused image patch Fused_Patch within the transition zone is recovered using the Laplacian pyramid reconstruction algorithm. To further optimize the visual effect, if a discontinuity in the gradient field between Fused_Patch and one side of the original repaired region (e.g., the building region corresponding to Image_Patch_B) is detected at the edge of the transition zone, Poisson fusion post-processing can be performed to achieve seamless fusion of the gradient domain, thereby eliminating any subtle brightness or color jumps. Finally, the system fills the optimized Fused_Patch back into the corresponding transition zone position, replacing the original stitching boundary. All image restoration areas and the transition zones between areas processed as described above are merged to generate a complete and visually consistent target restoration image.

[0087] Figure 5 This is an exemplary structural diagram of an image restoration apparatus according to some embodiments of this specification. In some embodiments, such as... Figure 5 As shown, the image restoration device 500 may include a region sensing module 510, a restoration scheme determination module 520, a restoration module 530, and a fusion module 540.

[0088] In some embodiments, the region-aware module 510 can be used to segment at least two regions of interest with different degradation scenes from the image to be repaired. In some embodiments, the repair scheme determination module 520 can be used to determine an image repair scheme corresponding to each region of interest based on a preset empirical knowledge base. The image repair scheme includes at least one image repair operation and the execution order of each image repair operation. In some embodiments, the repair module 530 can be used to repair each region of interest based on the corresponding image repair scheme to obtain the corresponding image repair region. In some embodiments, the fusion module 540 can be used to fuse the image repair regions to generate a target repaired image.

[0089] For more information on each unit, please refer to [link / reference]. Figures 2-4 The relevant explanations will not be repeated here. It should be understood that, as... Figure 5 The apparatus and its modules or units shown can be implemented in various ways. For example, in some embodiments, they can be implemented by hardware, software, or a combination of both. The hardware portion can be implemented using dedicated logic; the software portion can be stored in memory and executed by an appropriate instruction execution system, such as a microprocessor or dedicated-design hardware. Those skilled in the art will understand that the methods, apparatus, and systems described above can be implemented using computer-executable instructions and / or included in the control code of a processor, such as on a media such as a disk, CD, or DVD-ROM, or in the memory of a programmable device. The apparatus and its modules described herein can be implemented not only by hardware circuitry such as very large-scale integrated circuits or gate arrays, semiconductors such as logic chips or transistors, or programmable hardware devices such as field-programmable gate arrays or programmable logic devices, but also by software, for example, executed by various types of processors, or by a combination of the aforementioned hardware circuitry and software (e.g., firmware).

[0090] It should be noted that the above description of the device, modules, or units is for convenience only and should not be construed as limiting this specification to the embodiments described. It is understood that those skilled in the art, after understanding the principle of the device, can arbitrarily combine the various modules or units without departing from this principle to form sub-devices connected to other modules or units. Alternatively, some modules or units can be split to obtain more modules or units or corresponding multiple sub-units. Such modifications are all within the scope of this specification.

[0091] Figure 6This is a schematic diagram of the hardware architecture of a computer device 600 according to some embodiments of this specification. In some embodiments, the computer device 600 may include a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it can implement the image restoration method provided in any of the foregoing embodiments. In some embodiments, the computer device 600 may be a terminal device such as a smartphone, tablet computer, or personal computer. In other embodiments, the computer device 600 may be a rack server, blade server, tower server, or cabinet server (including a standalone server or a server cluster composed of multiple servers).

[0092] In some embodiments, such as Figure 6 As shown, the computer device 600 includes, but is not limited to, a memory 610, a processor 620, and a network interface 630 that are interconnected via a system bus. The memory 610 includes at least one type of computer-readable storage medium. In some embodiments, the memory 610 may be an internal storage module of the computer device 600, such as a hard disk or RAM. In other embodiments, the memory 610 may also be an external storage device of the computer device 600. Of course, the memory 610 may include both internal and external storage modules of the computer device 600. In this embodiment, the memory 610 is typically used to store the operating system and various application software installed on the computer device 600, such as the program code related to the image restoration method provided in any of the foregoing embodiments. Furthermore, the memory 610 can also be used to temporarily store various types of data that have been output or will be output.

[0093] In some embodiments, processor 620 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other chip. Processor 620 is typically used to control the overall operation of computer device 600, such as performing control and processing related to data interaction or communication with computer device 600. In this embodiment, processor 620 is used to run program code stored in memory 610 or process data.

[0094] In some embodiments, network interface 630 may include a wireless network interface or a wired network interface, which is typically used to establish a communication link between computer device 600 and other computer devices. For example, network interface 630 is used to connect computer device 600 to an external terminal via a network, establishing a data transmission channel and communication link between computer device 600 and the external terminal. The network may be an intranet, the Internet, Global System for Mobile communication (GSM), Wideband Code Division Multiple Access (WCDMA), 4G network, 5G network, Bluetooth, WiFi, or other wireless or wired networks.

[0095] It should be pointed out that, Figure 6 Only computer devices having components 610 to 630 are shown; however, it should be understood that implementation of all shown components is not required, and more or fewer components may be implemented alternatively. In some embodiments, the relevant program code for the image restoration method stored in memory 610 may also be divided into one or more program modules and executed by one or more processors (such as processor 620) to implement the embodiments of this specification, without limitation herein.

[0096] One or more embodiments of this specification also provide a computer-readable storage medium storing a computer program thereon, wherein the computer program, when executed by a processor, implements the image restoration method provided in the foregoing embodiments. In some embodiments, the computer-readable storage medium may include flash memory, a hard disk, a multimedia card, a card-type memory (e.g., SD or DX memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, a magnetic disk, an optical disk, etc. In some embodiments, the computer-readable storage medium may be an internal storage unit of a computer device, such as the hard disk or memory of the computer device. In other embodiments, the computer-readable storage medium may also be an external storage device of a computer device, such as a plug-in hard disk, a smart media card (SMC), a secure digital card (SD), a flash card, etc., provided on the computer device. Of course, the computer-readable storage medium may also include both internal storage units and external storage devices of a computer device. In this embodiment, the computer-readable storage medium is typically used to store the operating system and various application software installed on the computer device, such as the program code of the service access method and / or service invocation method in this embodiment. Furthermore, the computer-readable storage medium can also be used to temporarily store various types of data that have been output or will be output.

[0097] One or more embodiments of this specification also provide a computer program product, including a computer program that, when executed by a processor, can implement the image restoration method provided in any of the foregoing embodiments. In some embodiments, the computer program product may relate to a computer program that may be carried on a storage medium or a processing device. In other embodiments, the computer program product may also be a storage medium or a processing device containing the aforementioned computer program. The processing device may include one or more processors, and a storage medium.

[0098] The basic concepts have been described above. It is obvious that the detailed disclosure above is merely illustrative and does not constitute a limitation of this specification. Although not explicitly stated herein, various modifications, improvements, and corrections may be made to this specification by those skilled in the art. Such modifications, improvements, and corrections are taught in this specification and therefore remain within the spirit and scope of the exemplary embodiments described herein.

Claims

1. An image restoration method, characterized in that, The method includes: Segment at least two regions of interest with distinct degradation scenes from the image to be restored; Based on a pre-set knowledge base, an image restoration scheme corresponding to each of the regions of interest is determined. The image restoration scheme includes at least one image restoration operation and the execution order of each image restoration operation. Based on the corresponding image restoration scheme, each region of interest is restored to obtain the corresponding image restoration region; Based on each of the image restoration regions, a target restored image is generated by fusion.

2. The method according to claim 1, characterized in that, The process of segmenting at least two regions of interest with different degradation scenes from the image to be repaired includes: Obtain the degradation analysis results and semantic segmentation results of the image to be repaired; Based on the degradation analysis results and the semantic segmentation results, the region of interest is segmented; The degradation analysis results include degradation index data and degradation distribution of different degradation types in the image to be repaired, and the semantic segmentation results include the location information of different semantic regions in the image to be repaired.

3. The method according to claim 2, characterized in that, The process of segmenting the region of interest based on the degradation analysis results and the semantic segmentation results includes: Based on the degradation analysis results and the semantic segmentation results, the feature vectors of each pixel in the image to be repaired are obtained. Cluster the pixels according to their feature vectors, and obtain the regions of interest and the degradation scenes of the regions of interest based on the clustering results. The degradation scenario includes the semantic category, degradation type, and degradation degree corresponding to each region of interest.

4. The method according to claim 3, characterized in that, The step of obtaining the feature vector of each pixel in the image to be repaired based on the degradation analysis results and the semantic segmentation results includes: Based on the degradation analysis results, the degradation features corresponding to each pixel are obtained; Based on the semantic segmentation results, the semantic features corresponding to each pixel are obtained; The degradation features and semantic features corresponding to each pixel are fused to generate a feature vector for each pixel.

5. The method according to claim 2, characterized in that, The process of obtaining the degradation analysis results and semantic segmentation results of the image to be repaired includes: The image to be repaired is processed using a preset visual language model to obtain the degradation type and semantic segmentation result of the image to be repaired, and a degradation heatmap that can characterize the degradation distribution corresponding to the degradation type is generated. Using a preset image quality assessment network, degradation index data for different degradation types in the image to be repaired are obtained based on the degradation heatmap corresponding to each degradation type.

6. The method according to claim 1, characterized in that, The experience knowledge base is generated based on historical restoration records of sample images; The experience knowledge base includes multiple image restoration strategies, which include sample degradation scenarios, corresponding image restoration schemes, and image restoration effects.

7. The method according to claim 6, characterized in that, The process of determining image restoration schemes corresponding to each region of interest based on a preset knowledge base includes: Based on the degradation scenarios of each region of interest, the degradation scenarios of the samples with a similarity higher than a preset threshold are matched from the experience knowledge base, and the image restoration schemes corresponding to the matched degradation scenarios are used as candidate image restoration schemes for the corresponding regions of interest. From the candidate image restoration schemes for the corresponding region of interest, select the image restoration scheme with the highest restoration success rate and / or the best image restoration effect as the image restoration scheme for the corresponding region of interest.

8. The method according to claim 6, characterized in that, The experience knowledge base also includes image restoration operation execution order rules and / or image restoration operation early warning rules; The step of determining the image restoration scheme corresponding to each of the regions of interest based on a preset experience knowledge base further includes: Based on the image restoration operation execution order rules, verify the image restoration scheme corresponding to each region of interest; and / or Based on the image restoration operation warning rules, when there is an image restoration operation that triggers a warning in the image restoration scheme corresponding to each region of interest, corresponding warning information is provided.

9. The method according to claim 1, characterized in that, Each of the image restoration operations includes processing steps executed in a preset order; The repair of each region of interest based on the corresponding image restoration scheme includes: Based on the corresponding image restoration scheme, each image restoration operation is performed according to the corresponding execution order to restore each region of interest; During the execution of at least one of the image restoration operations, the restoration quality of the intermediate restored image obtained by the monitored current processing step is evaluated. When the restoration quality assessment is passed, the next processing step in the corresponding image restoration operation is executed according to the preset order.

10. The method according to claim 9, characterized in that, The repair of each region of interest based on the corresponding image restoration scheme further includes: If the repair quality assessment fails, the execution of the current processing step is canceled, and the previous valid intermediate repair image obtained from the previous processing step is acquired. Based on the previous effective intermediate repaired image and the experience knowledge base, update the corresponding image repair scheme; and Based on the image restoration operations and execution order included in the updated image restoration scheme, the previous valid intermediate restoration image is restored.

11. The method according to claim 1, characterized in that, The step of fusing and generating a target restored image based on each of the image restoration regions includes: A transition zone is established at the boundary of each of the image restoration areas; The image restoration regions are stitched together, and the overlapping image blocks of adjacent image restoration regions within each transition zone are subjected to weighted fusion processing to generate the target restored image.

12. The method according to claim 11, characterized in that, The weighted fusion process for overlapping image blocks of adjacent image restoration regions within each of the transition zones includes: The overlapping image blocks within each transition zone are decomposed into different frequency layers using Laplacian pyramid decomposition, and the overlapping image blocks are weighted and fused using different fusion weights at the different frequency layers.

13. The method according to claim 12, characterized in that, For frequency layers with frequencies higher than a preset frequency threshold, the fusion weights are assigned to the corresponding image blocks based on the image gradient magnitude. For frequency layers with frequencies lower than the preset frequency threshold, the fusion weights are assigned to the corresponding image blocks based on the semantic segmentation results.

14. An image restoration device, characterized in that, include: A region-aware module is used to segment at least two regions of interest with different degradation scenes from the image to be restored; The scheduling module is used to determine the image restoration scheme corresponding to each of the regions of interest based on a preset experience knowledge base. The image restoration scheme includes at least one image restoration operation and the execution order of each image restoration operation. The repair module is used to repair each region of interest based on the corresponding image repair scheme to obtain the corresponding image repair region; The fusion module is used to fuse the images based on the respective image restoration regions to generate a target restoration image.

15. A computer device, characterized in that, The computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements the method of any one of claims 1 to 13.

16. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method described in any one of claims 1 to 13.

17. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the method described in any one of claims 1 to 13.

Images