An image highlight removal method, system, storage medium and program product

Abstract

The application discloses a kind of image highlight removal method, system, storage medium and program product, it is related to general image data processing or generation field, the method comprises: receiving input image, conversion to luminance chroma separation color space, identify highlight pixel area, generate highlight mask;With the boundary of highlight mask as reference outward expansion, construct annular analysis area;Calculate pixel gradient amplitude and pixel gradient direction, generate linear structure feature dataset;Calculate pixel color mean and pixel color standard deviation, generate abnormal area feature dataset;Based on the direction information in linear structure feature dataset and abnormal area feature dataset, defect distribution inference is generated Structure priori guidance map;According to structure priori guidance map, process highlight pixel area to obtain output image, source patch constraint is carried out.The application is implemented, can optimize highlight removal process, guarantee the integrity of equipment defect feature in image.

Description

Technical Field

[0001] This application relates to the general field of image data processing or generation, and more particularly to an image highlight removal method, system, storage medium, and program product. Background Technology

[0002] Intelligent visual monitoring systems are widely used in power grids, power plants, and substations to achieve automatic inspection of equipment status and fault early warning. In outdoor environments, the surfaces of monitored equipment (insulators, transformers, and switchgear, etc.) are mostly made of ceramic or smooth metal, which easily form specular reflections under natural outdoor sunlight, creating bright areas. These bright areas are extremely bright, often saturating the image sensor and completely obscuring the details of the equipment surface below the highlights.

[0003] In related technologies, a specular removal method is used. This method converts the image from the RGB color space to a luminance-chrominance separated color space, segments the specular pixels in the luminance channel, and generates an initial specular mask. To ensure a smooth transition at the edges of the repaired area, a Gaussian filter is typically used to blur this initial mask. The luminance channel is then repaired based on the blurred mask to generate the final image.

[0004] However, when oil seepage occurs on the transformer tank surface, the fresh oil stains exhibit bright reflections. Related technologies often use the texture of the clean metal surface surrounding the highlights to repair them, resulting in the oil stain area appearing smaller or even eliminated in the processed image. Similarly, when there are fine cracks on the insulator surface, related technologies may cover the linear highlights created by the cracks themselves with the surrounding intact glaze texture. Therefore, it is evident that these technologies, in removing highlights, can lead to missed defects due to blurring and repair processes. Summary of the Invention

[0005] This application provides an image highlight removal method, system, storage medium, and program product for optimizing the highlight removal process and ensuring the integrity of device defect features in the image.

[0006] In a first aspect, this application provides an image highlight removal method applied to an image processing system. The method includes: receiving an input image; converting the input image to a color space with luminance and chrominance separation; identifying highlight pixel regions in the luminance channel that exceed a preset luminance threshold and generating a highlight mask; constructing a ring-shaped analysis region surrounding the highlight pixel regions by extending a preset pixel distance outward from the boundary of the highlight mask; calculating the pixel gradient magnitude and pixel gradient direction within the ring-shaped analysis region, identifying high-gradient pixels, and generating a linear structure feature dataset; calculating the pixel color mean and pixel color standard deviation within the ring-shaped analysis region, identifying color-abnormal pixels, and generating an abnormal region feature dataset; performing defect distribution inference based on the direction information in the linear structure feature dataset and the abnormal region feature dataset to generate a structural prior guidance map; applying source patch constraints according to the structural prior guidance map, and processing the highlight pixel regions using a sample block-based image inpainting algorithm to obtain an output image.

[0007] In the above embodiments, the image processing system analyzes the structural and color features of the surrounding area before repairing the highlights, and generates a structural prior guidance map to constrain the repair process. This avoids the potential defects that may exist by directly using the texture of the intact area to cover the highlights, thereby ensuring the integrity of equipment defects such as oil stains and cracks after highlight removal and improving the accuracy of subsequent defect identification.

[0008] In conjunction with some embodiments of the first aspect, in some embodiments, before the steps of performing source patch constraints based on the structural prior guidance map, processing the highlight pixel region using a sample block-based image inpainting algorithm, and obtaining the output image, the method further includes: performing morphological analysis on the structural prior guidance map to determine whether the potential defect type of the highlight pixel region is a linear defect or a planar defect, and generating a defect type label; selecting a matching specialized inpainting model from a preset inpainting model library based on the defect type label, and using the specialized inpainting model as the image inpainting algorithm.

[0009] In the above embodiments, the image processing system automatically selects the optimal specialized repair model based on the inferred defect type (linear or planar). Compared with the general model, it can handle defects of different shapes more precisely, improve the repair quality of highlight areas and the restoration of defect details, and make the repair results more realistic and natural.

[0010] In conjunction with some embodiments of the first aspect, in some embodiments, the step of constructing a ring-shaped analysis region surrounding the highlight pixel region by extending a preset pixel distance outward based on the boundary of the highlight mask specifically includes: performing connected component analysis on the highlight mask to calculate the area and the maximum bounding rectangle of the highlight pixel region; determining the preset pixel distance through a preset mapping function based on the area and the length of the long side of the maximum bounding rectangle; and extending the preset pixel distance outward with the boundary of the highlight mask as the inner boundary to obtain the outer boundary, thereby constructing a ring-shaped analysis region enclosed by the inner and outer boundaries.

[0011] In the above embodiments, the image processing system dynamically adjusts the size of the annular analysis region according to the area and size of the highlight region, ensuring that the analysis range can contain sufficient effective information while avoiding the introduction of irrelevant interference, thereby improving the accuracy of feature extraction and the reliability of subsequent defect reasoning, and enhancing the universality of the method.

[0012] In conjunction with some embodiments of the first aspect, in some embodiments, after performing source patch constraints based on the structural prior guidance map, processing the highlight pixel region using a sample block-based image inpainting algorithm, and obtaining the output image, the method further includes: displaying the structural prior guidance map and the input image overlaid on a user interface; receiving editing operations performed by the user on the structural prior guidance map, and updating the structural prior guidance map to a structural prior correction map, so as to use the structural prior correction map for source patch constraints.

[0013] In the above embodiments, the image processing system incorporates a user interaction element, allowing professionals to edit and correct the automatically generated structural prior guidance map on a visual interface. This compensates for potential misjudgments by the algorithm in complex or blurry scenes, enhancing the robustness of specular restoration and the accuracy of the final result.

[0014] In conjunction with some embodiments of the first aspect, in some embodiments, after receiving the user's editing operation on the structural prior guidance map and updating the structural prior guidance map to a structural prior correction map for using the structural prior correction map to perform source patch constraints, the method further includes: based on the structural prior correction map, performing secondary repair processing on the highlight pixel region using a sample block-based image inpainting algorithm to generate a corrected output image; calculating the integrity score and repair quality score of the defect region in the output image and the corrected output image respectively to generate an image repair quality assessment result; storing the editing operation and the quality assessment result in a user interaction database to construct an editing-repair correlation model.

[0015] In the above embodiments, the image processing system associates and stores the user's editing operations with the restoration quality assessment results, constructing a continuously learning editing-restoration association model. This model can continuously optimize the algorithm, enabling the system to learn from manual corrections and improve the intelligence and accuracy of future automatic restorations.

[0016] In conjunction with some embodiments of the first aspect, in some embodiments, after performing source patch constraints based on the structural prior guidance map, processing the highlight pixel region using a sample block-based image inpainting algorithm to obtain the output image, the method further includes: identifying defect regions in the output image, extracting the geometric topological parameters and texture feature parameters of the defect regions, and generating a defect feature data package; calculating the position, shape, and brightness distribution of the highlight pixel region, and generating a highlight feature data package; combining the defect feature data package and the highlight feature data package to generate an observation feature vector; and appending a timestamp of the current processing and the identifier of the monitored device to the observation feature vector to generate a multi-dimensional state snapshot.

[0017] In the above embodiments, the image processing system structures the single processing result and generates a multi-dimensional status snapshot containing defect features, highlight information, time and device identifier. This provides a standardized and information-rich data foundation for longitudinal historical tracing and multi-dimensional correlation analysis of device status, and realizes refined management of device status.

[0018] In conjunction with some embodiments of the first aspect, in some embodiments, after the step of attaching a multi-dimensional state snapshot to the observed feature vector containing the current processing timestamp and the identifier of the monitored device, the method further includes: retrieving multiple historical multi-dimensional state snapshots that match the identifier of the monitored device, and sorting them chronologically to obtain a set of historical state snapshots; extracting the defect feature data packet of each snapshot in the set of historical state snapshots to generate a defect evolution sequence; extracting the highlight feature data packet of each snapshot in the set of historical state snapshots to generate an illumination environment evolution sequence; and determining the image re-inspection time based on the illumination environment evolution sequence when the defect region is determined to be in a stable state based on the defect evolution sequence.

[0019] In the above embodiments, the image processing system can track the evolution trend of defects by analyzing historical snapshot sequences. When the defects are stable, it can also intelligently recommend the optimal re-inspection time by combining the changes in lighting environment, realizing the transformation from passive repair to proactive predictive maintenance and improving inspection efficiency.

[0020] In a second aspect, embodiments of this application provide an image processing system comprising: one or more processors and a memory; the memory is coupled to the one or more processors and is used to store computer program code, the computer program code including computer instructions, wherein the one or more processors invoke the computer instructions to cause the image processing system to perform the method described in the first aspect and any possible implementation thereof.

[0021] Thirdly, embodiments of this application provide a computer program product containing instructions that, when the computer program product is run on an image processing system, cause the image processing system to perform the method described in the first aspect and any possible implementation thereof.

[0022] Fourthly, embodiments of this application provide a computer-readable storage medium including instructions that, when executed on an image processing system, cause the image processing system to perform the method described in the first aspect and any possible implementation thereof.

[0023] Understandably, the image processing system provided in the second aspect, the computer program product provided in the third aspect, and the computer storage medium provided in the fourth aspect are all used to execute the methods provided in the embodiments of this application. Therefore, the beneficial effects they can achieve can be referred to the beneficial effects in the corresponding methods, and will not be repeated here.

[0024] One or more technical solutions provided in the embodiments of this application have at least the following technical effects or advantages:

[0025] 1. By employing a technique that analyzes the features of the region surrounding the highlight before repairing it, and by constructing a ring-shaped analysis region to calculate pixel gradients to generate a linear structure feature dataset and pixel color features to generate an anomaly region feature dataset, this approach can infer the possible defect distribution within the highlight region based on reliable information from outside the highlight region, generating a structural prior guidance map. This guidance map is used to constrain subsequent image inpainting algorithms based on sample blocks, prioritizing patches with inferred defect structures rather than blindly selecting surrounding intact textures. This effectively solves the problem in existing technologies where blurring and indiscriminate repair lead to the incorrect repair or elimination of equipment defects (such as oil stains and cracks) under highlights, resulting in missed detections. Furthermore, it achieves effective highlight removal while preserving and restoring the original features of equipment defects to the greatest extent possible, ensuring the integrity of image information.

[0026] 2. By employing a morphological analysis of the prior structural guidance map to determine the defect type and then selecting a specialized repair model from a pre-set library, the system can further refine the inferred defect information into specific defect morphologies, such as elongated linear defects (corresponding to cracks) or regional planar defects (corresponding to oil stains). For these two distinct defect patterns, the system can utilize specially optimized repair algorithms. For instance, the model for linear defects excels at maintaining structural continuity and sharpness, while the model for planar defects focuses more on smooth texture transitions and color consistency. This effectively solves the problem of existing technologies using a single, general repair model to handle all situations, resulting in poor repair effects, such as blurred linear cracks or artifacts at the edges of planar oil stains. It achieves differentiated and refined repair of different types of defects, improving the image quality and realism after highlight removal, making the repair results more consistent with human visual habits and professional diagnostic requirements.

[0027] 3. By employing a technical solution that overlays and displays the structural prior guidance diagram on the user interface and receives user editing operations to generate a structural prior correction diagram, this application combines the machine's automatic reasoning ability with human professional judgment. The algorithm-generated guidance diagram serves as an initial suggestion, presented to domain experts. Experts can correct potential deviations in the algorithm based on their experience and understanding of complex scenarios through simple drawing operations (such as adding, deleting, or correcting lines or regions). This manually corrected guidance diagram is undoubtedly more accurate and reliable than the original automatically generated diagram. This effectively solves the problems of insufficient robustness, susceptibility to misjudgment, and lack of error correction mechanisms in existing fully automated processes when facing ambiguity, noise, or atypical defects. It thus achieves a human-machine collaborative repair mode, improving the reliability and accuracy of the method in complex and critical scenarios. Attached Figure Description

[0028] Figure 1 This is a schematic flowchart of an image highlight removal method in an embodiment of this application;

[0029] Figure 2 This is another schematic flowchart of the image highlight removal method in the embodiments of this application;

[0030] Figure 3 This is a schematic diagram of the physical device structure of an image processing system in an embodiment of this application. Detailed Implementation

[0031] The terminology used in the following embodiments of this application is for the purpose of describing particular embodiments only and is not intended to be limiting of this application. As used in the specification of this application, the singular expressions “a,” “an,” “the,” “the,” and “this” are intended to include the plural expressions as well, unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in this application refers to any or all possible combinations including one or more of the listed items.

[0032] Hereinafter, the terms "first" and "second" are used for descriptive purposes only and should not be construed as implying or suggesting relative importance or implicitly indicating the number of indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature, and in the description of the embodiments of this application, unless otherwise stated, "multiple" means two or more.

[0033] In the implementation scenarios of this application, such as intelligent inspection of power equipment, many key equipment such as insulators, transformers, and transformer housings have smooth surfaces that are prone to gloss under outdoor sunlight, obscuring their surface condition.

[0034] The technical terms used in this application are proposed in this context. A color space with luminance and chrominance separation, such as YCrCb or Lab space, refers to separating the luminance information (Y or L channel) from the color information (CrCb or ab channel) of an image. This allows for the separate processing of excessively bright highlights without affecting the image's color. A highlight mask is a binary image where white pixels precisely mark the areas in the input image identified as highlights, serving as the target area for subsequent processing. A ring-shaped analysis region is a strip-shaped area constructed around the boundary of the highlight mask, providing reliable, uncontaminated contextual information. Pixel gradients, particularly their magnitude and direction, describe the severity and direction of local brightness changes in the image, and are crucial for detecting linear structures such as edges and cracks; this information constitutes a linear structure feature dataset. Pixel color mean and standard deviation are used to quantify the color distribution within a region; regions with color anomalies (such as oil stains) will exhibit different statistical characteristics, which constitute an anomaly region feature dataset.

[0035] The structural prior guidance map is the core of this application. It is an image that integrates the results of the linear and regional feature inferences described above, depicting a blueprint of the most likely structures (such as crack direction or oil stain extent) beneath the highlights. Finally, the source patch constraint is used in sample block-based image inpainting (a technique that fills missing areas by copying small patches of pixels from other parts of the image) to limit and guide the inpainting process, ensuring that the inpainted content is consistent with the inferred underlying structure, rather than being arbitrarily filled.

[0036] The following describes the process of the method provided in this implementation. Please refer to [link / reference]. Figure 1 This is a flowchart illustrating an image highlight removal method in an embodiment of this application.

[0037] S101. Receive the input image, convert the input image to a color space with luminance and chrominance separation, identify the highlight pixel area above the preset luminance threshold in the luminance channel, and generate a highlight mask.

[0038] The input image refers to the equipment status image captured by intelligent inspection equipment (such as cameras mounted on drones or robots). A color space representation with luminance-chrominance separation, such as YCrCb, HSV, or Lab color models, has the advantage of decoupling luminance and color information. A specular mask is a binary image used to mark the locations of pixels in the input image that are saturated due to excessive reflection.

[0039] Specifically, when the image processing system receives a device image to be processed, it first performs a color space conversion, such as converting from the common RGB space to the YCrCb space. The purpose of this step is to allow subsequent operations to analyze and process only the luminance (Y) channel without affecting the original color (Cr, Cb) information. Next, the system applies a preset luminance threshold (e.g., 240 for an 8-bit image) to the Y channel, identifies all pixels with luminance values ​​higher than this threshold as highlight pixels, and records the positions of these pixels to form an initial binary highlight mask.

[0040] In some embodiments, highlight pixel identification can be achieved in several ways: Optionally, a global fixed thresholding method can be used, applying a uniform brightness value as a threshold for segmentation across the entire brightness channel image; this method is simple and fast. Optionally, an adaptive thresholding method can be used, such as the Otsu algorithm or the local mean method, dynamically calculating the threshold based on the brightness distribution of local regions of the image, which can better adapt to images under different lighting conditions. It is understood that other methods, such as machine learning-based segmentation models, can also be used to identify highlight regions; this is not limited here.

[0041] In some embodiments, a preset brightness threshold needs to be selected. If the threshold is too high, not all highlight areas may be identified, resulting in incomplete restoration; if the threshold is too low, some normal, high-brightness areas may be misidentified as highlights. To solve this problem, a dynamic threshold adjustment strategy can be adopted: the system first generates a mask using a suitable initial threshold, and then analyzes the saturation characteristics of the mask area. If color information still exists in the highlight area (i.e., not fully saturated), the threshold is appropriately increased; if there are a large number of misidentified unsaturated highlight areas around the highlight area, the threshold is appropriately decreased, and the optimal highlight mask is obtained through iterative fine-tuning.

[0042] S102. Extend a preset pixel distance outward from the boundary of the highlight mask to construct a ring-shaped analysis area surrounding the highlight pixel region.

[0043] The boundary of the specular mask refers to the boundary between the specular and non-spectral regions. The preset pixel distance defines the width of the analysis region. The annular analysis region is a strip of pixels enclosed by the specular mask boundary (inner boundary) and a new boundary formed by its outward expansion (outer boundary). Pixels within this region are considered reliable and usable for analysis.

[0044] Specifically, after generating the specular mask, the image processing system needs to determine a region for analyzing contextual information. Since the specular region itself loses information, and regions too far from the specular area may be irrelevant to the texture beneath it, a ring-shaped region immediately surrounding the specular area is most suitable. The system uses the outline of the specular mask generated in step S101 as the inner boundary and extends it outward by a fixed pixel distance (e.g., 15 pixels) to form the outer boundary. The region between these two boundaries is the ring-shaped analysis region, and subsequent feature extraction will only be performed within this region.

[0045] In some embodiments, the construction of the annular analysis region can be achieved in several ways: Optionally, morphological dilation is used to dilate the specular mask image a specified number of times, and then the original mask is subtracted from the dilated mask to obtain the difference set, which is the annular region; Optionally, the contour coordinates of the specular mask are extracted first, and then for each contour point, the outer boundary point set is obtained by extending it outward along its normal direction by a preset distance, thereby constructing the annular region. It is understood that other methods such as distance transformation can also be used to implement this step, which are not limited here.

[0046] In some embodiments, the fixed nature of the preset pixel distance has limitations. For very small specular highlights, a fixed extension distance may be too large, introducing irrelevant background information; while for very large specular highlights, the distance may be too small, failing to contain sufficient effective texture information. To address this issue, the distance can be designed as a dynamic value related to the specular size. For example, the area or maximum bounding rectangle of the specular region can be calculated first, and then the size can be mapped to a suitable extension distance using a mapping function (such as a linear function), achieving adaptive analysis of the analysis region size.

[0047] S103. Calculate the pixel gradient magnitude and pixel gradient direction within the annular analysis region, identify high gradient pixels, and generate a linear structure feature dataset.

[0048] In this dataset, the pixel gradient magnitude represents the rate at which image brightness changes at a point; a larger magnitude indicates a more drastic change, typically corresponding to edges or lines. The pixel gradient direction represents the direction of the fastest brightness change. The linear structure feature dataset stores the locations of pixels with edge features (i.e., high gradients) and their gradient directions within a circular analysis region.

[0049] Specifically, within the annular analysis region constructed in step S102, the image processing system calculates the gradient for each pixel. This is typically accomplished through convolution operations, such as using the Sobel or Scharr operators to calculate the gradient components in the horizontal and vertical directions, respectively, and then calculating the total gradient magnitude and direction based on these two components. The system then sets a gradient magnitude threshold, identifying pixels with magnitudes higher than this threshold as high-gradient pixels, which are considered to represent potential cracks, scratches, or object edges. The locations of these high-gradient pixels and their corresponding gradient direction information are collected to form a linear structure feature dataset.

[0050] In some embodiments, high-gradient pixel recognition can be achieved in several ways: Optionally, the Sobel operator can be used, which is fast and effectively detects edges; alternatively, the Canny edge detection algorithm can be used, which includes steps such as Gaussian filtering, non-maximum suppression, and double thresholding, and can generate more accurate and finer single-pixel edges, but the calculation is slightly more complex. It is understood that other gradient calculation methods or edge detection algorithms can also be used, and no limitation is made here.

[0051] In some embodiments, noise interference exists. Random noise in the image can also produce high gradient magnitudes, which may be misidentified as linear structures. To address this issue, a slight smoothing filter (such as Gaussian filtering or median filtering) can be applied to the circular analysis region before calculating the gradient to suppress noise. Furthermore, post-processing can be performed after generating the dataset, such as removing isolated points without neighboring high-gradient pixels, or retaining only high-gradient pixels that can be connected into chains of a certain length, thereby improving the reliability of linear structure features.

[0052] S104. Calculate the mean and standard deviation of pixel colors within the annular analysis area, identify pixels with abnormal colors, and generate an abnormal area feature dataset.

[0053] The pixel color mean and standard deviation are statistics calculated on the chroma channels (such as Cr and Cb channels), used to describe the average hue and consistency of color variation within a local area. Color anomalous pixels are pixels whose color statistical characteristics differ from most of their surrounding local neighborhood. The anomalous region feature dataset is a collection storing the location information of these color anomalous pixels.

[0054] Specifically, in parallel or sequential with step S103, the image processing system also performs analysis within the annular analysis region, but this time on the chroma channels. The system takes a small neighborhood window (e.g., 5x5) centered on each pixel and calculates the color mean and standard deviation of all pixels within that window in the Cr and Cb channels. By comparing this with the global color statistics of the entire annular analysis region or a preset normal color range, if the color characteristics of a pixel's neighborhood deviate significantly from the normal range (e.g., large difference in color mean or high standard deviation), the pixel is identified as a color aberration pixel, potentially corresponding to surface defects such as oil stains or rust. The locations of all these aberration pixels are recorded, forming an aberration region feature dataset.

[0055] In some embodiments, the identification of pixels with abnormal colors can be achieved in several ways: Optionally, based on statistical methods, the Mahalanobis distance between the local window and the global background is calculated, and if the distance is greater than a threshold, it is judged as abnormal; Optionally, based on clustering methods, K-Means or other clustering methods are applied to the pixel colors within the annular region, and the cluster with the fewest pixels or the cluster furthest from the main cluster is identified as an abnormal region. It is understood that more complex models such as Gaussian Mixture Models (GMM) can also be used for identification, which is not limited here.

[0056] In some embodiments, the impact of illumination variations on color needs to be considered. Even in the chroma channel, slight changes in illumination can cause color values ​​to drift, making anomaly detection based on fixed thresholds unreliable. To address this issue, a strategy of relative comparison rather than absolute comparison can be employed. For example, instead of comparing a local color to a fixed "normal" value, it can be compared to the colors of multiple reference blocks at different orientations within a ring-shaped analysis region. If the color of a region differs from all reference blocks, it is more likely to be classified as an anomaly, and this approach is more robust to global illumination variations.

[0057] S105. Based on the directional information in the linear structural feature dataset and the abnormal region feature dataset, perform defect distribution reasoning and generate a structural prior guidance map.

[0058] Defect distribution inference refers to combining linear and planar features to infer their extension and distribution within the highlight area. The structural prior guidance map is a grayscale or multi-channel image of the same size as the input image, whose pixel values ​​encode the probability or type of each point within the highlight area belonging to a potential defect (such as a crack, edge, or stain).

[0059] Specifically, the image processing system enters the inference phase. It first analyzes the linear structural features obtained in step S103, particularly those high-gradient pixels located at the boundaries of the annular region (i.e., highlight edges). Using the gradient direction information of these pixels, the system linearly extends into the highlight region, predicting the possible paths of cracks or edges under the highlight. Simultaneously, it analyzes the anomalous region features obtained in step S104. If a color anomaly region intersects with a highlight region, it infers that the anomalous region will extend into the highlight region. Finally, the system fuses these two inference results to generate a structural prior guidance map. For example, the predicted crack path pixel value in the image is set to 255, the predicted oil stain region pixel value is set to 128, and other regions are set to 0.

[0060] It should be noted that the working principle of defect distribution inference lies in "propagating" or "extending" the features observed within the annular analysis region to the highlight regions where information is missing. This process can be broken down into separate processing of linear structures and anomalous regions, followed by subsequent fusion.

[0061] For linear structure inference, a geodesic path planning-based algorithm can be used. First, high-gradient pixels detected on the boundary of the annular analysis region (from the linear structure feature dataset) are used as "starting points." Then, a cost field is constructed within the region defined by the specular mask. The cost C(p, q) for any pixel p to move to its neighboring pixel q within the field is defined as a function related to the consistency of the gradient direction, for example, C(p, q) = 1 + α*f(|θ_p - angle(qp)|), where θ_p is the estimated structural direction at point p (initially assigned by the gradient direction of the starting point and continuously updated during propagation), angle(qp) is the vector angle from p to q, f is a monotonically increasing function (such as a square function), and α is a weighting coefficient. This cost function implies that the movement along the current structural direction has the lowest cost. Using shortest path algorithms such as Dijkstra's or A*, the minimum cumulative cost path to all other points within the specular region is calculated from each starting point. These paths constitute the predicted extension trajectory of the crack or edge under the specular light.

[0062] For inference of anomalous regions, methods based on level sets or fast marching can be employed. Color-abnormal pixels identified on the boundary of the annular analysis region (from the anomalous region feature dataset) are used as "seed regions" or initial contours. A velocity function F(x, y) is then defined, whose value is inversely proportional to the probability that the point belongs to the background (normal region). Within the annular analysis region, this probability can be calculated based on color statistical features (such as the Mahalanobis distance to the global color mean). Starting from the seed region, the contour line evolves inward into the highlight region at a velocity defined by F(x, y). This evolution naturally fills in areas similar to the external anomalous region features until it encounters the other boundary of the highlight region or the velocity function value becomes very small (indicating entry into the inferred normal region). The area finally enclosed by the contour line represents the inferred area of ​​the planar defect.

[0063] Finally, the two inference results are fused to generate a structural prior guidance map. For example, on a blank image the same size as the input image, the pixel values ​​on the inferred linear structural paths are set to a high value (e.g., 255), and the pixel values ​​in the inferred abnormal regions are set to a medium value (e.g., 128). If there is overlap, the final pixel value can be determined according to a preset priority (usually linear structures have a higher priority).

[0064] In some embodiments, this step can also be implemented using a specially trained deep learning model, such as a convolutional neural network with a U-Net structure.

[0065] First, a large-scale training dataset needs to be constructed. The dataset generation process is as follows: select a large number of clear device images without highlights as raw samples; on each image, highlight regions are artificially or programmatically simulated to form highlight masks; for regions occluded by highlights, their original, unoccluded structural information (e.g., edge maps extracted using the Canny operator) and regional information (e.g., oil stain area maps obtained through color segmentation) are saved as "ground truth" labels for training; simultaneously, within the annular analysis region of the newly generated highlight image, linear structure features (gradient maps) and anomalous region features (color statistics maps) are extracted as input data for the model. The training goal is to enable the model to learn the ability to map features from annular regions to the true structure inside the highlights. The training criterion is to minimize the difference between the predicted structure map and the ground truth structure map output by the model, typically using pixel-level cross-entropy loss functions or Dice loss functions, the latter being particularly effective for segmentation tasks involving unevenly sized targets such as lines and areas.

[0066] This model is a neural network with an encoder-decoder structure. The encoder extracts multi-level abstract features from the input annular region feature map through a series of convolutional and pooling layers. The decoder then uses upsampling and convolutional layers to progressively restore these abstract features to the original resolution and utilizes skip connections to fuse the shallow features from the encoder at the corresponding layers to preserve detailed information. The model's input is multi-channel; for example, one channel represents the brightness map of the annular region, another the gradient direction map, and a third the color anomaly probability map. The model's output is also a multi-channel image, with each channel representing a probability distribution map of a defect type; for example, channel one represents the probability of linear defects, and channel two represents the probability of planar defects.

[0067] In practical applications, the system combines the feature maps extracted from the annular analysis region of the input image into a multi-channel tensor, which is then input into the pre-trained U-Net model. The model performs one forward propagation calculation, and its output probability map, after thresholding, directly generates a structural prior guidance map. This map clearly indicates the predicted defect location, shape, and type under the highlight region. Compared to traditional algorithms, this approach has stronger nonlinear modeling capabilities and better generalization ability for complex patterns.

[0068] S106. Based on the structural prior guidance map, source patch constraints are applied, and a sample block-based image inpainting algorithm is used to process the highlight pixel region to obtain the output image.

[0069] In this context, source patch constraint refers to modifying the patch search and matching strategy of standard algorithms during image inpainting, making them prefer to select source patches that match the prior structural guidance map. Patch-based image inpainting algorithms are a classic technique, such as the Criminisi algorithm. Their core idea is to find the most similar pixel blocks (patches) from known regions of the image to fill unknown areas. The output image is the final result after highlight removal and restoration of underlying texture and structure.

[0070] Specifically, the image processing system uses the structural prior guidance map generated in step S105 to guide the inpainting process. Standard patch inpainting algorithms typically only consider color and texture similarity when searching for the best-matching source patch for a target patch in a highlight region (target region). This application modifies this by adding a structural similarity term when calculating the matching cost. This structural term compares the structure (e.g., gradient) of the source patch and its neighborhood to see if it matches the structure indicated by the structural prior guidance map at that location. For example, if the guidance map indicates a vertical crack should be present, a source patch containing a vertical edge will receive a higher matching priority. In this way, the inpainting process is forced to "draw" the defects depicted by the guidance map, thus achieving highlight removal and defect preservation.

[0071] It should be noted that the core of the source patch constraint lies in modifying the cost function for finding the best matching patch in sample block-based image inpainting algorithms (such as the Criminisi algorithm). Standard cost functions typically only consider color or texture similarity and are usually measured using the sum of squared differences (SSD). This application improves upon this by introducing a structural constraint term.

[0072] Specifically, for a target patch Ψ_p (centered at p) on the boundary δΩ of the region to be repaired (highlight region Ω), the algorithm needs to find a source patch Ψ_q (centered at q) in the known region Φ of the image to minimize the matching cost between the two. The modified cost function Cost(Ψ_p, Ψ_q) can be defined as:

[0073] Cost(Ψ_p, Ψ_q) = SSD_color(Ψ_p, Ψ_q) + λ*D_struct(Ψ_p, Ψ_q); where SSD_color is the traditional color similarity term, calculated as Σ_{i∈Ψ_p∩Φ}(I(i)-I(i-p+q))^2, where I(i) represents the color value of the image at pixel i. This term is calculated only for pixels known in the target patch Ψ_p.

[0074] D_struct is a structural similarity term introduced in this application, used to penalize source patches that do not conform to the prior structural guidance map. λ is a weighting coefficient used to balance the importance of color fidelity and structural consistency. D_struct is calculated as follows: Let G be the prior structural guidance map, and ∇I be the gradient field of the image. D_struct calculates the difference between the intrinsic structure of the source patch Ψ_q and the structure expected by the guidance map G at the target position p. Its mathematical form can be: D_struct(Ψ_p, Ψ_q) = Σ_{i∈Ψ_p}w(i)*||∇I(i-p+q)-T(G(i))||^2. Here, T(G(i)) is a transformation function that maps the scalar value (e.g., 0, 128, 255) of the guidance map G at point i to a desired gradient vector. For example, if G(i) represents a 30-degree crack, T(G(i)) outputs a gradient vector with a direction of 120 degrees (perpendicular to the crack) and a large magnitude. w(i) is a weight, which is larger when G(i) indicates the presence of structure, and smaller otherwise.

[0075] For example, if the structural prior guidance map indicates a vertical crack within the target patch Ψ_p region, then a source patch Ψ_q containing a strong vertical edge, even if its color differs slightly from the known pixels surrounding Ψ_p, will have a very small D_struct term, potentially outperforming a source patch with a perfectly matched color but a flat, unstructured interior in terms of total cost. In this way, the repair process is forced to fill along the structure depicted in the guidance map, thus preserving the defect.

[0076] In some embodiments, a suitable source patch may not be found. If the defect under the highlight area is unique, and no similar texture can be found anywhere else in the image, a patch transformation mechanism can be introduced to address this issue. When a perfectly matching source patch cannot be found, the system can search for the closest patch and perform geometric transformations (rotation, scaling) or lighting adjustments to better match the requirements of the guide map. For example, if a crack patch oriented at a 30-degree angle is needed, but only patches with vertical cracks are found, the system can rotate them by 30 degrees before using them for filling.

[0077] The following provides a more detailed description of the process of the method provided in this implementation. Please refer to [link / reference]. Figure 2 This is another flowchart illustrating the image highlight removal method in this application embodiment.

[0078] S201. Receive the input image, convert the input image to a color space with luminance and chrominance separation, identify the highlight pixel area above the preset luminance threshold in the luminance channel, and generate a highlight mask.

[0079] Refer to step S101, which will not be repeated here.

[0080] S202. Perform connected component analysis on the specular mask to calculate the area of ​​the specular pixel region and the maximum bounding rectangle.

[0081] Connected component analysis refers to the process of finding interconnected clusters of pixels (i.e., connected components) in a binary image, where each cluster represents an independent highlight region. Area refers to the total number of pixels contained within a connected component. The maximum bounding rectangle is the smallest rectangle whose edges are parallel to the coordinate axes and can perfectly enclose a connected component.

[0082] Specifically, after generating the specular mask in step S201, the image processing system may detect multiple unconnected specular spots in the image. To personalize the processing of each specular spot, the system first performs connected component analysis on the specular mask. This analysis traverses the mask image, assigning a unique label to each independent white region. After the analysis, the system obtains the number of specular regions and the pixel set of each specular region. Next, for each labeled connected component, the system calculates its key geometric properties: obtaining its area by counting the number of pixels it contains, and determining the width and height of its maximum bounding rectangle by finding the maximum and minimum values ​​of its pixel coordinates.

[0083] In some embodiments, this step can be calculated in several ways: Optionally, a two-pass algorithm can be used for connected component labeling, which is efficient and easy to implement; for geometric attribute calculation, the area can be directly counted, while the maximum bounding rectangle is obtained by traversing the x and y coordinates of all pixels within the connected component and recording x_min, x_max, y_min, and y_max. Optionally, functions encapsulated in image processing libraries such as OpenCV can be used, such as:

[0084] The `cv2.connectedComponentsWithStats` function returns a labeled graph of all connected components, along with statistical information such as the area and bounding rectangle of each component. It's understandable that other methods, such as seed fill algorithms from computer graphics, could also be used to identify connected components; this is not a limitation here.

[0085] In some embodiments, it is necessary to analyze contiguous highlight regions. When highlights from two or more independent device components merge into a large, irregularly shaped connected region in an image due to their proximity, analyzing them as a whole can introduce errors. To address this issue, a morphological separation step can be added after connected region analysis. For example, a watershed algorithm can be used, which can separate contiguous objects from their "bottlenecks." By applying the watershed algorithm to the distance-transformed image, a large connected region can be intelligently segmented into multiple more reasonable, convex sub-regions, and then the geometric properties of each sub-region can be calculated separately.

[0086] S203. Determine the preset pixel distance using a preset mapping function based on the area and the length of the longest side of the largest bounding rectangle.

[0087] The mapping function refers to a predefined mathematical formula or lookup table, whose input is the geometric dimensions of the highlight (such as area or long side), and whose output is a pixel distance value. The preset pixel distance is the width of the annular analysis region tailored to the specific highlight area being processed.

[0088] Specifically, the image processing system no longer uses a globally fixed expansion distance, but dynamically determines an optimal expansion distance for each specular connected region identified in step S202. The system takes the area of ​​this connected region and the length of the longest side of its largest bounding rectangle as input, and substitutes them into a preset mapping function. The design principle of this function is that the larger the specular area, the wider the surrounding context region to be analyzed should be, to ensure that sufficiently rich structural and texture information can be captured. For example, the mapping function can be a simple linear function: Expansion distance = a * length of the longest side + b * sqrt(area) + c, where a, b, and c are empirical coefficients. This calculated value is the preset pixel distance specific to that specular region.

[0089] It should be noted that the design of the preset mapping function aims to achieve adaptive analysis of the analysis region width. Its core mathematical logic is to establish a positive correlation between the specular size and the required amount of contextual information, while avoiding unreasonable results caused by extreme sizes. A robust mapping function f, whose inputs are the area A of the specular region and the longest side L_max of the maximum bounding rectangle, and whose output is the preset pixel distance D_ext, can be designed as a weighted combination with boundary constraints.

[0090] D_ext = max(D_min, min(D_max, w_L*L_max + w_A*sqrt(A))); D_min and D_max are the minimum and maximum thresholds for the extended distance, respectively. D_min (e.g., 5 pixels) ensures that there is a sufficient analysis area even for extremely small highlight points; D_max (e.g., 10% of the shorter side length of the image) prevents the analysis range from being expanded indefinitely for extremely large highlight areas, thus preventing the introduction of too much irrelevant background.

[0091] w_L and w_A are weighting coefficients used to balance the influence of the "elongation" and "scale" of the highlight region on its analytical scope. sqrt(A) converts the two-dimensional area to a one-dimensional scale, making it consistent with the dimensions of L_max. The values ​​of w_L and w_A are determined experimentally; for example, the value of w_L can be increased if the structure around a slender highlight (such as a reflection on a metal rod) is considered more instructive.

[0092] For example, in a 1920x1080 image, you can set:

[0093] Given D_min=10, D_max=100, w_L=0.1, and w_A=0.2, for a highlight region with L_max=50 and A=1000, the calculated median value is 0.1*50 + 0.2*sqrt(1000) ≈ 5 + 0.2*31.6 ≈ 11.3. Since 10 < 11.3 < 100, the final extension distance D_ext is 11 pixels. However, for a huge highlight with L_max=800 and A=200000, the calculated median value far exceeds 100, and D_ext will ultimately be limited to 100 pixels by D_max.

[0094] In some embodiments, it is necessary to consider the adaptability of the mapping function to extreme sizes. If a highlight region is particularly large (e.g., occupying half of the image) or particularly small (only a few pixels), a simple linear function may give unreasonable results (too large or too small). To address this issue, the mapping function should be designed with boundary constraints. For example, a minimum (e.g., 5 pixels) and a maximum (e.g., 10% of the shorter side of the image) value can be set for a preset pixel distance. Regardless of the original value calculated by the function, the final result will be limited to this reasonable range, ensuring that the algorithm remains stable and effective when dealing with highlights of various extreme sizes.

[0095] S204. Using the boundary of the specular mask as the inner boundary, extend outward by a preset pixel distance to obtain the outer boundary, and construct a ring-shaped analysis region enclosed by the inner and outer boundaries.

[0096] The inner boundary is the outline of the currently processed specular connected region. The preset pixel distance is a specific value dynamically calculated for this connected region in step S203. The outer boundary is the envelope formed by translating each point on the inner boundary outward along its normal direction by the preset pixel distance. The annular analysis region is the area enclosed by these two boundaries.

[0097] Specifically, after calculating the specific preset pixel distance for the current specular connected component, the image processing system begins to construct the corresponding adaptive annular analysis region. The system first obtains the precise contour of the connected component as its inner boundary. Then, it generates the outer boundary through a morphological dilation operation: based on the binary mask of the connected component, it dilates it using a structuring element whose size is related to the calculated preset pixel distance. The dilated mask contour is the outer boundary. Finally, subtracting the original connected component mask from the dilated mask yields a difference image that precisely defines this adaptively sized annular analysis region. Subsequent feature extraction will be strictly confined to this region.

[0098] In some embodiments, this step can be implemented in several ways: Optionally, a difference method after morphological dilation can be used, which is the most direct and computationally efficient method, and the structuring element can be circular or square; Optionally, a distance transformation method can be used, first calculating the distance map from each pixel in the non-highlight region to the nearest highlight pixel, and then filtering out all pixels whose distance values ​​are within the range of (0, preset pixel distance], and the set of these pixels constitutes the annular analysis region. It is understood that for complex non-convex highlight shapes, its skeleton can be calculated first, and then extended to both sides along the skeleton to construct an annular region of more uniform width, which is not limited here.

[0099] In some embodiments, the annular region is truncated by the image boundary. When the highlight area is near the image edge, its outward expansion may exceed the image range. Ignoring this portion directly would result in incomplete analysis information, especially at corners. To address this issue, the system needs to consider the image boundary when constructing the annular region. When the dilation operation exceeds the boundary, only the portion still within the image is retained. Simultaneously, in subsequent feature analysis, the analysis region near the boundary can be assigned a lower weight because this part of the information is incomplete. Alternatively, image padding techniques, such as mirror padding, can be used to temporarily expand the image boundary before analysis to construct a complete annular region, which is then cropped back to its original size after analysis.

[0100] S205. Calculate the pixel gradient magnitude and pixel gradient direction within the annular analysis region, identify high gradient pixels, and generate a linear structure feature dataset.

[0101] Refer to step S103, which will not be repeated here.

[0102] S206. Calculate the mean and standard deviation of pixel colors within the annular analysis area, identify pixels with abnormal colors, and generate an abnormal area feature dataset.

[0103] Refer to step S104, which will not be repeated here.

[0104] S207. Based on the directional information in the linear structural feature dataset and the abnormal region feature dataset, perform defect distribution reasoning and generate a structural prior guidance map.

[0105] Refer to step S105, which will not be repeated here.

[0106] S208. Based on the structural prior guidance diagram, source patch constraints are applied, and a sample block-based image inpainting algorithm is used to process the highlight pixel region to obtain the output image.

[0107] Refer to step S106, which will not be repeated here.

[0108] S209. Identify the defective regions in the output image, extract the geometric topological parameters and texture feature parameters of the defective regions, and generate a defect feature data package.

[0109] In this context, the defect region in the output image refers to the area on the image repaired in step S208 where the potential defect is located, as indicated by the structural prior guidance map. Geometric topology parameters include areas, perimeter, aspect ratio, and circularity, which describe the shape and size of the defect. Texture feature parameters, such as Local Binary Pattern (LBP) and Gray-Level Co-occurrence Matrix (GLCM) features, are used to describe the surface texture of the defect region. The defect feature data package is a structured dataset containing all the aforementioned parameters.

[0110] Specifically, after highlight restoration is completed and the output image is obtained, the image processing system performs this step to quantify and record the recovered defects. First, the system uses a priori structural guidance map or defect location information recorded during the restoration process to accurately segment the defect region on the output image. Then, a series of measurements are performed on this segmented region: the total number of pixels is calculated to obtain the area, its contour is traced to obtain the perimeter, its bounding rectangle is calculated to obtain the aspect ratio, and geometric parameters such as roundness are calculated based on the area and perimeter. Simultaneously, the system analyzes the pixel grayscale or color distribution within the region, extracting texture feature parameters such as LBP histogram, GLCM contrast, and energy. All these quantified values ​​are packaged into a structured defect feature data package.

[0111] In some embodiments, feature extraction can be achieved in several ways: optionally, geometric parameters can be extracted using contour analysis and region attribute calculation functions in image processing libraries; optionally, texture features can be extracted using LBP or GLCM feature extraction interfaces provided by libraries such as scikit-image to calculate the grayscale image of the defect region. It is understood that more advanced features, such as HOG (Histogram of Oriented Gradients) features or deep features extracted through deep learning networks, can also be extracted to more comprehensively describe the defect; this is not limited here.

[0112] In some embodiments, the stability of defect features needs to be considered. Due to the inherent uncertainty in the repair process, the defect details obtained from each repair may differ slightly, leading to fluctuations in the extracted feature values, which is detrimental to subsequent comparison and analysis. To address this issue, feature normalization and robust feature selection can be employed. For example, all geometric parameters can be normalized relative to the image size or a reference size. When selecting texture features, those that are insensitive to illumination and minor deformations should be prioritized. Furthermore, the same defect region can be repaired multiple times (e.g., using different random seeds), and then the average or median of the extracted features can be taken to obtain a more stable feature description.

[0113] S210. Calculate the position, shape, and brightness distribution of the highlight pixel region and generate a highlight feature data packet.

[0114] The highlight pixel region refers to the original highlight region identified in step S201. Position, shape, and brightness distribution are parameters describing the physical properties of this highlight region. The highlight feature data package is a structured data set that stores these highlight attributes.

[0115] Specifically, in parallel with the defect analysis in step S209, the image processing system also goes back to the state before processing to perform a quantitative analysis of the cause of information loss—the specular highlights themselves. The system uses the specular mask generated in S201 to calculate its centroid coordinates as the position of the specular highlight; reuses the area and bounding rectangle calculated in S202 as shape parameters; and statistically analyzes the brightness values ​​of all pixels within the specular mask area on the brightness channel of the original input image to generate a brightness distribution histogram. This data describing the characteristics of the specular highlights is integrated into a specular feature data packet, which records the specific circumstances of the illumination interference in this event.

[0116] In some embodiments, specular features can be calculated in several ways: optionally, the position can be obtained by calculating the centroid of the mask image; shape parameters can be obtained directly from the connected component analysis results; optionally, in addition to the histogram, more concise statistics such as average brightness, brightness variance, and kurtosis can be calculated to more compactly represent the characteristics of the brightness distribution. It is understood that more complex features such as the edge sharpness (gradient) of the specular highlights can also be analyzed to determine whether it is a hard or soft highlight, but this is not limited here.

[0117] In some embodiments, these specular features can be utilized. Recording specular features in isolation may seem irrelevant to the repair task itself. However, these features are crucial for understanding the relationship between defects and the environment. For example, long-term correlation analysis might reveal that specular highlights of a particular shape and brightness (potentially caused by sunlight at a specific angle) are consistently associated with certain types of missed or false positives. Therefore, the solution lies in storing specular feature data packets in strong correlation with defect feature data packets and other metadata (such as time and device ID), providing a basis for subsequent data mining and root cause analysis.

[0118] S211. Combine the defect feature data packet and the highlight feature data packet to generate the observation feature vector.

[0119] Here, "combination" refers to concatenating all numerical features from two data packets in a predetermined order. The observation feature vector is a one-dimensional array of numbers that comprehensively describes everything that happens in a given observation: what the repaired defect looks like, and what the specular highlights that caused the repair look like.

[0120] Specifically, the image processing system merges the defect feature data packet generated in step S209 and the highlight feature data packet generated in step S210. For example, the first half of the vector may contain values ​​such as the area, perimeter, circularity, and LBP histogram of the defect, while the second half contains values ​​such as the x-coordinate, y-coordinate, area, and average brightness of the highlight. The final result is a flattened one-dimensional vector, which is the observation feature vector. This vector is a highly condensed and quantified summary of a single highlight restoration event.

[0121] In some embodiments, the combination process can be implemented in several ways: optionally, the features can be directly concatenated in a fixed order, which is simple and straightforward; alternatively, the two feature sets can be standardized separately before concatenation (e.g., Z-score standardization) to make features of different dimensions comparable, which is crucial for subsequent machine learning tasks using the vector. Understandably, some cross-features, such as the ratio of defect area to highlight area, can also be added during combination, which may contain richer information; this is not limited here.

[0122] In some embodiments, there are issues with excessively high vector dimensionality and sparsity. If a large number of features are extracted (e.g., high-dimensional histogram features are used), the observed feature vectors become very long, posing challenges for storage and subsequent processing. To address this issue, dimensionality reduction techniques can be applied before generating the final vectors. For example, principal component analysis (PCA) or an autoencoder can be used to reduce the dimensionality of defect and highlight features separately, extracting their most prominent components, and then these low-dimensional representations can be concatenated. This reduces the dimensionality of the feature vectors while retaining most of the information, improving processing efficiency.

[0123] S212. Add a current processing timestamp and the identifier of the monitored device to the observed feature vector to generate a multi-dimensional state snapshot.

[0124] The current processing timestamp is the precise time when this image processing was performed. The monitored device identifier is an ID that uniquely identifies the device being photographed, such as a utility pole number or transformer asset number. The multidimensional state snapshot is the final and most complete data recording unit, binding quantized feature vectors with spatiotemporal context information.

[0125] Specifically, after generating the observation feature vector, the image processing system appends contextual metadata to it, completing the final data encapsulation. The system obtains the current system time from its operating environment and formats it as a standard timestamp. Simultaneously, it extracts the unique identifier of the monitored device from the task information or image metadata (EXIF). Then, the timestamp and device identifier, along with the observation feature vector generated in step S211, are stored in a data structure or database record. This record is called a multidimensional state snapshot, which fully records "when, on which device, what kind of defect was observed, and what kind of specular occlusion caused it."

[0126] In some embodiments, this step can be implemented in several ways: Optionally, all information (vector, timestamp, device ID) can be stored as a single record in a database for easy retrieval and querying later; alternatively, each snapshot can be serialized into a separate file (such as JSON or XML format), named with the device ID and timestamp, and stored in a hierarchical file system. It is understood that the snapshot data can also be sent to a time-series database in the cloud (such as InfluxDB), which is specifically optimized for processing timestamped data; this is not limited here.

[0127] In some embodiments, data management and consistency planning are required. As inspections proceed, a massive number of multi-dimensional status snapshots are generated. Efficiently storing, indexing, and retrieving this data, while ensuring the accuracy and uniqueness of device identifiers, presents a challenge. To address this issue, a robust asset management database needs to be established. Before processing images, the system should first query this database to associate each image with a unique device ID. All device identifiers should adhere to a unified naming convention. For data storage, a database system supporting efficient time-series and attribute queries should be used, and appropriate indexes should be established to support subsequent complex historical sequence analysis.

[0128] In some embodiments, the image processing system further optimizes the restoration model. Specifically, the image processing system performs morphological analysis on the structural prior guidance map to determine whether the potential defect type in the highlight pixel region is a linear defect or a planar defect, and generates a defect type label. Based on the defect type label, a matching specialized restoration model is selected from a preset restoration model library, and the specialized restoration model is used as the image restoration algorithm.

[0129] Morphological analysis refers to using operations such as opening, closing, and skeleton extraction to analyze the shape characteristics of defect regions in the structural prior guidance image. The defect type label is a classification identifier, such as "line" or "area". Specialized repair models are image inpainting algorithms specifically trained and optimized for specific types of defects (such as cracks or stains).

[0130] Specifically, after generating the prior structural guidance map, but before performing the final repair, the image processing system performs a check on it. The system performs morphological analysis on the non-zero regions in the guidance map. For example, by calculating the ratio of the region's skeleton length to its area, or by analyzing its aspect ratio, it determines whether it resembles a linear defect or a planar defect. If the skeleton is long and the area is small, the system labels it as a "linear defect"; conversely, if the region is relatively "full," it is labeled as a "planar defect." Then, the system accesses a pre-defined repair model library and selects the most suitable model based on this label. For example, it selects a repair model that excels at maintaining line sharpness and continuity (such as a GAN-based edge repair model) to handle linear defects, and a model that excels at generating smooth, natural textures (such as a diffusion-based texture synthesis model) to handle planar defects.

[0131] It should be noted that the specialized repair model here refers to a collection of deep learning repair models that have been specifically optimized and trained for different defect morphologies (such as linear and planar defects).

[0132] Linear defect repair models (e.g., the EdgeConnect model based on Generative Adversarial Networks (GANs)): The training dataset for this model consists of a large number of images containing real or synthetic linear defects (such as scratches and cracks). During training, the input is an image with a linear mask and the edge map of the masked region. The model typically consists of two stages: the first stage of the GAN network is responsible for predicting and completing the edge structure within the masked region; the second stage of the GAN network uses the completed edge map as guidance to repair the original masked image and generate the final result. The training criterion is to minimize a composite loss function, which includes an L1 loss to ensure pixel-level consistency, a perceptual loss to guarantee high-level semantic feature similarity, and an adversarial loss to make the generated result more realistic.

[0133] Planar defect restoration models (e.g., diffusion-based models or GAN models with attention mechanisms): The training dataset for this model consists of a large number of images containing planar masks of various shapes and textures (simulating oil stains, rust, etc.). The input is an image with a planar mask. The goal of the model is to generate filling content that is highly consistent with the surrounding environment in terms of texture, color, and structure. If a diffusion model is used, the training objective is to learn a denoising process to gradually restore clear image content from pure noise; if a GAN is used, the training criterion is also a composite loss function, but it may additionally include style loss or coherent semantic attention mechanisms to borrow similar texture information from distant parts of the image to ensure the naturalness of large-area restorations.

[0134] The model library physically represents a set of pre-trained neural network weight files (e.g., .pth or .h5 format) stored on a hard drive, each file associated with a defect type label (e.g., 'line', 'area'). Upon receiving an image to be repaired, the system first determines the defect type label based on the morphological analysis results of the structural prior guidance map. Then, the system loads the corresponding specialized repair model from the model library based on this label. Finally, the image to be repaired and a specular mask are fed into the model, which outputs the repaired image content. The system then seamlessly integrates this content back into the original image, completing the repair process.

[0135] In some embodiments, it is necessary to identify complex defects. A highlight area may simultaneously contain cracks and oil stains, i.e., linear and planar defects coexist. Simple binary classification cannot handle this. To address this issue, different pixel values ​​can be used in the structural prior guidance map to represent different inference types. For example, the inferred crack path might be represented by a value of 255, and the oil stain area by a value of 128. During morphological analysis, regions with different values ​​are processed separately. During repair, a step-by-step repair strategy can be adopted: first, the linear repair model is invoked to repair regions with a value of 255 with high priority; then, based on the results of the first repair, the planar repair model is invoked to process regions with a value of 128. This approach better handles complex and complex defect scenarios.

[0136] In some embodiments, the image processing system introduces a manual correction mechanism, that is, the image processing system overlays the structural prior guidance map and the input image on the user interface; receives the user's editing operation on the structural prior guidance map, updates the structural prior guidance map to a structural prior correction map, and uses the structural prior correction map to perform source patch constraints.

[0137] The user interface is a graphical software interface that allows operators to view images and intermediate results. Overlay display refers to drawing a semi-transparent structural prior guide diagram on top of the original input image. Editing operations include drawing and erasing using a mouse or stylus. The structural prior correction diagram is a guide diagram that has been manually modified.

[0138] Specifically, when the automatically generated structural prior guidance map has low confidence, or for particularly critical detection tasks, the image processing system submits the guidance map for manual review. On a user interface, the system uses the original input image as a background and overlays the guidance map as a semi-transparent color layer (e.g., red for linear structures and blue for planar structures). The operator can clearly see the location and shape of defects inferred by the algorithm. If the operator finds an error in the inference—for example, the algorithm misses a tiny crack or incorrectly identifies reflection as oil—they can directly modify the guidance map using tools similar to drawing software (such as a brush and eraser). All these editing operations update the guidance map in real time, forming the final structural prior correction map. This correction map is then sent back to the algorithm flow for subsequent repair steps.

[0139] In some embodiments, editing operations can be implemented in several ways: optionally, basic brush and eraser tools are provided, allowing users to freely draw or erase different types of defect indicators; optionally, more advanced editing tools are provided, such as a "line tool" for quickly drawing cracks, a "fill tool" for quickly marking stained areas, and a "move / scale tool" for adjusting the position and size of existing marks. Understandably, the interface can also provide undo / redo functionality, as well as keyboard shortcuts to show / hide guide layers, to enhance the user experience; this is not limited here.

[0140] A core system issue that may be encountered during the implementation of the solution is the efficiency of human-computer interaction. If every image requires manual intervention, the automation level and processing speed of the entire system will be significantly reduced. To address this problem, an intelligent review trigger mechanism should be established. The system can automatically assess the "uncertainty" of the guidance image based on a series of indicators, such as whether there are conflicting inferences, whether the confidence level of features is below a threshold, or whether the complexity of the highlight region is too high. Only when the uncertainty exceeds a preset threshold will the system push the image to human review; for simple and clear cases, it will be processed automatically. This can ensure the accuracy of critical tasks while maximizing the system's high efficiency.

[0141] In some embodiments, the image processing system constructs a learning and evaluation closed loop, namely, the image processing system performs secondary repair processing on the highlight pixel region based on the structural prior correction map and the sample block-based image inpainting algorithm to generate a corrected output image; calculates the integrity score and repair quality score of the defect region in the output image and the corrected output image respectively to generate the image repair quality evaluation result; and stores the editing operation and quality evaluation result in the user interaction database to construct an editing and repair correlation model.

[0142] Secondary repair refers to re-performing the repair process using a manually corrected guide map. Integrity score and repair quality score are indicators for quantitatively evaluating the repair effect. The edit-repair correlation model is a data-driven model designed to learn "what kind of editing operation brings how much quality improvement".

[0143] Specifically, after receiving the user-submitted prior structural correction map, the image processing system uses this more accurate guide map to re-execute the repair step in step S208, generating a higher-quality corrected output image. Next, the system enters the evaluation phase. It compares two versions of the output image: one generated based on the original guide map, and the other based on the correction map. The system calculates a defect integrity score (e.g., the degree of fit between the repaired defect and the guide map) and a repair quality score (e.g., evaluating the naturalness of the repaired area using a no-reference image quality assessment algorithm such as BRISQUE). The difference between these two scores quantifies the value brought by the user's editing. Finally, the system stores the user's specific editing operation (e.g., "added a line structure of length L at coordinates (x, y)"), the original image features, and the final quality assessment result as a record in the user interaction database. Over time, this database forms the foundation for building an edit-repair correlation model.

[0144] In some embodiments, the completeness score measures the extent to which the repaired defect reproduces the structure in the (manually corrected) structural prior guidance map. Its calculation logic compares the structural consistency between the defect region in the repaired image and the guidance map. Specifically, firstly, the actual structural map S_repaired within the repaired area is extracted from the repaired output image using an edge detection algorithm (such as Canny) or texture analysis method. Then, this map is compared with the structural prior correction map G_corrected, which serves as the "gold standard." The score can be calculated using the F1 score, commonly used in image segmentation.

[0145] Score_completeness=F1=2*(Precision*Recall) / (Precision+Recall), where:

[0146] Precision=|S_repaired∩G_corrected| / |S_repaired|,

[0147] Recall=|S_repaired∩G_corrected| / |G_corrected|.

[0148] A higher F1 score indicates better restoration integrity.

[0149] The restoration quality score is used to evaluate the visual realism and naturalness of the restored area, i.e., the presence of artifacts, blurring, or unnatural textures. Since a lossless reference image is usually unavailable, this score requires a No-Reference Image Quality Assessment (NR-IQA) algorithm. A typical algorithm is:

[0150] BRISQUE (Blind / Referenceless Image Spatial Quality Evaluator) is an algorithm that extracts a series of "Natural Scene Statistics" features from image patches in the repaired region, such as the distribution characteristics of local normalized brightness coefficients. These features are then fed into a Support Vector Regression (SVR) model pre-trained on a large number of human-rated images, and the model outputs a quality score. A lower BRISQUE score indicates higher image quality (more natural). For ease of use, it can be converted into a positive metric, such as Score_quality = 1 / (1 + Score_BRISQUE), where a higher score indicates better quality.

[0151] The final image restoration quality assessment result can be obtained by combining these two scores, for example, by weighted summation Q=w_c*Score_completeness+w_q*Score_quality to obtain a total score.

[0152] In addition, the edit-repair correlation model is a data-driven meta-learning model. Its core is to learn the potential relationship between user editing behavior and the improvement of repair quality. It aims to allow the system to gain "experience" from manual correction in order to achieve self-optimization.

[0153] The model's training data comes from a large number of records accumulated in the user interaction database. Each training sample is a data tuple (V_obs, O_edit, ΔQ), where:

[0154] V_obs: is the "observation feature vector" describing the initial state, which includes highlight features and defect features automatically inferred by the algorithm.

[0155] O_edit: This is a vectorized representation of the user's editing operations. For example, an "add line" operation can be represented as [op_code=1, x1, y1, x2, y2, ...], and an "erase region" operation can be represented as [op_code=2, cx, cy, radius, ...].

[0156] ΔQ: This is the improvement in the overall repair quality score brought about by the editing operation, i.e., Q_corrected - Q_initial.

[0157] The model itself can be a regression model, such as a gradient boosting decision tree (XGBoost) or a small multilayer perceptron (MLP). The goal of training is to teach the model to accurately predict ΔQ based on the inputs V_obs and O_edit. The training criterion is to minimize the mean squared error (MSE) between the predicted ΔQ_pred and the true ΔQ_true.

[0158] It can also be a pre-trained regressor that encapsulates knowledge about "under what circumstances (V_obs), what kind of editing (O_edit) is most likely to bring the greatest quality improvement (ΔQ)".

[0159] The functions that can be achieved by editing and repairing related models include:

[0160] 1. Offline Algorithm Optimization: By analyzing the feature importance of the model, systematic biases in the algorithm can be identified. For example, if the model shows that under the V_obs condition of "large and irregularly shaped specular area", "manually extending the linear structure" O_edit always results in a large ΔQ, this indicates that the original defect distribution inference algorithm has shortcomings in linear extension, and developers can make targeted improvements accordingly.

[0161] 2. Online Intelligent Recommendation: In the user interface, when an image is submitted for review, the system can pre-generate a series of high-probability candidate editing operations and use the model to predict the ΔQ of each operation. Then, it prioritizes or highlights the editing operations with the highest predicted ΔQ to the user, thereby assisting the user's decision-making and improving the efficiency of manual correction.

[0162] In some embodiments, the image processing system performs historical trend analysis and prediction. Specifically, the image processing system retrieves multiple historical multidimensional state snapshots that match the monitored device identifier, sorts them chronologically, and obtains a set of historical state snapshots. It then extracts the defect feature data packet of each snapshot in the set of historical state snapshots to generate a defect evolution sequence. It also extracts the highlight feature data packet of each snapshot in the set of historical state snapshots to generate a lighting environment evolution sequence. When the defect region is determined to be in a stable state based on the defect evolution sequence, the image re-inspection time is determined based on the lighting environment evolution sequence.

[0163] The historical state snapshot set consists of a series of snapshot records generated by step S212, belonging to the same device and arranged chronologically. The defect evolution sequence is a time series, where the data points are the changes in defect features (such as area) over time. Similarly, the lighting environment evolution sequence is the change in highlight features (such as brightness) over time.

[0164] Specifically, after processing a new image and generating a new multidimensional state snapshot, the image processing system can initiate a deeper analysis. The system uses the device identifier in the snapshot to retrieve all historical snapshots of that device from the database, sorting them by timestamp. Then, the system extracts two sets of time-series data from this snapshot set: first, it iterates through the defect feature data packets of each snapshot, extracting key indicators (such as defect area) to form a defect evolution sequence reflecting the change in defect size over time; second, it iterates through the highlight feature data packets of each snapshot, extracting key indicators (such as average brightness) to form a lighting environment evolution sequence reflecting the change in light intensity at the device location over time. The system first analyzes the defect evolution sequence. If it finds that the defect area has remained stable or changed very little in recent measurements, it determines that the defect has entered a stable state. At this point, the system further analyzes the lighting environment evolution sequence, searching for periodic periods with the weakest light and least obvious highlights (e.g., a cloudy afternoon in winter), and recommends an optimal re-inspection time to ensure that a clear image without highlights can be captured directly during the next inspection.

[0165] In some embodiments, this step can be implemented in several ways: Optionally, the analysis of the defect evolution sequence can employ time series analysis methods, such as moving average or regression analysis, to determine whether its trend is increasing, decreasing, or stable; Optionally, the analysis of the illumination environment evolution sequence can be combined with Fourier transform to find its daily and annual cycle patterns, thereby predicting future illumination intensity. It is understood that the determination of the re-inspection time can also be combined with other factors, such as the priority of the inspection task and weather forecasts, for comprehensive decision-making, which is not limited here.

[0166] In some embodiments, data gaps and irregular sampling exist. Because inspection tasks may not be performed strictly according to a fixed cycle, the time intervals between historical state snapshots are uneven, and there may even be long periods of missing data. This poses a challenge to standard time series analysis. To address this issue, the series can be preprocessed before analysis. For example, interpolation methods (linear interpolation or spline interpolation) can be used to fill in missing data points, generating a regularly sampled time series. For the early stages of data sparsity, simpler trend-judgment methods (such as comparing the initial and most recent measurements) can be used. As the amount of data increases, a more complex time series model can be switched to, such as ARIMA or LSTM, which have better handling capabilities for irregular sequences.

[0167] In this embodiment, by employing a technical solution that analyzes the structural and color features of the surrounding reliable area before removing highlights and generates a priori structural guidance map to constrain the repair process, this application fundamentally changes the blind filling repair mode and transforms it into a purposeful and evidence-based structured repair. Simultaneously, by introducing a series of enhancement mechanisms such as defect type identification, adaptive analysis, human-computer interaction correction, and historical data analysis, this application constructs a complete technical closed loop from single image processing to long-term status monitoring. This effectively solves the contradiction between highlight removal and defect preservation in existing technologies, as well as the problem of insufficient robustness of fully automated processes in complex scenarios. Thus, it achieves high-precision, high-reliability, and intelligent perception and management of equipment status under harsh lighting conditions.

[0168] The image processing system in the embodiments of this invention is described below from the perspective of hardware processing. Please refer to [link / reference needed]. Figure 3 This is a schematic diagram of the physical device structure of an image processing system in an embodiment of this application.

[0169] It should be noted that, Figure 3 The structure of the image processing system shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of the present invention.

[0170] like Figure 3 As shown, the image processing system includes a CPU 301, which can perform various appropriate actions and processes according to a program stored in ROM 302 or a program loaded from storage section 308 into RAM 303, such as executing the methods described in the above embodiments. RAM 303 also stores various programs and data required for system operation. The CPU 301, ROM 302, and RAM 303 are interconnected via bus 304. I / O interface 305 is also connected to bus 304.

[0171] The following components are connected to I / O interface 305: input section 306 including audio input devices, push-button switches, etc.; output section 307 including liquid crystal display (LCD) and audio output devices, indicator lights, etc.; storage section 308 including hard disks, etc.; and communication section 309 including network interface cards such as LAN (Local Area Network) cards, modems, etc. Communication section 309 performs communication processing via a network such as the Internet. Drive 310 is also connected to I / O interface 305 as needed. Removable media 311, such as disks, optical disks, magneto-optical disks, semiconductor memories, etc., are installed on drive 310 as needed so that computer programs read from them can be installed into storage section 308 as needed.

[0172] In particular, according to embodiments of the present invention, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of the present invention include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing computer programs for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 309, and / or installed from removable medium 311. When the computer program is executed by CPU 301, it performs the various functions defined in the present invention.

[0173] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. Each block in a flowchart or block diagram may represent a module, program segment, or portion of code, which contains one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those shown in the drawings.

[0174] Specifically, the image processing system of this embodiment includes a processor and a memory. The memory stores a computer program, and when the computer program is executed by the processor, it implements the image highlight removal method provided in the above embodiment.

[0175] In another aspect, the present invention also provides a computer-readable storage medium, which may be included in the image processing system described in the above embodiments; or it may exist independently and not assembled into the image processing system. The storage medium carries one or more computer programs that, when executed by a processor of the image processing system, cause the image processing system to implement the image highlight removal method provided in the above embodiments.

[0176] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit it. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

[0177] As used in the above embodiments, depending on the context, the term "when..." can be interpreted as meaning "if...", "after...", "in response to determining...", or "in response to detecting...". Similarly, depending on the context, the phrase "when determining..." or "if (the stated condition or event) is interpreted as meaning "if determining...", "in response to determining...", "when (the stated condition or event) is detected", or "in response to detecting (the stated condition or event)".

Claims

1. A method for removing highlights from an image, characterized in that, Applied to an image processing system, the method includes: Receive an input image, convert the input image to a color space with luminance and chrominance separation, identify the highlight pixel region with a luminance threshold above the preset luminance threshold in the luminance channel, and generate a highlight mask; Based on the boundary of the highlight mask, a predetermined pixel distance is extended outward to construct a ring-shaped analysis region surrounding the highlight pixel region; Within the annular analysis region, the pixel gradient magnitude and pixel gradient direction are calculated to identify high-gradient pixels and generate a linear structure feature dataset. Within the annular analysis area, the mean value and standard deviation of pixel color are calculated to identify pixels with abnormal color and generate an abnormal region feature dataset. Based on the directional information in the linear structural feature dataset and the abnormal region feature dataset, defect distribution inference is performed to generate a structural prior guidance map; Morphological analysis is performed on the prior guidance map of the structure to determine whether the potential defect type of the highlight pixel region is a linear defect or a planar defect, and a defect type label is generated. Based on the defect type label, a matching specialized repair model is selected from the preset repair model library, and the specialized repair model is used as the image repair algorithm. Based on the prior guidance map of the structure, source patch constraints are applied, and the highlight pixel region is processed using a sample block-based image inpainting algorithm to obtain the output image.

2. The method according to claim 1, characterized in that, The step of constructing a ring-shaped analysis region surrounding the highlight pixel region by extending a preset pixel distance outward from the boundary of the highlight mask specifically includes: Perform connected component analysis on the specular mask to calculate the area and maximum bounding rectangle of the specular pixel region; Based on the area and the length of the longer side of the largest bounding rectangle, a preset pixel distance is determined using a preset mapping function; Using the boundary of the specular mask as the inner boundary, the outer boundary is obtained by extending the preset pixel distance outward, thus constructing a ring-shaped analysis region enclosed by the inner boundary and the outer boundary.

3. The method according to claim 1, characterized in that, After the steps of performing source patch constraints based on the structural prior guidance map, processing the highlight pixel region using a sample block-based image inpainting algorithm, and obtaining the output image, the method further includes: On the user interface, the prior guidance diagram of the structure and the input image are displayed overlaid. The system receives user edits to the structural prior guidance diagram and updates it to a structural prior correction diagram for use in source patch constraints.

4. The method according to claim 3, characterized in that, After receiving the user's editing operation on the structural prior guidance diagram and updating the structural prior guidance diagram to a structural prior correction diagram for using the structural prior correction diagram to perform source patch constraints, the method further includes: Based on the prior correction map of the structure, the image inpainting algorithm based on sample blocks is used to perform secondary inpainting on the highlight pixel region to generate the corrected output image. The integrity score and repair quality score of the defective region in the output image and the corrected output image are calculated respectively to generate the quality assessment result of image restoration; The editing operations and the quality assessment results are stored in the user interaction database to construct an editing and repair correlation model.

5. The method according to claim 1, characterized in that, After the steps of performing source patch constraints based on the structural prior guidance map, processing the highlight pixel region using a sample block-based image inpainting algorithm, and obtaining the output image, the method further includes: Identify the defective regions in the output image, and extract the geometric topological parameters and texture feature parameters of the defective regions to generate a defect feature data package; Calculate the position, shape, and brightness distribution of the highlight pixel region to generate a highlight feature data packet; The defect feature data packet and the highlight feature data packet are combined to generate an observation feature vector; A multidimensional state snapshot is generated by appending a current processing timestamp and the identifier of the monitored device to the observed feature vector.

6. The method according to claim 5, characterized in that, After the step of appending a current processing timestamp and the identifier of the monitored device to the observed feature vector to generate a multidimensional state snapshot, the method further includes: Based on the monitored device identifier, retrieve multiple historical multidimensional state snapshots that match the monitored device identifier, and sort them in time sequence to obtain a set of historical state snapshots; Extract the defect feature data packet for each snapshot in the historical state snapshot set to generate a defect evolution sequence; Extract the highlight feature data packet of each snapshot in the historical state snapshot set to generate a lighting environment evolution sequence; When the defect region is determined to be in a stable state based on the defect evolution sequence, the image re-inspection time is determined based on the illumination environment evolution sequence.

7. An image processing system, characterized in that, The image processing system includes: one or more processors and a memory; the memory is coupled to the one or more processors, the memory is used to store computer program code, the computer program code including computer instructions, and the one or more processors call the computer instructions to cause the image processing system to perform the method as described in any one of claims 1-6.

8. A computer-readable storage medium comprising instructions, characterized in that, When the instructions are executed on the image processing system, the image processing system performs the method as described in any one of claims 1-6.

9. A computer program product, characterized in that, When the computer program product is run on the image processing system, the image processing system performs the method as described in any one of claims 1-6.

Images