A highlight removal method based on granularity interactive hybrid highlight removal model

Abstract

The present invention discloses a highlight removal method based on a granular interactive hybrid highlight removal model, which relates to the field of image processing technology. The method mainly comprises the following steps: constructing a first loss function based on the distance between a non-highlight blurred background prediction image and a non-highlight blurred background reference image; obtaining an extended feature map by connecting the non-highlight blurred background prediction image and the original highlight image through color channel, and performing a linear projection transformation on the extended feature map; obtaining a hybrid feature map based on the extended feature map after the linear projection transformation; performing correlation calculation and enhanced feature acquisition under enhancement based on the hybrid feature map; reconstructing a non-highlight prediction image based on the enhanced features, and constructing a second loss function based on the distance between the image and the corresponding true de-highlighted image; and optimizing the parameters of the hybrid highlight removal model under the minimum optimization of the total loss function. The present invention combines a coarse-grained and fine-grained two-stage optimization framework to enable the model to maintain a balance between background blur suppression and detail preservation in complex scenes.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a highlight removal method based on a granularity interactive hybrid highlight removal model. Background Art

[0002] Existing methods for specular highlight removal primarily rely on traditional image processing algorithms and early deep learning models, but these methods still have significant limitations. Traditional methods often employ physics-based illumination model optimization or low-level image processing techniques (such as clustering and filtering) to achieve highlight suppression by propagating information from diffuse pixels to specular regions. While these methods can adjust overall illumination intensity, they struggle to precisely locate highlight regions, resulting in suboptimal local restoration results. Furthermore, their high computational complexity makes them difficult to adapt to large-scale data scenarios. With the advancement of deep learning technology, supervised highlight removal models have made some progress through training with labeled data. However, these models rely on manually labeled highlight masks or intensity maps as auxiliary supervisory signals, which are not only costly to label but also susceptible to subjective judgment errors, limiting the model's generalization capabilities. Furthermore, existing methods generally lack effective modeling mechanisms for linking local details with global semantics. Consequently, the inpainting process suffers from insufficient interaction between bright and dark regions, which can easily lead to hue distortion and boundary artifacts. Especially for complex natural scenes (such as those with variable lighting and dense highlight distribution), existing models mostly adopt a single-stage restoration strategy, which makes it difficult to balance the needs of background blur suppression and detail preservation, resulting in texture blur or local detail loss in the reconstructed image, seriously affecting the input quality of downstream visual tasks (such as object detection and semantic segmentation). Summary of the Invention

[0003] In order to solve the problems existing in the existing specular highlight removal process, the present invention proposes a highlight removal method based on a granular interactive hybrid highlight removal model. The hybrid highlight removal model includes a coarse prediction network branch and a fine prediction network branch, including the following steps:

[0004] S1: Reconstruct the original highlight image through the coarse prediction network branch to obtain a background prediction image without highlight blur, and construct a first loss function based on the distance between the background prediction image without highlight blur and the reference image without highlight blur;

[0005] S2: The extended feature map is obtained by connecting the color channel of the non-highlight blurred background prediction map and the original highlight image, and the extended feature map is linearly projected and input into the fine prediction network branch;

[0006] S3: Extract features from the extended feature map after linear projection transformation through the fine prediction network branch to obtain a highlight-background mixed feature map;

[0007] S4: Generate a content mask based on the highlight-background mixed feature map, and perform correlation calculation and enhanced feature acquisition under enhancement on the highlight-background mixed feature map through the content mask;

[0008] S5: reconstructing a non-highlight prediction map by decoding the enhanced features, and constructing a second loss function based on the distance between the non-highlight prediction map and the corresponding true de-highlighted image;

[0009] S6: Optimize the parameters of the hybrid highlight removal model under the minimum optimization of the total loss function according to the weighted sum of the first loss function and the second loss function.

[0010] This method eliminates the need for manually annotated highlight masks and instead uses a coarse prediction network to generate a background reference map without highlight blurring as auxiliary supervision. This reduces annotation costs while also preventing subjective errors from limiting the model's generalization capabilities. Furthermore, the combination of the reference map generation strategy with a two-stage optimization framework of coarse-grained blur prediction and fine-grained content reconstruction enables the model to maintain a balance between background blur suppression and detail preservation in complex scenes, significantly improving texture clarity in restored images.

[0011] Furthermore, the coarse prediction network branch adopts a U-shaped convolutional neural network, including an encoder and a decoder; the encoder is composed of four 3×3 pooling convolution layers with a stride of 2 and one pooling convolution layer with a stride of 1, and each convolution layer is followed by a ReLU activation function; the decoder is composed of four deconvolution layers, which reuse the features of the encoder through jump connections, and the last layer is a 1×1 convolution layer with a Softmax activation function.

[0012] Furthermore, in the step S1, the method of generating the non-highlight blurred background reference image includes: applying blurring processing based on a morphological operator to the corresponding real de-highlighted image to obtain the non-highlight blurred background reference image.

[0013] Furthermore, in the S2 step, the specific method of connecting the color channels is: splicing the single-channel non-highlight blurred background prediction map with the RGB channel of the original highlight image along the channel dimension to form a four-channel extended feature map, and performing standardization and feature dimensionality reduction through a 3×3 convolution linear projection layer.

[0014] Furthermore, the fine prediction network branch adopts a Transformer-based UNet architecture, and its encoder and decoder both include multi-head self-attention modules. The multi-head self-attention module of the encoder includes a series of channel-level multi-head self-attention and a downsampling layer, and the multi-head self-attention module of the decoder includes a series of channel-level multi-head self-attention and an upsampling layer.

[0015] Furthermore, the encoder of the fine prediction network branch also introduces a background interaction attention module to guide the correlation between bright and dark areas. The formula is expressed as:

[0016]

[0017] Where, For the The output features of the layer Transformer include the query matrix Q, key matrix K and value matrix V in the attention mechanism. is the correlation matrix of bright and dark regions, is the background mask interactive attention, is the normalized exponential function, T is the matrix transpose, d is the feature dimension, is the weight coefficient, and E is the unit matrix.

[0018] Furthermore, in the step S1, the formula of the first loss function is expressed as:

[0019]

[0020] Where, For the first loss, is the original highlight image, This is a predicted image with no highlight blur background. To truly remove the highlight image, is the morphological operator, is the L1 norm distance.

[0021] Furthermore, in step S5, the formula of the second loss function is expressed as:

[0022]

[0023] Where, For the second loss, For the non-highlight prediction map, is a constant whose value is .

[0024] Furthermore, in step S6, the weights of the first loss function and the second loss function in the total loss function are adjusted according to the balance between the constraints of background blur prediction and the final reconstruction quality, and the formula is expressed as follows:

[0025]

[0026] Where, is the total loss, is the weight of the first loss, is the weight of the second loss.

[0027] Furthermore, in the S5 step, the reconstruction of the non-highlight prediction image also includes a post-processing step: dynamic range adjustment and sharpening enhancement processing are performed on the non-highlight prediction image, the dynamic range adjustment limits the pixel value to the interval of [0, 255], and the sharpening enhancement uses non-local mean filtering to improve the clarity of local details.

[0028] Compared with the prior art, the present invention has at least the following beneficial effects:

[0029] (1) The highlight removal method proposed in this paper is based on a granular interactive hybrid highlight removal model. It does not rely on manually labeled highlight masks. Instead, it only needs to generate a non-highlight blurred background reference image through a coarse prediction network as auxiliary supervision. This not only reduces the annotation cost but also avoids the limitation of subjective errors on the generalization ability of the model.

[0030] (2) By splicing the blurred background image with the original image and projecting them into extended features, combined with the global multi-head self-attention and background interactive attention modules of the fine prediction network, it is possible to dynamically capture the local-global correlation between bright and dark areas and effectively suppress hue distortion and boundary artifacts.

[0031] (3) The combination of the benchmark image generation strategy based on the morphological blur operator and the two-stage optimization framework of coarse-grained blur prediction and fine-grained content reconstruction enables the model to maintain a balance between background blur suppression and detail preservation in complex scenes, and the texture clarity of the restored image is significantly improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 A step diagram of a highlight removal method based on a granularity interactive hybrid highlight removal model;

[0033] Figure 2 Schematic diagram of the network structure of the mixed highlight removal network model. DETAILED DESCRIPTION

[0034] The following are specific embodiments of the present invention and the accompanying drawings to further describe the technical solutions of the present invention, but the present invention is not limited to these embodiments.

[0035] In the field of natural scene image processing, specular highlight removal is a key step in improving image quality and the performance of downstream visual tasks. However, existing methods based on traditional image processing or a single deep learning framework have significant bottlenecks: on the one hand, traditional methods rely on physical model optimization or low-level filtering operations, making it difficult to accurately distinguish between highlight areas and normal lighting areas, resulting in local oversmoothing or artifacts in the repaired images; on the other hand, although supervised deep learning models can learn highlight features through data-driven learning, they rely on manually labeled highlight masks or intensity maps as supervisory signals, which not only has high labeling costs, but is also prone to limited model generalization capabilities due to labeling errors. In addition, existing methods mostly use a single repair stage and lack a collaborative optimization mechanism for background blur perception and global content consistency. In complex scenes (such as high dynamic lighting and dense highlight interference), it is difficult to balance the needs of detail preservation and artifact suppression. To address the above problems, this paper proposes a highlight removal method based on a granular interactive hybrid highlight removal model. The coarse prediction network automatically generates a blurry background image without highlights as a priori guidance. Combined with the dynamic feature fusion mechanism of the multi-head self-attention (MSA) and background interactive attention (BIA) modules in the fine prediction network, the method can achieve accurate positioning and natural reconstruction of highlight areas. Figure 1 As shown, the specific steps include:

[0036] S1: Reconstruct the original highlight image through the coarse prediction network branch to obtain a background prediction image without highlight blur, and construct a first loss function based on the distance between the background prediction image without highlight blur and the reference image without highlight blur;

[0037] S2: The extended feature map is obtained by connecting the color channel of the non-highlight blurred background prediction map and the original highlight image, and the extended feature map is linearly projected and input into the fine prediction network branch;

[0038] S3: Extract features from the extended feature map after linear projection transformation through the fine prediction network branch to obtain a highlight-background mixed feature map;

[0039] S4: Generate a content mask based on the highlight-background mixed feature map, and perform correlation calculation and enhanced feature acquisition under enhancement on the highlight-background mixed feature map through the content mask;

[0040] S5: reconstructing a non-highlight prediction map by decoding the enhanced features, and constructing a second loss function based on the distance between the non-highlight prediction map and the corresponding true de-highlighted image;

[0041] S6: Optimize the parameters of the hybrid highlight removal model under the minimum optimization of the total loss function according to the weighted sum of the first loss function and the second loss function.

[0042] During implementation, data preparation is an important foundation for building a highlight removal model. This example uses a dataset containing original highlight images and their corresponding real-world highlight-removed images for model training and validation. The input data primarily comes from publicly available highlight restoration benchmark datasets (such as SHIQ and LIME), and includes diverse samples of natural scenes with varying lighting conditions and highlight distribution densities. To ensure data quality and model generalization, the input images undergo a standardized preprocessing process:

[0043] First, the original highlight image under the RGB channel is normalized, and the pixel values ​​are mapped to the range [0, 1] to eliminate differences in light intensity. Second, the image resolution is uniformly adjusted to M×M pixels (the pixel size is set according to actual needs) to avoid fluctuations in network training caused by inconsistent sizes. On this basis, data augmentation strategies such as random cropping and horizontal flipping are used to expand sample diversity to simulate complex perspectives and partial occlusion scenarios in real applications.

[0044] Furthermore, to address the high cost of highlight area labeling, this invention employs a weakly supervised learning framework, relying solely on paired original highlight-containing images and true highlight-removed images for end-to-end training. This eliminates the need for additional labeling of highlight masks or intensity maps, reducing the need for manual intervention. To verify the rationality of the data distribution, the dataset undergoes manual screening and quality assessment before training, removing anomalous samples with severe noise, motion blur, or incorrect labeling to ensure the validity and consistency of the input data.

[0045] After the training data is processed as above, the original highlight image (size is M×M×3) is input into the hybrid highlight removal network model to train the model. In the present invention, the hybrid highlight removal network model adopts a dual-branch collaborative architecture design including a coarse prediction network branch (CoarseNet) and a fine prediction network branch (such as Figure 2 As shown in Figure 2), it takes into account both local feature extraction and global semantic association.

[0046] Among them, the coarse prediction network branch serves as the front-end processing module, and quickly captures the blurred background information of the image based on a lightweight U-shaped convolutional structure: its encoder is composed of multiple layers of 3×3 pooled convolutions stacked in series, and gradually compresses the spatial dimension through a downsampling operation with a step size of 2, and combines it with the ReLU activation function to enhance the nonlinear representation capability; the decoder gradually restores the feature resolution through the deconvolution layer, and uses the jump connection mechanism to reuse the local detail features of different levels in the encoder, and finally outputs a single-channel non-highlight blurred background prediction image. Here, the present invention uses downsampling and upsampling operations to capture the detailed features of downsampling, reduce the amount of calculation during morphological processing, and then restores the image to its original resolution through upsampling to facilitate the calculation of the first loss.

[0047] Meanwhile, the refined prediction network branch (RefineFormer), serving as a backend optimization module, employs a UNet architecture that fuses Transformers and CNNs, dynamically modeling the global context of the image through a multi-head self-attention mechanism. The input image and the coarse prediction result are concatenated along the channel dimension and then mapped into low-dimensional embedded features via a linear projection layer, which serve as the input to the refined prediction network branch. This branch specifically incorporates a background interaction attention (BIA) module, using the blurred background image generated by the coarse prediction as a priori guidance to strengthen cross-regional correlations between bright and dark areas in the feature space, thus avoiding the tonal discontinuity caused by local restoration in traditional methods. The two branches complement each other through a cascaded approach. The coarse prediction network provides initial constraints for background blur suppression, while the refined prediction network performs detailed reconstruction based on this constraint. Ultimately, this forms an end-to-end highlight removal pipeline that balances processing efficiency and restoration quality.

[0048] Specifically, after receiving the processed original highlight image, the coarse prediction network branch uses its U-shaped convolutional architecture to extract and reconstruct multi-scale features of the input: the encoder uses multiple layers of 3×3 pooling convolution to gradually compress the image spatial dimensions and capture the low-frequency blur features of the background area; the decoder uses deconvolution layers combined with jump connections to reuse local detail information at different levels in the encoder, gradually restore the image resolution, and finally output a single-channel background prediction image without highlight blur.

[0049] To enhance the supervised robustness of the coarse prediction network, this paper further introduces morphological operators to generate a non-highlight blurred background reference image. First, the true de-highlighted image is downsampled to 20% of its original resolution. A dilation operation is used to blur the low-frequency regions, and then upsampled to its original size to simulate the blur characteristics of the natural background. A coarse-grained loss function (first loss function) is constructed by calculating the L1 normalized distance between the non-highlight blurred background prediction image and the non-highlight blurred background reference image:

[0050] ,

[0051] in, For the first loss, is the original highlight image, This is a predicted image with no highlight blur background. To truly remove the highlight image, is the morphological operator, is the L1 normalized distance. Here, by constraining the network to learn consistent features of background blur through a coarse-grained loss function, the computational complexity of the subsequent fine-grained prediction network is reduced. It also implicitly guides the network to distinguish between highlight and normal-light areas, avoiding the bias caused by direct reliance on manual annotation. This results in a predicted background image free of highlight blur, providing semantic priors for the fine-grained prediction network and enhancing background information through end-to-end optimization and adaptation.

[0052] The present invention then uses the non-highlight blurred background prediction map generated by the coarse prediction network as a bridge to enhance the completeness of the input information through a multimodal feature fusion strategy. Specifically, the present invention concatenates the single-channel non-highlight blurred background prediction map (size M×M×1) with the original highlight image of the RGB three-channel (size M×M×3) along the color channel dimension to form a four-channel extended feature map (size M×M×4). This preserves the original color and texture information while embedding the background blur prior provided by the coarse prediction network. To further adapt to the feature processing requirements of the fine prediction network, the extended feature map also needs to undergo a linear projection transformation:

[0053] A 3×3 convolutional layer is used to perform local context awareness and model inter-channel correlations on the concatenated features. A weight-sharing mechanism is used to extract common features across channels. A batch normalization (BatchNorm) layer is then used to normalize the feature distribution and suppress scale variations in the input data. This process maps the four-channel features into a low-dimensional embedding space (e.g., 64 dimensions), forming the initial input features for the fine prediction network. This fusion mechanism not only inherits the constraints on background blur areas imposed by the coarse prediction stage, but also preserves high-frequency details in the original image, effectively avoiding the inpainting bias caused by a single feature source.

[0054] After the encoder of the fine prediction network branch receives the expanded feature map after linear projection, it performs deep feature mining through multi-layer cascaded Transformer modules. Each Transformer module consists of a dual mechanism of channel-level multi-head self-attention (CW-MSA) and background interaction attention (BIA): Among them, the CW-MSA module divides the feature map into multiple subspaces across channels, and concurrently counts the global dependencies between different channels, thereby enhancing the model's ability to capture different semantic features through a multi-head mechanism. In order to further strengthen the correlation between the highlight area and the background, the encoder introduces the background interaction attention (BIA) module, which uses the blurred background image generated by the coarse prediction network as a priori to dynamically adjust the attention weight distribution. The calculation process of BIA can be expressed as:

[0055] ,

[0056] Where, For the The output features of the layer Transformer include the query matrix Q, key matrix K and value matrix V in the attention mechanism. is the correlation matrix of bright and dark regions, is the background mask interactive attention, is the normalized exponential function, T is the matrix transpose, d is the feature dimension, is the weight coefficient, and E is the identity matrix. This module effectively suppresses over-enhancement in highlight areas by fusing background priors with adaptive attention weights. A downsampling operation with a stride of 2 is also implemented at the end of each encoder layer to gradually compress the feature space dimensions, thereby focusing on extracting global semantic information. After processing by the multi-stage encoder, the feature map gradually transitions from high-frequency details to low-frequency semantic representations, ultimately outputting a mixed highlight-background feature map.

[0057] Based on the highlight-background mixed feature map output by the fine prediction network encoder, a channel-wise global average pooling (GAP) operation is performed on the mixed feature map to extract the spatial saliency distribution. This operation compresses the channel dimension to a single channel through a 1×1 convolutional layer, and then normalizes it using a sigmoid function to generate a content mask, where regions with high response values ​​correspond to potential highlight locations. Based on this, the mask is used to spatially weight the mixed feature map to enhance the contextual association of highlight areas.

[0058] After receiving the feature map after multi-level enhancement, the decoder of the fine prediction network branch gradually restores the image details through the inverse feature map and global-local information fusion. The decoder consists of multiple layers of cascaded multi-head self-attention (MSA) modules. Each layer contains channel-level multi-head self-attention (CW-MSA) and upsampling operations: the CW-MSA module captures the semantic association between the highlight area and the surrounding background by reconstructing the global dependency across channels; the upsampling layer gradually expands the spatial resolution of the feature map through bilinear interpolation or deconvolution, and combines the jump connection of the encoder stage to reuse contextual features of different scales to compensate for the loss of details in the upsampling process. In the last layer of the decoder, the high-dimensional features are mapped to the residual space through 3×3 convolution to generate a residual map. , and perform pixel-level superposition with the original input image, and finally output a non-highlight prediction image The non-highlight prediction image is subjected to dynamic range adjustment and sharpening enhancement. The dynamic range adjustment limits the pixel values ​​to the interval [0, 255]. The sharpening enhancement uses non-local mean filtering to improve the clarity of local details. To quantify the reconstruction effect, a second loss function based on Charbonnier loss is constructed:

[0059] ,

[0060] in, For the second loss, For the non-highlight prediction map, is a constant whose value is The loss function is smoothed by introducing a smoothing factor This effectively alleviates the optimization oscillation problem of the L1 / L2 loss in gradient-sparse regions and improves the model's sensitivity to subtle texture differences. Furthermore, the channel attention mechanism (CCA) embedded in the decoder works in synergy with the spatial weighting strategy to ensure natural brightness transitions in highlight areas and clear, coherent background textures during reconstruction.

[0061] Finally, based on the constructed first loss function and second loss function, the final total loss function can be obtained:

[0062] ,

[0063] in, is the total loss, is the weight of the first loss, is the weight of the second loss. By giving a higher weight to the fine prediction loss, the model focuses more on optimizing the quality of detail reconstruction in the later stage of training, while the coarse prediction loss serves as an auxiliary constraint to ensure the stability of the background blur feature.

[0064] In a specific optimization example, the Adam optimizer was used with an initial learning rate of 0.0002 and an exponential decay of 0.5 every 10 training epochs for a total of 200 epochs to balance convergence speed and training accuracy. To avoid overfitting, a dynamic gradient clipping strategy was introduced during training, limiting the gradient norm to no more than 0.1. An early stopping mechanism (with a patience of 20) was also implemented to monitor the validation set loss and prevent the model from falling into a local optimum. Experiments demonstrated that this optimization strategy effectively coordinated the parameter update rhythm of the two-branch network. The coarse prediction network quickly converged to a stable representation of background blur features, while the fine prediction network gradually refined local texture and illumination consistency. After full training, the model achieved a PSNR exceeding 36dB and an SSIM exceeding 0.99 on the SHIQ and LIME test sets. The inference time for a single image was only 0.075 seconds. The model demonstrated strong robustness in natural scenes with varying illumination intensities and high light density, providing an efficient and reliable solution for practical applications.

[0065] In summary, the highlight removal method proposed in the present invention, which is based on a granular interactive hybrid highlight removal model, does not need to rely on manually labeled highlight masks. It only needs to generate a highlight-free blurred background reference image through a coarse prediction network as auxiliary supervision, which not only reduces the labeling cost but also avoids the limitation of subjective errors on the generalization ability of the model.

[0066] By splicing the blurred background image with the original image and projecting them into extended features, and combining the global multi-head self-attention and background interactive attention modules of the precision prediction network, it is possible to dynamically capture the local-global correlation of bright and dark areas and effectively suppress hue distortion and boundary artifacts.

[0067] The combination of a benchmark image generation strategy based on morphological blur operators and a two-stage optimization framework of coarse-grained blur prediction and fine-grained content reconstruction enables the model to maintain a balance between background blur suppression and detail retention in complex scenes, and the texture clarity of the restored image is significantly improved.

[0068] It should be noted that all directional indications (such as up, down, left, right, front, back, etc.) in the embodiments of the present invention are only used to explain the relative position relationship, movement status, etc. between the various components under a certain specific posture (as shown in the accompanying drawings). If the specific posture changes, the directional indication will also change accordingly.

[0069] In addition, in the present invention, descriptions such as "first," "second," and "one" are for descriptive purposes only and should not be understood to indicate or imply their relative importance or implicitly specify the number of the technical features indicated. Therefore, a feature specified as "first" or "second" may explicitly or implicitly include at least one of such features. In the description of the present invention, "plurality" means at least two, such as two, three, etc., unless otherwise specifically defined.

[0070] In the present invention, unless otherwise specified or limited, the terms "connection" and "fixation" should be understood in a broad sense. For example, "fixation" can mean fixed connection, detachable connection, or integration; mechanical connection or electrical connection; direct connection or indirect connection through an intermediate medium; internal communication between two elements or interaction between two elements, unless otherwise specified. Those skilled in the art will be able to understand the specific meanings of the above terms in the present invention based on specific circumstances.

[0071] In addition, the technical solutions between the various embodiments of the present invention can be combined with each other, but it must be based on the fact that ordinary technicians in this field can implement it. When the combination of technical solutions is mutually contradictory or cannot be implemented, it should be deemed that such a combination of technical solutions does not exist and is not within the scope of protection required by the present invention.

Claims

1. A highlight removal method based on a granularity interactive hybrid highlight removal model, characterized in that: The hybrid highlight removal model includes a coarse prediction network branch and a fine prediction network branch, including the following steps: S1: Reconstruct the original highlight image through the coarse prediction network branch to obtain a background prediction image without highlight blur, and construct a first loss function based on the distance between the background prediction image without highlight blur and the reference image without highlight blur; S2: The extended feature map is obtained by connecting the color channel of the non-highlight blurred background prediction map and the original highlight image, and the extended feature map is linearly projected and input into the fine prediction network branch; S3: Extract features from the extended feature map after linear projection transformation through the fine prediction network branch to obtain a highlight-background mixed feature map; S4: Generate a content mask based on the highlight-background mixed feature map, and perform correlation calculation and enhanced feature acquisition under enhancement on the highlight-background mixed feature map through the content mask; S5: reconstructing a non-highlight prediction map by decoding the enhanced features, and constructing a second loss function based on the distance between the non-highlight prediction map and the corresponding true de-highlighted image; S6: Optimize the parameters of the hybrid highlight removal model under the minimum optimization of the total loss function according to the weighted sum of the first loss function and the second loss function.

2. The highlight removal method based on the granularity interactive hybrid highlight removal model according to claim 1, characterized in that: The coarse prediction network branch adopts a U-shaped convolutional neural network, including an encoder and a decoder; the encoder is composed of four 3×3 pooling convolution layers with a stride of 2 and one pooling convolution layer with a stride of 1, each convolution layer is followed by a ReLU activation function; the decoder is composed of four deconvolution layers, which reuse the features of the encoder through jump connections, and the last layer is a 1×1 convolution layer with a Softmax activation function.

3. The highlight removal method based on the granularity interactive hybrid highlight removal model according to claim 1, characterized in that: In the step S1, the method for generating the non-highlight blurred background reference image includes: applying blurring processing based on a morphological operator to the corresponding real de-highlighted image to obtain the non-highlight blurred background reference image.

4. The highlight removal method based on the granularity interactive hybrid highlight removal model according to claim 1, characterized in that: In the S2 step, the specific method of connecting the color channels is: splicing the single-channel non-highlight blurred background prediction map with the RGB channels of the original highlight image along the channel dimension to form a four-channel extended feature map, and performing standardization and feature dimensionality reduction through a 3×3 convolution linear projection layer.

5. The highlight removal method based on the granularity interactive hybrid highlight removal model according to claim 1, characterized in that: The fine prediction network branch adopts the Transformer-based UNet architecture, and its encoder and decoder both contain multi-head self-attention modules. The multi-head self-attention module of the encoder includes a series of channel-level multi-head self-attention and downsampling layers, and the multi-head self-attention module of the decoder includes a series of channel-level multi-head self-attention and upsampling layers.

6. The highlight removal method based on the granularity interactive hybrid highlight removal model according to claim 5, characterized in that: The encoder of the fine prediction network branch also introduces a background interaction attention module to guide the correlation between bright and dark areas. The formula is expressed as: Where, For the The output features of the layer Transformer include the query matrix Q, key matrix K and value matrix V in the attention mechanism. is the correlation matrix of bright and dark regions, is the background mask interactive attention, is the normalized exponential function, T is the matrix transpose, d is the feature dimension, is the weight coefficient, and E is the unit matrix.

7. The highlight removal method based on the granularity interactive hybrid highlight removal model according to claim 1, characterized in that: In the step S1, the formula of the first loss function is expressed as: Where, For the first loss, is the original highlight image, This is a predicted image with no highlight blur background. To truly remove the highlight image, is the morphological operator, is the L1 norm distance.

8. The highlight removal method based on the granularity interactive hybrid highlight removal model according to claim 7, characterized in that: In the step S5, the formula of the second loss function is expressed as: Where, For the second loss, For the non-highlight prediction map, is a constant whose value is .

9. The highlight removal method based on the granularity interactive hybrid highlight removal model according to claim 8, characterized in that: In step S6, the weights of the first loss function and the second loss function in the total loss function are adjusted according to the balance between the constraints of background blur prediction and the final reconstruction quality, and the formula is expressed as follows: Where, is the total loss, is the weight of the first loss, is the weight of the second loss.

10. The highlight removal method based on the granularity interactive hybrid highlight removal model according to claim 1, characterized in that: In the step S5, after the reconstruction of the non-highlight prediction image, a post-processing step is also included: dynamic range adjustment and sharpening enhancement processing are performed on the non-highlight prediction image, the dynamic range adjustment limits the pixel value to the interval [0, 255], and the sharpening enhancement uses non-local mean filtering to improve the clarity of local details.

Images