Highlight removal method constructed based on granularity interactive mixing highlight removal model

Abstract

The invention discloses a highlight removal method constructed based on a granularity interactive mixing highlight removal model, and relates to the technical field of image processing, and the method mainly comprises the steps: constructing a first loss function based on the distance between a non-highlight fuzzy background prediction image and a non-highlight fuzzy background reference image; performing color channel connection on the non-highlight fuzzy background prediction image and the original highlight image to obtain an expanded feature map, and performing linear projection transformation on the expanded feature map; obtaining a mixed feature map based on the extended feature map after linear projection transformation; correlation calculation and enhanced feature acquisition under enhancement are carried out based on the mixed feature map; reconstructing a highlight-free prediction map based on the enhanced features, and constructing a second loss function of the distance between the highlight-free prediction map and the corresponding real highlight-free image; and performing mixed highlight removal model parameter optimization under the minimum optimization of the total loss function. According to the invention, through combination of two-stage optimization frameworks of coarse and fine granularity, the model keeps balance between background fuzzy suppression and detail retention in a complex scene.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and particularly relates to a specular highlight removal method based on the construction of a granularity interaction hybrid highlight removal model. Background Art

[0002] In the prior art, specular highlight removal methods mainly rely on traditional image processing algorithms and early deep learning models, but there are still significant limitations. Traditional methods mostly adopt physically-based illumination model optimization or low-level image processing techniques (such as clustering, filtering, etc.), and achieve highlight suppression through the information propagation of diffuse reflection pixels to the specular reflection area. Although such methods can adjust the overall illumination intensity, it is difficult to accurately locate the highlight area, resulting in poor local repair effects, high computational complexity, and difficulty in adapting to large-scale data scenarios. With the development of deep learning technology, supervised highlight removal models have made certain progress through training with labeled data. However, they rely on manually labeled highlight masks or intensity maps as auxiliary supervision signals, which not only have high labeling costs but are also easily affected by subjective judgment errors, restricting the generalization ability of the models. In addition, existing methods generally lack an effective modeling mechanism for the local details and global semantic associations, with insufficient interaction between bright and dark regions during the repair process, which is prone to problems such as tone distortion and boundary artifacts. Especially for complex natural scenes (such as cases with variable illumination and dense highlight distribution), existing models mostly adopt a single-stage repair strategy, making it difficult to balance the requirements of background blur suppression and detail preservation, resulting in texture blur or local detail loss in the reconstructed images, seriously affecting the input quality of downstream visual tasks (such as object detection, semantic segmentation). Summary of the Invention

[0003] To solve the problems existing in the existing specular highlight removal process, the present invention proposes a specular highlight removal method based on the construction of a granularity interaction hybrid highlight removal model. The hybrid highlight removal model includes a coarse prediction network branch and a fine prediction network branch, and the method includes the following steps: S1: Reconstruct the original highlight image through the coarse prediction network branch to obtain a non-highlight blurred background prediction map, and construct a first loss function based on the distance between the non-highlight blurred background prediction map and the non-highlight blurred background reference map; S2: Obtain an extended feature map by connecting the color channels of the non-highlight blurred background prediction map and the original highlight image, and input the linearly projected and transformed extended feature map into the fine prediction network branch; S3: Extract features from the linearly projected and transformed extended feature map through the fine prediction network branch to obtain a highlight-background hybrid feature map; S4: Generate a content mask based on the highlight-background hybrid feature map, and obtain enhanced features through correlation calculation and enhancement of the highlight-background hybrid feature map with the content mask; S5: Decode the enhanced features to reconstruct the specular-free prediction map, and construct a second loss function based on the distance between the specular-free prediction map and the corresponding ground truth de-specularized image; S6: Optimize the parameters of the hybrid specular removal model under the minimum optimization of the total loss function according to the weighted sum of the first loss function and the second loss function.

[0004] The present invention does not need to rely on manually annotated specular masks. Only by generating a specular-free blurred background reference map through a coarse prediction network as auxiliary supervision, it not only reduces the annotation cost, but also avoids the limitation of the subjective error on the generalization ability of the model. At the same time, the combination of the reference map generation strategy and the two-stage optimization framework of coarse-grained blurred prediction and fine-grained content reconstruction enables the model to still maintain the balance between background blur suppression and detail retention in complex scenes, and the texture clarity of the repaired image is significantly improved.

[0005] Further, the coarse prediction network branch adopts a U-shaped convolutional neural network, including an encoder and a decoder; the encoder is composed of four 3×3 pooling convolutional layers with a stride of 2 and a pooling convolutional layer with a stride of 1 connected in series, and each convolutional layer is followed by a ReLU activation function; the decoder is composed of four transposed convolutional layers, and the features of the encoder are reused through skip connections, and the last layer is a 1×1 convolutional layer with a Softmax activation function.

[0006] Further, in the S1 step, the generation method of the specular-free blurred background reference map includes: performing fuzzy processing on the corresponding ground truth de-specularized image by using a morphological operator to obtain the specular-free blurred background reference map.

[0007] Further, in the S2 step, the specific connection method of the color channels is: splicing the single-channel specular-free blurred background prediction map and the RGB channels of the original specular image along the channel dimension to form a four-channel extended feature map, and performing normalization and feature dimensionality reduction through a linear projection layer of 3×3 convolution.

[0008] Further, the fine prediction network branch adopts a UNet architecture based on Transformer, and both its encoder and decoder contain multi-head self-attention modules. The multi-head self-attention module of the encoder includes a channel-level multi-head self-attention and a downsampling layer connected in series, and the multi-head self-attention module of the decoder includes a channel-level multi-head self-attention and an upsampling layer connected in series.

[0009] Further, in the encoder of the fine prediction network branch, the guidance of the correlation between bright and dark regions is also carried out by introducing a background interaction attention module, and the formula is expressed as: In the formula, is the The output features of the layer Transformer include the query matrix Q, the key matrix K, and the value matrix V in the attention mechanism. is the correlation matrix of bright and dark regions. is the background mask interactive attention. is the normalized exponential function, T is the matrix transpose, and d is the feature dimension. is the weight coefficient, and E is the identity matrix.

[0010] Furthermore, in the step S1, the formula of the first loss function is expressed as: In the formula, is the first loss. is the original highlight image. is the predicted map of the blurred background without highlights. is the real highlight-removed image. is the morphological operator. is the L1 norm distance.

[0011] Furthermore, in the step S5, the formula of the second loss function is expressed as: In the formula, is the second loss. is the predicted map without highlights. is a constant, and its value is .

[0012] Furthermore, in the step S6, the weights of the first loss function and the second loss function in the total loss function are adjusted according to the balance between the constraint of the background blur prediction and the final reconstruction quality. The formula is expressed as follows: In the formula, is the total loss. is the weight of the first loss. is the weight of the second loss.

[0013] Furthermore, in the step S5, after reconstructing the predicted map without highlights, there is also a post-processing step: dynamically adjusting the range and sharpening and enhancing the predicted map without highlights. The dynamic range adjustment limits the pixel values to the interval [0, 255], and the sharpening and enhancement uses non-local mean filtering to improve the local detail sharpness.

[0014] Compared with the prior art, the present invention has at least the following beneficial effects: (1)A highlight removal method based on the construction of a granularity interaction hybrid highlight removal model proposed by the present invention does not rely on manually annotated highlight masks. It only needs to generate a highlight-free blurred background reference map through a coarse prediction network as auxiliary supervision, which not only reduces the annotation cost but also avoids the limitation of the model generalization ability caused by subjective errors. (2)By splicing the blurred background map with the original image and projecting it into extended features, combined with the global multi-head self-attention and background interaction attention modules of the fine prediction network, it can dynamically capture the local-global correlation of bright and dark regions, effectively suppressing the problems of tone distortion and boundary artifacts. (3)The combination of the reference map generation strategy based on the morphological fuzzy operator and the two-stage optimization framework of coarse-grained fuzzy prediction and fine-grained content reconstruction enables the model to still maintain the balance between background blur suppression and detail retention in complex scenes, and the texture clarity of the restored image is significantly improved. Description of the Drawings

[0015] Figure 1 It is a step diagram of a highlight removal method based on the construction of a granularity interaction hybrid highlight removal model; Figure 2 It is a schematic diagram of the network structure of the hybrid highlight removal network model. Detailed Embodiments

[0016] The following are specific embodiments of the present invention in combination with the drawings, and the technical solutions of the present invention are further described, but the present invention is not limited to these embodiments.

[0017] In the field of natural scene image processing, specular highlight removal is a key step in improving image quality and the performance of downstream vision tasks. However, existing methods based on traditional image processing or single deep learning frameworks have significant bottlenecks: on the one hand, traditional methods rely on physical model optimization or low-level filtering operations, and it is difficult to accurately distinguish highlight regions from normal illumination regions, resulting in local over-smoothing or artifacts in the restored images; on the other hand, although supervised deep learning models can learn highlight features through data-driven methods, they rely on manually annotated highlight masks or intensity maps as supervision signals, which not only have high annotation costs but also are prone to model generalization ability limitations due to annotation errors. In addition, existing methods mostly adopt a single repair stage and lack a collaborative optimization mechanism for background blur perception and global content consistency, and it is difficult to balance the requirements of detail retention and artifact suppression in complex scenes (such as high dynamic illumination, dense highlight interference). To address the above problems, the present invention proposes a highlight removal method based on the construction of a granularity interaction hybrid highlight removal model. By automatically generating a highlight-free blurred background map through a coarse prediction network as a priori guidance, combined with the dynamic feature fusion mechanism of the multi-head self-attention (MSA) and background interaction attention (BIA) modules in the fine prediction network, accurate localization and natural reconstruction of highlight regions are achieved, as Figure 1As shown in the figure, it specifically includes the following steps: S1: Reconstruct the original specular highlight image through the coarse prediction network branch to obtain a specular highlight-free blurred background prediction map, and construct a first loss function based on the distance between the specular highlight-free blurred background prediction map and the specular highlight-free blurred background reference map; S2: Obtain an extended feature map by connecting the color channels of the specular highlight-free blurred background prediction map and the original specular highlight image, and input the linearly projected and transformed extended feature map into the fine prediction network branch; S3: Extract features from the linearly projected and transformed extended feature map through the fine prediction network branch to obtain a specular highlight-background mixed feature map; S4: Generate a content mask based on the specular highlight-background mixed feature map, and obtain an enhanced feature through correlation calculation and enhancement of the specular highlight-background mixed feature map with the content mask; S5: Reconstruct a specular highlight-free prediction map by decoding the enhanced feature, and construct a second loss function based on the distance between the specular highlight-free prediction map and the corresponding real specular highlight-removed image; S6: Optimize the parameters of the mixed specular highlight removal model under the minimum optimization of the total loss function according to the weighted sum of the first loss function and the second loss function.

[0018] In the specific implementation process, the data preparation stage is an important foundation for constructing the specular highlight removal model. In this embodiment, a data set containing the original specular highlight image and its corresponding real specular highlight-removed image is used for model training and verification. The input data mainly comes from publicly available specular highlight repair benchmark data sets (such as SHIQ, LIME), and at the same time covers diverse samples with different lighting conditions and specular highlight distribution densities in natural scenes. To ensure data quality and model generalization ability, the input images need to go through a standardized preprocessing process: First, perform a normalization operation on the original specular highlight image in the RGB channel to map the pixel values to the interval [0, 1] to eliminate the illumination intensity difference; second, uniformly adjust the image resolution to M×M pixels (set the pixel size according to actual needs) to avoid network training fluctuations caused by inconsistent sizes; on this basis, expand the sample diversity through data augmentation strategies such as random cropping and horizontal flipping to simulate complex perspectives and local occlusion scenarios in actual applications.

[0019] In addition, to address the problem of high annotation cost for specular highlight regions, the present invention adopts a weakly supervised learning framework and relies only on paired original specular highlight images and real specular highlight-removed images for end-to-end training, without the need for additional annotation of specular highlight masks or intensity maps, reducing the need for manual intervention. To verify the rationality of the data distribution, the data set needs to be manually screened and quality evaluated before training, and abnormal samples with serious noise, motion blur, or annotation errors are excluded to ensure the effectiveness and consistency of the input data.

[0020] After the above processing of the training data, the original highlight image (with a size of M×M×3) is input into the hybrid highlight removal network model for training. In the present invention, the hybrid highlight removal network model adopts a dual-branch collaborative architecture design including a coarse prediction network branch (CoarseNet) and a fine prediction network branch (as shown in Figure 2 ), so as to take into account both local feature extraction and global semantic association.

[0021] Among them, the coarse prediction network branch, as the front-end processing module, quickly captures the blurred background information of the image based on the lightweight U-shaped convolutional structure: its encoder is composed of multiple layers of 3×3 pooling convolutions stacked in series, gradually compressing the spatial dimension through the downsampling operation with a stride of 2, and enhancing the non-linear representation ability by combining with the ReLU activation function; the decoder gradually restores the feature resolution through the deconvolution layer, and uses the skip connection mechanism to reuse the local detail features of different levels in the encoder, and finally outputs a single-channel highlight-free blurred background prediction map. Here, in the present invention, through the downsampling and upsampling operations, the computational amount during morphological processing is reduced by capturing the detail features of downsampling, and then through upsampling, the image is restored to the original resolution for the calculation of the first loss.

[0022] At the same time, the fine prediction network branch (RefineFormer), as the back-end optimization module, adopts a UNet architecture integrating Transformer and CNN, and dynamically models the global context relationship of the image through the multi-head self-attention mechanism. After the input image and the coarse prediction result are concatenated along the channel dimension, they are mapped into low-dimensional embedded features through the linear projection layer as the input of the fine prediction network branch. This branch particularly introduces a background interaction attention (BIA) module, uses the blurred background map generated by the coarse prediction as a prior guide, and strengthens the cross-region association between the bright area and the dark area in the feature space to avoid the tone break problem caused by local repair in traditional methods. The two branches achieve feature complementarity through the cascading method. The coarse prediction network provides the initial constraint for background blur suppression, and the fine prediction network then performs detailed refinement reconstruction on this basis, finally forming an end-to-end highlight removal pipeline, taking into account both processing efficiency and repair quality.

[0023] Specifically, after receiving the processed original highlight image, the coarse prediction network branch performs multi-scale feature extraction and reconstruction on the input through its U-shaped convolutional architecture: the encoder uses multiple layers of 3×3 pooling convolutions to gradually compress the spatial dimension of the image and capture the low-frequency blurred features of the background area; the decoder then combines the deconvolution layer with the skip connection to reuse the local detail information of different levels in the encoder, gradually restores the image resolution, and finally outputs a single-channel highlight-free blurred background prediction map.

[0024] To enhance the supervision robustness of the coarse prediction network, the present invention further introduces a morphological operator to generate a no-specular highlight blurred background reference map. First, the real specular highlight removed image is downsampled to 20% of the original resolution, and the dilation operation is used to blur the low-frequency region, and then upsampled to the original size to simulate the blurred characteristics of the natural background. By calculating the L1 norm distance between the no-specular highlight blurred background prediction map and the no-specular highlight blurred background reference map, a coarse-grained loss function (the first loss function) is constructed: , where, is the first loss, is the original specular highlight image, is the no-specular highlight blurred background prediction map, is the real specular highlight removed image, is the morphological operator, is the L1 norm distance. Here, by constraining the network to learn the consistent features of background blur through the coarse-grained loss function, not only the computational complexity of the subsequent fine prediction network is reduced, but also the network is implicitly guided to distinguish the specular highlight region from the normal illumination region, avoiding the deviation caused by directly relying on manual annotation. The generation of the no-specular highlight blurred background prediction map not only provides a semantic prior for the fine prediction network, but also adaptively enhances the background information through end-to-end optimization.

[0025] Then, taking the no-specular highlight blurred background prediction map generated by the coarse prediction network as a bridge, the present invention enhances the completeness of the input information through a multi-modal feature fusion strategy. Specifically, the present invention splices the single-channel no-specular highlight blurred background prediction map (with a size of M×M×1) and the RGB three-channel original specular highlight image (with a size of M×M×3) along the color channel dimension to form a four-channel extended feature map (with a size of M×M×4), so as to embed the background blur prior provided by the coarse prediction network while retaining the original color and texture information. To further adapt to the feature processing requirements of the fine prediction network, the extended feature map also needs to undergo a linear projection transformation: A 3×3 convolutional layer is used to perform local context awareness and inter-channel correlation modeling on the spliced features, extract the common features across channels through the weight sharing mechanism, and combine the batch normalization (BatchNorm) layer to standardize the feature distribution and suppress the scale difference of the input data. This process maps the four-channel features to a low-dimensional embedding space (such as 64 dimensions) to form the initial input features of the fine prediction network. Through this fusion mechanism, the model not only inherits the constraint on the background blur region in the coarse prediction stage, but also retains the high-frequency details of the original image, effectively avoiding the restoration deviation problem caused by a single feature source.

[0026] After the encoder of the fine prediction network branch receives the extended feature map after linear projection, it performs deep feature mining through a multi-level cascaded Transformer module. Each Transformer module consists of a dual mechanism of channel-wise multi-head self-attention (CW-MSA) and background interaction attention (BIA): Among them, the CW-MSA module divides the feature map into multiple cross-channel subspaces, and statistically calculates the global dependencies between different channels in parallel. The multi-head mechanism is used to enhance the model's ability to capture different semantic features. To further strengthen the correlation between the highlight region and the background, the encoder introduces a background interaction attention (BIA) module, which uses the blurred background map generated by the coarse prediction network as a prior to dynamically adjust the attention weight distribution. The calculation process of BIA can be expressed as: , In the formula, is the output feature of the -th layer of the Transformer, which includes the query matrix Q, key matrix K, and value matrix V in the attention mechanism, is the correlation matrix between the bright and dark regions, is the background mask interaction attention, is the normalized exponential function, T is the matrix transpose, d is the feature dimension, is the weight coefficient, and E is the identity matrix. This module effectively suppresses the over-enhancement phenomenon in the highlight region by fusing the background prior and the adaptive attention weight. A downsampling operation with a stride of 2 is also set at the end of each layer of the encoder to gradually compress the feature space dimension, thereby focusing on the extraction of global semantic information. After being processed by multiple levels of the encoder, the feature map gradually transitions from high-frequency details to low-frequency semantic representations, and finally outputs a highlight-background mixed feature map.

[0027] Based on the highlight-background mixed feature map output by the encoder of the fine prediction network, a global average pooling (GAP) operation is performed on the mixed feature map in the channel dimension to extract the saliency distribution in the spatial dimension. This operation compresses the channel dimension to a single channel through a 1×1 convolutional layer, and then generates a content mask through normalization by the Sigmoid function. The high-response value regions in it correspond to the potential highlight positions. On this basis, the mask is used to perform spatial weighting on the mixed feature map to strengthen the context correlation of the highlight region.

[0028] After receiving the feature map enhanced by multiple levels, the decoder of the fine prediction network branch gradually restores the image details through inverse feature mapping and global-local information fusion. The decoder consists of multiple cascaded multi-head self-attention (MSA) modules, each layer containing channel-wise multi-head self-attention (CW-MSA) and upsampling operations: the CW-MSA module captures the semantic associations between the highlight regions and the surrounding background by reconstructing the cross-channel global dependencies; the upsampling layer gradually enlarges the spatial resolution of the feature map through bilinear interpolation or transposed convolution, and combines the skip connections in the encoder stage to reuse the context features at different scales to make up for the detail loss during the upsampling process. In the last layer of the decoder, the high-dimensional feature map is mapped to the residual space through a 3×3 convolution to generate a residual map , which is pixel-wise superimposed with the original input image to finally output a highlight-free prediction map , and through dynamic range adjustment and sharpening enhancement processing on the highlight-free prediction map, the dynamic range adjustment restricts the pixel values to the interval [0, 255], and the sharpening enhancement uses non-local mean filtering to improve the local detail sharpness. To quantify the reconstruction effect, a second loss function based on the Charbonnier loss is constructed: , where is the second loss, is the highlight-free prediction map, is a constant, and its value is . By introducing the smoothing factor , this loss function effectively alleviates the optimization oscillation problem of the L1 / L2 loss in the gradient sparse region and improves the sensitivity of the model to small texture differences. Meanwhile, the channel attention mechanism (CCA) and spatial weighting strategy embedded in the decoder further cooperate to ensure natural brightness transition in the highlight regions and clear and coherent background textures during the reconstruction process.

[0029] Finally, according to the constructed first loss function and second loss function, the final total loss function can be obtained: , where is the total loss, is the weight of the first loss, is the weight of the second loss. By assigning a higher weight to the fine prediction loss, the model focuses more on optimizing the detail reconstruction quality in the later stage of training, while the coarse prediction loss serves as an auxiliary constraint to ensure the stability of the background blur features.

[0030] In a specific optimization example, the Adam optimizer is adopted in the optimization process. The initial learning rate is set to 0.0002, and it decays exponentially at a rate of 0.5 every 10 training epochs for a total of 200 epochs to balance the convergence speed and training accuracy. To avoid overfitting, a dynamic gradient clipping strategy is introduced during training to limit the gradient norm to no more than 0.1. At the same time, an early stopping mechanism (patience = 20) is used to monitor the validation set loss to prevent the model from falling into local optima. Experiments show that this optimization strategy can effectively coordinate the parameter update rhythm of the dual-branch network. The coarse prediction network quickly converges to a stable expression of the background-blurred features, while the fine prediction network gradually refines the local texture and illumination consistency. After complete training, the model achieves a PSNR of over 36 dB and an SSIM of over 0.99 on the SHIQ and LIME test sets. The inference time for a single image is only 0.075 seconds, and it shows strong robustness in natural scenes with different light intensities and highlight densities, providing an efficient and reliable solution for practical applications.

[0031] In summary, the highlight removal method proposed by the present invention for constructing a highlight removal model based on granularity interaction and mixing does not rely on manually annotated highlight masks. It only needs to generate a highlight-free blurred background reference map through the coarse prediction network as auxiliary supervision, which not only reduces the annotation cost but also avoids the limitation of the model generalization ability caused by subjective errors.

[0032] By splicing the blurred background map and the original image and projecting them into extended features, combined with the global multi-head self-attention and background interaction attention modules of the fine prediction network, it can dynamically capture the local-global correlation of bright and dark regions, effectively suppressing the problems of hue distortion and boundary artifacts.

[0033] The combination of the reference map generation strategy based on the morphological blur operator and the two-stage optimization framework of coarse-grained blur prediction and fine-grained content reconstruction enables the model to still maintain a balance between background blur suppression and detail retention in complex scenes, and the texture clarity of the repaired image is significantly improved.

[0034] It should be noted that all directional indications (such as up, down, left, right, front, back...) in the embodiments of the present invention are only used to explain the relative position relationship and movement conditions between components in a specific posture (as shown in the drawings). If the specific posture changes, the directional indications will also change accordingly.

[0035] In addition, in the present invention, descriptions such as "first", "second", "one", etc. are for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the quantity of the indicated technical features. Thus, features defined as "first" and "second" may explicitly or implicitly include at least one such feature. In the description of the present invention, the meaning of "a plurality" is at least two, for example, two, three, etc., unless otherwise specifically defined.

[0036] In the present invention, unless otherwise clearly defined and limited, the terms "connection", "fixation", etc. shall be understood in a broad sense. For example, "fixation" may be a fixed connection, a detachable connection, or integrated; it may be a mechanical connection or an electrical connection; it may be directly connected or indirectly connected through an intermediate medium, and it may be the internal connection of two components or the interaction relationship between two components, unless otherwise clearly limited. For those of ordinary skill in the art, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0037] In addition, the technical solutions between various embodiments of the present invention can be combined with each other, but it must be based on the ability of those of ordinary skill in the art to implement. When the combination of technical solutions is contradictory or cannot be implemented, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection required by the present invention.

Claims

1. A highlight removal method based on the construction of a granularity interaction hybrid highlight removal model, characterized in that, The hybrid specular highlight removal model includes a coarse prediction network branch and a fine prediction network branch, and comprises the steps of: S1: Reconstruct the original specular highlight image through the coarse prediction network branch to obtain a specular highlight-free blurred background prediction map, and construct a first loss function based on the distance between the specular highlight-free blurred background prediction map and the specular highlight-free blurred background reference map; S2: Obtain an extended feature map by connecting the color channels of the specular highlight-free blurred background prediction map and the original specular highlight image, and input the linearly projected and transformed extended feature map into the fine prediction network branch; S3: Extract features from the linearly projected and transformed extended feature map through the fine prediction network branch to obtain a specular highlight-background mixed feature map; S4: Generate a content mask based on the specular highlight-background mixed feature map, and obtain an enhanced feature through correlation calculation and enhancement of the specular highlight-background mixed feature map with the content mask; S5: Reconstruct a specular highlight-free prediction map by decoding the enhanced feature, and construct a second loss function based on the distance between the specular highlight-free prediction map and the corresponding real specular highlight-removed image; S6: Optimize the parameters of the hybrid specular highlight removal model by minimizing the total loss function according to the weighted sum of the first loss function and the second loss function.

2. The highlight removal method based on the construction of a granularity interaction mixing highlight removal model as claimed in claim 1, wherein The coarse prediction network branch adopts a U-shaped convolutional neural network, including an encoder and a decoder; the encoder is composed of four 3×3 pooling convolutional layers with a stride of 2 and a pooling convolutional layer with a stride of 1 connected in series, and each convolutional layer is followed by a ReLU activation function; the decoder is composed of four transposed convolutional layers, and the features of the encoder are reused through skip connections, and the last layer is a 1×1 convolutional layer with a Softmax activation function.

3. The highlight removal method based on the construction of a granularity interaction hybrid highlight removal model according to claim 1, wherein, In the step S1, the generation method of the specular highlight-free blurred background reference map includes: performing blur processing based on a morphological operator on the corresponding real specular highlight-removed image to obtain the specular highlight-free blurred background reference map.

4. The highlight removal method based on the construction of a granularity interaction mixing highlight removal model according to claim 1, characterized in that, In the step S2, the specific method of the color channel connection is: splicing the single-channel specular highlight-free blurred background prediction map and the RGB channels of the original specular highlight image along the channel dimension to form a four-channel extended feature map, and performing normalization and feature dimensionality reduction through a linear projection layer of 3×3 convolution.

5. A highlight removal method based on the construction of a granularity interaction hybrid highlight removal model as claimed in claim 1, characterized in that, The fine prediction network branch adopts a UNet architecture based on Transformer, and both its encoder and decoder contain multi-head self-attention modules. The multi-head self-attention module of the encoder includes a channel-level multi-head self-attention and a downsampling layer connected in series, and the multi-head self-attention module of the decoder includes a channel-level multi-head self-attention and an upsampling layer connected in series.

6. The highlight removal method based on the construction of a granularity interaction mixing highlight removal model according to claim 5, characterized in that, In the encoder of the fine prediction network branch, a background interaction attention module is further introduced to guide the attention to the correlation between bright and dark regions, and the formula is expressed as: Wherein, is the output feature of the -layer Transformer, including the query matrix Q, the key matrix K, and the value matrix V in the attention mechanism, is the bright-dark region correlation matrix, is the background mask interactive attention, is the normalized exponential function, T is the matrix transpose, d is the feature dimension, is the weight coefficient, and E is the identity matrix.

7. The highlight removal method based on the construction of a granularity interaction mixing highlight removal model according to claim 1, wherein In the step S1, the formula of the first loss function is expressed as: Wherein, is the first loss, is the original high - light image, is the high - light - free blurred background prediction map, is the true high - light - removed image, is the morphological operator, is the L1 - norm distance.

8. The highlight removal method based on the construction of a granularity interaction hybrid highlight removal model according to claim 7, characterized in that, In the step S5, the formula of the second loss function is expressed as: Wherein, is the second loss, is the non-highlight prediction map, is a constant, and its value is .

9. The highlight removal method based on the construction of a granularity interaction hybrid highlight removal model according to claim 8, characterized in that, In the step S6, the weights of the first loss function and the second loss function in the total loss function are adjusted according to the balance between the constraint of the background blur prediction and the final reconstruction quality, and the formula is expressed as follows: In the formula, is the total loss, is the weight of the first loss, is the weight of the second loss.

10. The highlight removal method based on the construction of a granularity interaction mixing highlight removal model according to claim 1, characterized in that, In the S5 step, after reconstructing the highlight-free prediction map, a post-processing step is further included: performing dynamic range adjustment and sharpening enhancement on the highlight-free prediction map. The dynamic range adjustment restricts the pixel values to the interval [0, 255], and the sharpening enhancement uses non-local mean filtering to improve the local detail sharpness.

Images