A method for detecting and removing highlights
By using multi-scale feature extraction based on U-Net network and mask-guided gating feature fusion, combined with multi-path processing and dual-color reflection model, the problems of accurate localization and incomplete removal in specular highlight detection and removal are solved, achieving fine removal of highlight areas and preservation of image details.
Patent Information
- Application Number
- CN202611374408.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-09-07
- Publication Date
- 2026-11-17
AI Technical Summary
Existing methods struggle to accurately locate specular highlight regions of different scales during specular highlight detection and removal. Furthermore, in scenes with large-area reflections, no clear boundaries, and overlapping textures and highlights, highlight removal is incomplete, texture recovery is insufficient, or tones are inconsistent, making it difficult to balance the accuracy of highlight detection with the ability to preserve image details.
A U-Net network based on gated void spatial pyramid pooling is used for multi-scale feature extraction. Combined with mask-guided gated feature fusion and multi-path gated refinement, a specular detection and removal network is trained through a multi-task combined loss function. The image is reconstructed using a two-color reflection model to achieve refined specular removal.
It improves the detection adaptability of specular highlight areas, reduces highlight residue and restoration artifacts, improves the restoration effect of structure, texture and color, and enhances the accuracy of highlight removal and image quality.
Smart Images

Figure CN123118568A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of industrial visual inspection and intelligent image processing, specifically to a method for highlight detection and removal. Background Technology
[0002] Specular highlights are bright areas formed by the directional reflection of light on glass, metal, transparent plastics, and coated surfaces. They can easily obscure color, texture, and edge information in images, and affect target recognition, defect detection, and industrial visual analysis. Therefore, it is necessary to detect and remove specular highlights in images.
[0003] Existing methods mainly include detection methods based on brightness or color features, separation methods based on reflection models, and image reconstruction methods based on deep learning. Some methods can achieve specular removal in general scenes, but in scenes with large-area reflections, no obvious boundaries, and overlapping textures and specular highlights, issues such as incomplete specular localization, insufficient texture recovery, specular residue, or inconsistent tones may still occur, making it difficult to balance specular detection accuracy with image detail preservation.
[0004] Therefore, how to accurately locate specular highlight areas of different scales while suppressing interference from non-highlight backgrounds, and further improve the structure preservation, texture restoration, and tone consistency after highlight removal, remains a problem to be solved in the process of specular highlight detection and removal. Summary of the Invention
[0005] Based on the shortcomings of the prior art described above, the purpose of this invention is to provide a method for detecting and removing highlights in order to solve the aforementioned technical problems.
[0006] To achieve the above objectives, the present invention provides the following technical solution: a method for detecting and removing highlights, comprising:
[0007] S1: Obtain the original specular image for training, as well as the manually labeled binary ground truth mask, real specular reflection image, and original ground truth image without specular reflection required for network training, and construct the training dataset;
[0008] S2: Input the original specular image for training into the U-Net network based on gated void spatial pyramid pooling, extract multi-scale features, and output the specular mask for training.
[0009] S3: Input the original specular image and specular mask for training into the encoding and decoding network, perform mask-guided gating feature fusion processing under the guidance of the specular mask for training, output the predicted specular reflection image for training, and reconstruct the two-color reflection model based on the original specular image and the predicted specular reflection image for training to obtain the coarse diffuse reflection image for training.
[0010] S4: Based on the original specular image, the predicted specular reflection image, and the coarse diffuse reflection image used for training, and combined with the specular region information provided by the specular mask used for training, multi-path gating refinement processing is performed. Through multi-path parallel feature processing and hierarchical feature fusion, the refined diffuse reflection image used for training is output.
[0011] S5: Based on the training specular mask, manually labeled binary ground truth mask, training predicted specular reflection image, ground truth specular reflection image, training refined diffuse reflection image and original ground truth image without specular reflection, a multi-task combined loss function is constructed. The specular detection and removal network is iteratively trained using the multi-task combined loss function to obtain the trained specular detection and removal network.
[0012] S6: Input the original highlight image to be processed into the trained highlight detection and removal network, and perform highlight detection processing, coarse highlight removal processing and fine highlight removal processing in sequence to obtain the reconstructed image after removing highlights.
[0013] S7: During the verification phase, specular removal performance metrics are calculated based on the reconstructed image after specular removal and the original ground truth image without specular removal. Specular removal performance evaluation results are generated by combining these with preset evaluation rules. The specular removal performance evaluation results include: peak signal-to-noise ratio, structural similarity index, and pixel-level reconstruction error.
[0014] The present invention is further configured such that S1 includes:
[0015] The original specular image, manually labeled binary ground truth mask, real specular reflection image, and original ground truth image without specular reflection used for training are sized uniformly. The correspondence between the original specular image, manually labeled binary ground truth mask, real specular reflection image, and original ground truth image without specular reflection used for training is established to form training samples and construct a training dataset.
[0016] The present invention is further configured such that generating a training specular mask through multi-scale feature extraction includes:
[0017] Based on the U-Net network, feature extraction is performed on the original specular image used for training to obtain the input feature map of gated hole spatial pyramid pooling processing;
[0018] The input feature map is subjected to three gated dilated convolutions, a convolution process, and a global pooling process, respectively. The dilation rates of the three gated dilated convolutions are set to 1, 4, and 8, respectively.
[0019] The present invention is further configured to perform three gated dilated convolution processes, a convolution process, and a global pooling process on the input feature map, respectively:
[0020] Each gated dilated convolution process is processed through parallel feature transformation and spatial gate weight generation to obtain feature transformation results and spatial gate weights. The spatial gate weights are then multiplied element-wise with the feature transformation results to obtain the gated dilated convolution processing result.
[0021] The results of the three gated dilated convolutions, the convolution process, and the global pooling process are concatenated along the channel dimension, and the concatenated result is then convolved to generate a specular mask for training.
[0022] The present invention is further configured such that the mask-guided gating feature fusion processing includes:
[0023] Mask-guided gating feature fusion processing is set before the downsampling stage of each encoder in the encoding / decoding network;
[0024] The training of a single channel is convolved using a specular mask to obtain mask features with the same dimension as the input feature channels in the encoding stage.
[0025] The input features in the encoding stage are subjected to two consecutive convolutions, batch normalization, and activation processing to obtain enhanced features;
[0026] The mask features and enhancement features are concatenated along the channel dimension to obtain joint features. The joint features are then subjected to gated convolution and connected with the enhancement features through residuals to obtain fused features.
[0027] The present invention is further configured to output a predicted specular reflection image for training and obtain a coarse diffuse reflection image for training, comprising:
[0028] Encoding and decoding are performed based on fused features to output a predicted specular reflection image for training.
[0029] Based on the two-color reflectance model, the original specular image used for training is subtracted from the predicted specular image used for training to obtain the coarse diffuse image used for training.
[0030] The present invention is further configured such that multi-path parallel feature processing includes:
[0031] The original specular image used for training is convolved to perform channel dimension alignment and feature compression, resulting in the first path of features;
[0032] The training coarse diffuse image is subjected to three consecutive gated convolutions and activations to obtain the second-path features;
[0033] The training image of the predicted specular reflection is preprocessed by convolution to unify the channel dimensions and enhance cross-channel correlation, resulting in a third-path feature.
[0034] The present invention is further configured such that the output of the refined diffuse reflection image for training includes:
[0035] The first and second features are concatenated along the channel dimension and then convolved to obtain the intermediate fused features.
[0036] The intermediate fusion features and the third-path features are concatenated along the channel dimension and then convolved to obtain a refined diffuse reflection image for training.
[0037] The present invention is further configured such that the multi-task combination loss function includes: mask segmentation loss, specular reconstruction loss, and image restoration loss;
[0038] Based on the training specular mask and the manually labeled binary ground truth mask, the binary cross-entropy loss and Dice loss are calculated and combined to obtain the mask segmentation loss;
[0039] Based on the predicted specular reflection image used for training, the real specular reflection image, and the specular mask used for training, the mask-weighted mean square error loss in the specular region and the perceptual loss based on the pre-trained VGG-16 network are calculated and combined to obtain the specular reconstruction loss.
[0040] Based on the refined diffuse reflection image used for training and the original ground truth image without highlights, the perceptual loss and the style loss based on the feature map Gram matrix are calculated and combined to obtain the inpainted image loss.
[0041] The mask segmentation loss, specular reconstruction loss, and image restoration loss are weighted and combined to obtain a multi-task combined loss function.
[0042] The present invention is further configured such that S7 includes:
[0043] Pixel-level reconstruction error, peak signal-to-noise ratio, and structural similarity index are calculated based on the reconstructed image after removing highlights and the original ground truth image without highlights.
[0044] When the peak signal-to-noise ratio is greater than the preset first signal-to-noise threshold and the structural similarity index is greater than the preset first similarity threshold, the highlight removal is deemed complete and the repair effect is deemed satisfactory.
[0045] When the peak signal-to-noise ratio is greater than the preset second signal-to-noise threshold and less than the preset first signal-to-noise threshold, and the structural similarity index is greater than the preset second similarity threshold and less than the preset first similarity threshold, it is determined that the highlights have been basically removed.
[0046] When the peak signal-to-noise ratio is less than the preset second signal-to-noise threshold or the structural similarity index is less than the preset second similarity threshold, the highlight removal effect is judged to be poor and the repair quality is unqualified.
[0047] This invention provides a method for specular detection and removal. The method comprises the following steps: S1: Acquiring the original specular image for training, along with manually labeled binary ground truth mask, real specular reflection image, and original specular-free ground truth image required for network training, to construct a training dataset; S2: Inputting the original specular image for training into a U-Net network based on gated void spatial pyramid pooling, and extracting features at multiple scales to output a specular mask for training; S3: Inputting the original specular image and the specular mask for training into an encoding / decoding network, performing mask-guided gated feature fusion processing under the guidance of the specular mask, outputting a predicted specular reflection image for training, and reconstructing a two-color reflection model based on the original specular image and the predicted specular reflection image to obtain a coarse diffuse reflection image for training; S4: Performing multi-path gated refinement processing based on the original specular image, the predicted specular reflection image, and the coarse diffuse reflection image, combined with specular region information provided by the specular mask, through multi-path parallel processing. Feature processing and hierarchical feature fusion are performed to output a refined diffuse reflection image for training; S5: Based on the training specular mask, manually annotated binary ground truth mask, training predicted specular reflection image, ground truth specular reflection image, training refined diffuse reflection image, and original ground truth image without specular highlights, a multi-task combined loss function is constructed. The specular detection and removal network is iteratively trained using the multi-task combined loss function to obtain the trained specular detection and removal network; S6: The original specular image to be processed is input into the trained specular detection and removal network, and specular detection processing, coarse specular removal processing, and fine specular removal processing are performed sequentially to obtain the reconstructed image after specular removal; S7: In the validation phase, specular removal performance metrics are calculated based on the reconstructed image after specular removal and the original ground truth image without specular highlights. Combined with preset evaluation rules, specular removal performance evaluation results are generated. The specular removal performance evaluation results include: peak signal-to-noise ratio, structural similarity index, and pixel-level reconstruction error. The beneficial effects include:
[0048] Multi-scale specular detection capability: Feature extraction of specular regions at different scales is performed through gated void spatial pyramid pooling, and specular-related features are filtered using spatial gating weights. This helps to improve the detection adaptability of small specular regions, local specular regions and large-area specular regions, and reduce the interference of background texture on specular localization results.
[0049] Highlight separation and image restoration quality: By training with a highlight mask to guide the fusion of encoded features, combined with a two-color reflection model to separate specular reflection components, and using multi-path gating thinning to fuse complementary information from the original image, coarse diffuse image and predicted specular reflection image, it helps to reduce highlight retention and restoration artifacts, and improve the structure, texture and color restoration effect of highlight areas.
[0050] Network training and evaluation reliability: Highlight detection, specular reflection reconstruction, and diffuse reflection image inpainting are synergistically constrained by mask segmentation loss, specular reconstruction loss, and inpainting image loss. The results are then evaluated by combining pixel-level reconstruction error, peak signal-to-noise ratio, and structural similarity index. This helps to improve the stability of the network training process and provides a clearer quantitative evaluation basis for the highlight removal effect.
[0051] The above description is only an overview of the technical solution of this application. In order to better understand the technical means of this application and to implement it in accordance with the contents of the specification, and to make the above and other objects, features and advantages of this application more obvious and understandable, the following are specific embodiments of this application. Attached Figure Description
[0052] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. In the drawings:
[0053] Figure 1 A flowchart illustrating a highlight detection and removal method is shown as an exemplary embodiment of the present invention;
[0054] Figure 2 A schematic diagram of a highlight detection and removal network structure for a highlight detection and removal method according to an exemplary embodiment of the present invention is shown.
[0055] Figure 3 A schematic diagram of a gated void space pyramid pooling processing structure for a highlight detection and removal method of the present invention is shown as an exemplary embodiment of the present invention.
[0056] Figure 4 A schematic diagram of a mask-guided gating feature fusion processing structure for a specular detection and removal method is shown as an exemplary embodiment of the present invention.
[0057] Figure 5 A dataset showing the highlight removal effect of a highlight detection and removal method according to an exemplary embodiment of the present invention;
[0058] Figure 6 This image shows a comparison of highlight removal results for an industrial object using a highlight detection and removal method, as an exemplary embodiment of the present invention. Detailed Implementation
[0059] The embodiments of the present invention will be described below with reference to the accompanying drawings and preferred embodiments. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be understood that the preferred embodiments are only for illustrating the present invention and not for limiting the scope of protection of the present invention.
[0060] It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of the present invention. Therefore, the drawings only show the components related to the present invention and are not drawn according to the actual number, shape and size of the components in the actual implementation. In the actual implementation, the form, quantity and proportion of each component can be arbitrarily changed, and the layout of the components may also be more complex.
[0061] In the following description, numerous details are explored to provide a more thorough explanation of embodiments of the invention. However, it will be apparent to those skilled in the art that embodiments of the invention may be practiced without these specific details. In other embodiments, well-known structures and devices are shown in block diagram form rather than in detail to avoid obscuring embodiments of the invention.
[0062] Example:
[0063] A method for detecting and removing highlights, such as Figure 1 As shown, it includes:
[0064] S1: Obtain the original specular image for training, as well as the manually labeled binary ground truth mask, real specular reflection image, and original ground truth image without specular reflection required for network training, and construct the training dataset;
[0065] S2: Input the original specular image for training into the U-Net network based on gated void spatial pyramid pooling, extract multi-scale features, and output the specular mask for training.
[0066] S3: Input the original specular image and specular mask for training into the encoding and decoding network, perform mask-guided gating feature fusion processing under the guidance of the specular mask for training, output the predicted specular reflection image for training, and reconstruct the two-color reflection model based on the original specular image and the predicted specular reflection image for training to obtain the coarse diffuse reflection image for training.
[0067] S4: Based on the original specular image, the predicted specular reflection image, and the coarse diffuse reflection image used for training, and combined with the specular region information provided by the specular mask used for training, multi-path gating refinement processing is performed. Through multi-path parallel feature processing and hierarchical feature fusion, the refined diffuse reflection image used for training is output.
[0068] S5: Based on the training specular mask, manually labeled binary ground truth mask, training predicted specular reflection image, ground truth specular reflection image, training refined diffuse reflection image and original ground truth image without specular reflection, a multi-task combined loss function is constructed. The specular detection and removal network is iteratively trained using the multi-task combined loss function to obtain the trained specular detection and removal network.
[0069] S6: Input the original highlight image to be processed into the trained highlight detection and removal network, and perform highlight detection processing, coarse highlight removal processing and fine highlight removal processing in sequence to obtain the reconstructed image after removing highlights.
[0070] S7: During the verification phase, specular removal performance metrics are calculated based on the reconstructed image after specular removal and the original ground truth image without specular removal. Specular removal performance evaluation results are generated by combining these with preset evaluation rules. The specular removal performance evaluation results include: peak signal-to-noise ratio, structural similarity index, and pixel-level reconstruction error.
[0071] The overall structure of the highlight detection and removal network is as follows: Figure 2 As shown, the specular detection and removal network comprises a specular detection stage, a coarse specular removal stage, and a fine specular removal stage connected in sequence. The specular detection stage is used to generate a specular mask for training; the coarse specular removal stage is used to generate a predicted specular reflection image and a coarse diffuse reflection image for training; and the fine specular removal stage is used to generate a fine diffuse reflection image for training.
[0072] The present invention is further configured such that S1 includes:
[0073] The original specular image, manually annotated binary ground truth mask, real specular reflection image, and original ground truth image without specular highlights used for training are resized to a uniform size. A correspondence is established between the original specular image, the manually annotated binary ground truth mask, the real specular reflection image, and the original ground truth image without specular highlights, forming training samples and constructing a training dataset. Specifically, the original specular image used for training is a color image containing specular highlight regions, used as input data for the specular detection and removal network; the manually annotated binary ground truth mask is a single-channel binary image corresponding to the original specular image used for training, where a pixel value of 1 represents a highlight region and a pixel value of 0 represents a non-highlight region, used to supervise the generation of the specular mask; the real specular reflection image is a ground truth image of the specular reflection component corresponding to the original specular image used for training, used to supervise the generation of the predicted specular reflection image; and the original ground truth image without specular highlights is a reference image without specular highlights corresponding to the original specular image used for training, used to supervise the generation of the refined diffuse reflection image. The training images—original specular images, real specular reflection images, and original ground truth images without specular highlights—are in a three-channel color image format consisting of red, green, and blue channels. Manually annotated binary ground truth masks are in a single-channel image format. Training data can utilize the SHIQ, SD1, and LIME datasets. SHIQ is a high-quality real-world specular highlight dataset, including specular images of transparent plastic, glass, and metal materials. 35,300 images are used for training, 4,000 for testing, and 4,000 for validation. SD1 is a text specular image dataset, containing 36,000 training images, 4,000 test images, and 1,950 validation images, used to examine the texture information recovery effect during specular removal. LIME is a synthetic dataset, containing 278,000 training images, 32,000 test images, and 32,000 validation images. The HLR dataset was used for generalization ability testing. It includes 183 images of industrial parts with different sizes and shapes, such as bolts, ducts, and wheelsets. The Glass and Aperture datasets were used for generalization ability testing in industrial image scenes. The Glass dataset includes 648 glass images, and the Aperture dataset includes 55 camera circle images. Both datasets were acquired using a flexible image acquisition platform with a multi-degree-of-freedom robotic arm and multi-light source coordinated illumination. The platform uses the robotic arm as the motion execution platform and adjustable light sources are arranged above, to the left, and to the right. By changing the lighting conditions and shooting angle, industrial images containing varying degrees of specular highlight areas were acquired.Training images from the SHIQ, SD1, and LIME datasets can be used as training data for the specular detection and removal network, while the corresponding test images are used to test the specular removal performance after training. The HLR, Glass, and Aperture datasets are used for generalization ability testing after training and are not involved in the training or parameter updates of the corresponding specular detection and removal networks. For the same training sample, corresponding training images, manually annotated binary ground truth masks, real specular reflection images, and original images without specular highlights are read, and the four types of images are uniformly resized to 256×256 pixels using existing image resizing methods. The four types of images use the same cropping range and resizing parameters, and during the resizing process, the same pixel coordinates are maintained to correspond to the same subject position to avoid spatial misalignment between the training images and the three types of supervised images. After resizing, the manually annotated binary ground truth masks continue to retain both pixel values of 1 and 0. After unifying the dimensions, a correspondence is established between the original specular image, manually annotated binary ground truth mask, real specular reflection image, and original ground truth image without specular highlights, based on the sample numbers. The same sample number corresponds to the same subject, the same shooting angle, and the same image spatial location. Each training sample includes one original specular image, one manually annotated binary ground truth mask, one real specular reflection image, and one original ground truth image without specular highlights. When constructing training samples, the sample number, image size, and spatial location of the four types of images are first checked for consistency. If all four are consistent, the four types of images are combined into one training sample; if any condition is inconsistent, the four types of images are not combined into a training sample, and the corresponding images are re-matched. All training samples are compiled to form the training dataset. During training, the original specular image is input into the specular detection and removal network, the manually annotated binary ground truth mask is used in the mask segmentation loss calculation, the real specular reflection image is used in the specular reconstruction loss calculation, and the original ground truth image without specular highlights is used in the image restoration loss calculation. By unifying image size and establishing fixed correspondences, the positions of highlight regions, specular reflection components, and non-highlight reference content are kept consistent in pixel space, providing mutually matching input and supervision data for subsequent highlight mask generation, specular reflection image prediction, and refined diffuse reflection image reconstruction.
[0074] The present invention is further configured such that generating a training specular mask through multi-scale feature extraction includes:
[0075] Based on the U-Net network, feature extraction is performed on the original specular image used for training to obtain the input feature map of gated hole spatial pyramid pooling processing;
[0076] The input feature map is subjected to three gated dilated convolutions, a convolution process, and a global pooling process, respectively. The dilation rates of the three gated dilated convolutions are set to 1, 4, and 8, respectively.
[0077] The input feature map is processed by three gated dilated convolutions, convolutions, and global pooling:
[0078] Each gated dilated convolution process is processed through parallel feature transformation and spatial gate weight generation to obtain feature transformation results and spatial gate weights. The spatial gate weights are then multiplied element-wise with the feature transformation results to obtain the gated dilated convolution processing result.
[0079] The results of the three gated dilated convolutions, the convolutional processing, and the global pooling are concatenated along the channel dimension, and then convolution is performed on the concatenated result to generate a specular mask for training. Specifically, the original specular image for training with uniform size, completed by S1, is read. The original specular image for training uses a 256×256 pixel three-channel color image format. An existing U-shaped encoder-decoder network is used to perform encoding feature extraction, decoding resolution restoration, and same-scale feature connection. The encoding process extracts brightness, color, edge, texture, and spatial structure information from the original specular image for training layer by layer, forming the input feature map for gated dilated spatial pyramid pooling processing. The input feature map is denoted as... The input feature map contains multiple feature channels, each recording a unique image feature response. To improve the accuracy of the specular mask, gated dilated spatial pyramid pooling is employed in the U-shaped encoder-decoder network. This gated dilated spatial pyramid pooling uses three gated dilated convolutions with dilation rates set to 1, 4, and 8, respectively. The specific structure of the gated dilated spatial pyramid pooling is as follows: Figure 3As shown, the input feature map is processed by three gated dilated convolution branches with dilation rates of 1, 4, and 8, one 1×1 convolution branch, and one global pooling branch. The global pooling result, after 1×1 convolution and upsampling, is concatenated along the channel dimension with the results of the three gated dilated convolutions and the 1×1 convolution, and then further processed by convolution to form the output feature map. Each gated dilated convolution contains two parallel branches: one for feature transformation and the other for generating spatial gating weights. A dilation rate of 1 corresponds to a smaller receptive field, used to extract minute highlight details; a dilation rate of 4 corresponds to a medium-scale receptive field, used to extract local highlight region information; and a dilation rate of 8 corresponds to a larger receptive field, used to extract large highlight regions and surrounding spatial information. The three dilation rates are fixed at 1, 4, and 8, and do not change with training samples. All three gated dilated convolutions use 3×3 dilated kernels, and edge padding is used to maintain consistent spatial dimensions of the output feature map. Each gated dilated convolution's feature transformation branch uses a 3×3 dilated convolution with the corresponding dilation rate to transform the input feature map, then performs a non-linear mapping using a linear rectified activation function to obtain the feature transformation result. The spatial gate weight generation branch uses a 3×3 dilated convolution with the corresponding dilation rate to generate a gated response, then uses a sigmoid activation function to restrict the gated response to the range of 0 to 1, obtaining the spatial gate weights. The spatial gate weights and the feature transformation result are multiplied element-wise according to the same spatial location and the same feature channel to obtain the gated dilated convolution processing result. The spatial gate weight generation branch uses an indexed... The dilated convolution kernel and bias term, and the feature transformation branch use subscripted... The dilated convolution kernel and bias term, the result of gated dilated convolution is calculated according to the following formula: In the formula, Indicates the void ratio The output of the gated dilated convolution processing result; Take 1, 4, and 8 in sequence; Indicates the input feature map; subscript Indicates the spatial gating weight generation branch; subscript Indicates the feature transformation branch; This indicates that the spatial gating weight generation branch is based on the void ratio. The dilated convolution kernel used below; The feature transformation branch represents the porosity. The dilated convolution kernel used below; The bias term represents the branch used to generate the spatial gating weights; The bias term represents the feature transformation branch; Indicates the void ratio dilated convolution operation; ReLU represents the Sigmoid activation function; ReLU represents the rectified linear activation function. This represents element-wise multiplication. Spatial gating weights close to 1 preserve the feature response at the corresponding position, while spatial gating weights close to 0 suppress the feature response at the corresponding position. The input feature map simultaneously undergoes 1×1 convolution and global pooling. The 1×1 convolution transforms the feature channels and preserves local information in the input feature map; global pooling aggregates global contextual information from the input feature map. The global pooling result is then processed by 1×1 convolution and upsampling to restore the spatial dimensions to the same level as the results of the three gated dilated convolutions. The results of the three gated dilated convolutions, along with the convolution and global pooling results, are concatenated through channels and then convolved to obtain the final output. Gated dilated spatial pyramid pooling, leveraging multi-scale gated dilated convolutions, more accurately captures multi-scale information from minute highlight details to large highlight areas, while dynamically filtering and enhancing highlight features using a gating mechanism to filter out background interference. In practice, the gated dilation convolution results, 1×1 convolution results, and global pooling results after upsampling are concatenated along the feature channel dimension for dilation rates of 1, 4, and 8. Then, a 1×1 convolution is used to fuse the feature channels, forming the gated dilation spatial pyramid pooling result. This result is input into the subsequent decoding process of the U-shaped encoder-decoder network. Through upsampling and same-scale feature connections, the spatial location, edge, and texture information of the highlight region are gradually recovered. Finally, the decoded features are converted into a single-channel training highlight mask through output convolution. The training highlight mask maintains a 256×256 pixel spatial correspondence with the original training highlight image. Each pixel value in the training highlight mask is between 0 and 1, representing the confidence that the corresponding location belongs to the highlight region. The training highlight mask is used for subsequent mask-guided gated feature fusion processing and participates in the mask segmentation loss calculation together with the manually labeled binary ground truth mask. Complete calculation process: First, read a 256×256 pixel original specular image for training. Input the original specular image into the U-shaped encoder-decoder network, and obtain the input feature map through encoding feature extraction. The second step is to input the feature map. Three gated dilated convolutions with dilation rates of 1, 4, and 8 are input respectively. Each gated dilated convolution simultaneously performs feature transformation and spatial gating weight generation, and then performs element-wise multiplication according to the formula for calculating the gated dilated convolution processing result, yielding the three gated dilated convolution processing results. The third step involves processing the input feature map... The input is processed by a 1×1 convolution to obtain the convolution result; simultaneously, the input feature map is processed... The input undergoes global pooling, followed by 1×1 convolution and upsampling to obtain the global pooling result. In the fourth step, the results of the three gated dilated convolutions, the convolutional processing, and the global pooling are concatenated along the feature channel dimension, and then fused using a 1×1 convolution to obtain the gated dilated spatial pyramid pooling result. In the fifth step, the gated dilated spatial pyramid pooling result is input into the subsequent decoding process of the U-shaped encoder-decoder network. Upsampling and same-scale feature connections restore the image spatial resolution, and output convolution generates a single-channel training spectrophoto mask. In the sixth step, the training spectrophoto mask is passed to subsequent masks to guide gated feature fusion processing. During network training, manually labeled binary ground truth masks are used to calculate the mask segmentation loss, completing the generation of the training spectrophoto mask.
[0080] The present invention is further configured such that the mask-guided gating feature fusion processing includes:
[0081] Mask-guided gating feature fusion processing is set before the downsampling stage of each encoder in the encoding / decoding network;
[0082] The training of a single channel is convolved using a specular mask to obtain mask features with the same dimension as the input feature channels in the encoding stage.
[0083] The input features in the encoding stage are subjected to two consecutive convolutions, batch normalization, and activation processing to obtain enhanced features;
[0084] The mask features and enhancement features are concatenated along the channel dimension to obtain joint features. The joint features are then subjected to gated convolution and connected with the enhancement features through residuals to obtain fused features.
[0085] Encoding and decoding are performed based on fused features to output a predicted specular reflection image for training.
[0086] Based on the two-color reflectance model, the original specular image used for training is subtracted from the predicted specular reflection image used for training to obtain the coarse diffuse reflection image used for training. Specifically, the coarse specular removal process employs a U-shaped encoder-decoder network, with a mask-guided gating feature fusion process performed before downsampling by each encoder. The specific structure of the mask-guided gating feature fusion process is as follows... Figure 4As shown, the training specular mask is processed by 1×1 convolution to form mask features. The input features in the encoding stage undergo two consecutive 3×3 convolutions, batch normalization, and linear rectified activation to form enhanced features. The mask features and enhanced features are concatenated along the channel dimension and then processed by 1×1 gated convolution. The result of the 1×1 gated convolution is then added to the enhanced features using residuals to obtain the fused features. The training specular mask serves as prior information for the specular region, enhancing the feature response corresponding to the specular region and suppressing interference from non-spectral backgrounds. Residual connections are used to preserve the basic structural, color, and texture information in the enhanced features, reducing the possibility of losing effective information due to gating. The input features in any encoding stage are denoted as... The training specular mask is denoted as The training specular mask is a single-channel probability map, with each pixel value ranging from 0 to 1. A pixel value closer to 1 indicates a higher confidence level that the corresponding location belongs to a specular region; a pixel value closer to 0 indicates a higher confidence level that the corresponding location belongs to a non-spectral region. Before entering the mask-guided gating feature fusion process, the training specular mask maintains the same spatial dimension as the input features in the encoding stage. First, a convolutional layer maps the training specular mask to the same channel dimension as the input features in the encoding stage, generating a mask feature representation. The calculation formula is as follows: In the formula, Indicates mask features; Indicates the specular mask used for training; This indicates a 1×1 convolutional process. A 1×1 convolution maps single-channel specular probability information to multiple feature channels, enabling element-wise spatial interaction between the mask features and the input features in the encoding stage. Subsequently, the input features in the encoding stage undergo two consecutive convolution-batch normalization-activation processes to obtain enhanced features, calculated using the following formula: In the formula, Indicates enhanced features; This represents the input features during the encoding stage; This indicates that convolution, batch normalization, and linear rectified activation are performed sequentially. Convolution is used to extract local spatial features, batch normalization is used to adjust the distribution of batch features, and linear rectified activation is used to complete nonlinear feature mapping. Two consecutive convolution-batch normalization-activation processes are used to enhance feature representation and alleviate gradient vanishing during network training. The enhanced features and mask features are concatenated along the channel dimension to form joint features; the joint features are then processed by gated convolution and joined with the enhanced features through a residual connection to output fused features. The calculation formula is as follows: In the formula, Indicates fusion characteristics; Indicates enhanced features; Indicates mask features; This indicates splicing along the channel dimension; This indicates a gated convolution process with a kernel size of 1×1. The gated convolution process adjusts the response intensity of different spatial locations and feature channels based on joint features, enhancing features related to highlight regions and suppressing non-highlight background features. Residual connections directly add the enhanced features to the gated convolution result to maintain the integrity of the image's basic information. After completing the mask-guided gated feature fusion process in the current encoding stage, the fused features are input into the downsampling process of the current encoder. Each encoding stage repeatedly executes mask convolution mapping, encoded feature enhancement, channel concatenation, gated convolution, and residual connections, ensuring that encoded features at different spatial scales are guided by the training highlight mask. After completing all encoding stages, decoding is performed based on the fused features formed in each encoding stage. The decoding process uses upsampling and same-scale encoded feature connections to gradually restore the image's spatial resolution, ultimately generating the training predicted specular reflection image through output convolution. The training predicted specular reflection image is denoted as... This is used to characterize the specular reflection component predicted from the original specular image used for training. After encoding and decoding based on mask-guided gating feature fusion, the predicted specular reflection image used for training is output, and physical consistency reconstruction is performed based on the two-color reflection model. The calculation formula is as follows: In the formula, This represents a coarse diffuse image used for training; This represents the original specular image used for training; This represents the training predicted specular reflection image. The original training specular image and the training predicted specular reflection image have the same spatial dimensions and number of color channels. Subtraction is performed at the same pixel locations and with the same color channels to separate the predicted specular reflection component from the original training specular image, generating a preliminary training coarse diffuse reflection image with specular highlights removed. The training predicted specular reflection image is used for subsequent specular reconstruction loss calculation, and the training coarse diffuse reflection image is used for subsequent multipath gating thinning processing. Complete calculation process: Read the input features of the current encoding stage and the specular mask for training. Adjust the specular mask for training to the same spatial size as the input features of the encoding stage. Map the specular mask for training to a mask feature with the same channel dimension as the input features of the encoding stage through 1×1 convolution. Perform two consecutive convolution-batch normalization-linear rectified activation processes on the input features of the encoding stage to obtain enhanced features. Concatenate the enhanced features and the mask features along the channel dimension to form joint features. Perform 1×1 gated convolution on the joint features and add the gated convolution result to the enhanced features through residual connection to obtain fused features. Input the fused features into the downsampling process of the current encoder and repeat the above process in subsequent encoding stages. After completing all encoding stages, perform upsampling and decoding based on the fused features formed in each encoding stage to output the predicted specular reflection image for training. Finally, according to the two-color reflection model, subtract the predicted specular reflection image for training from the original specular image for training to obtain the coarse diffuse reflection image for training.
[0087] The present invention is further configured such that multi-path parallel feature processing includes:
[0088] The original specular image used for training is convolved to perform channel dimension alignment and feature compression, resulting in the first path of features;
[0089] The training coarse diffuse image is subjected to three consecutive gated convolutions and activations to obtain the second-path features;
[0090] The training image of the predicted specular reflection is preprocessed by convolution to unify the channel dimensions and enhance cross-channel correlation, resulting in a third-path feature.
[0091] The first and second features are concatenated along the channel dimension and then convolved to obtain the intermediate fused features.
[0092] The intermediate fusion features and the third-path features are concatenated along the channel dimension and then convolved to obtain the refined diffuse reflection image for training. Specifically, multi-path gated refinement is used to further refine the rough diffuse reflection image for training generated in the rough specular removal stage. Combining the complementary information between the original specular image for training, the rough diffuse reflection image for training, and the predicted specular reflection image for training, the structure and texture information obscured by specular highlights is restored through three-path parallel feature processing and two-level hierarchical feature fusion, while suppressing residual specular components and image artifacts in the rough diffuse reflection image for training. The original specular image for training, the rough diffuse reflection image for training, and the predicted specular reflection image for training have the same image height, image width, and number of color channels. The rough diffuse reflection image for training is obtained by subtracting the predicted specular reflection image for training from the original specular image for training. The first path takes the original specular image for training as input and is used to preserve the spatial structure and intensity distribution features outside the specular region. The original specular image for training is denoted as... The first path uses a 1×1 convolutional layer to perform channel dimension alignment and feature compression on the original specular image used for training, reducing redundant information in different color channels and feature channels to obtain the first path features. The first path features are calculated according to the following formula: In the formula, Indicates the first-path feature; This represents the original specular image used for training; This indicates a 1×1 convolutional processing step. The 1×1 convolution does not change the spatial dimensions of the original specular image used for training; instead, it maps the original specular image to a preset number of feature channels using trainable convolution parameters. The preset number of feature channels matches the channel configuration required when the second-path features participate in the first channel stitching. The specific value is set according to the number of feature channels in the encoding / decoding network and remains unchanged during the same model training cycle. The second path takes the training coarse diffuse image as input and is used to refine the image content that has already undergone preliminary specular separation. The training coarse diffuse image is denoted as... The second path sequentially performs three gated convolutions and linear rectified activation processes. Each gated convolution uses a 3×3 kernel, and edge padding is used to maintain the spatial size of the second path features. The gated convolution dynamically adjusts the activation state of different spatial locations and feature channels based on the feature content in the training coarse diffuse image, suppressing noise propagation in non-target areas and enhancing the feature responses of highlight edges, object edges, and textured regions. A single gated convolution and linear rectified activation process is expressed as follows: In the formula, This represents a single gated convolution and linear rectified activation process; This represents a gated convolution with a kernel size of 3×3; This indicates gated convolution processing; The symbol represents the linear rectified activation function; the dot represents the input feature. The trainable parameters in the gated convolution are updated via backpropagation through the subsequent multi-task combined loss function, without a manually fixed gating threshold. The second-path features are calculated according to the following formula: In the formula, Indicates the second-path feature; This represents a coarse diffuse image used for training; This indicates that three consecutive gated convolutions and linear rectified activations (LCEs) are performed. The first gated convolution and LCE is used to extract preliminary local features from the training coarse diffuse image. The second gated convolution and LCE is used to expand the correlation between local features. The third gated convolution and LCE is used to further enhance edge and texture responses, resulting in the second set of features. The third set of features takes the training predicted specular reflection image as input and is used to perform independent feature processing on the separated specular reflection components. The training predicted specular reflection image is denoted as... The third path uses a 1×1 convolutional layer for feature preprocessing. Channel transformation unifies the channel dimensions of the third path features and enhances the correlation between different color channels and feature channels in the predicted specular reflection image used for training. The third path features are calculated according to the following formula: In the formula, Indicates the third-way feature; This represents the predicted specular reflection image used for training; This indicates a 1×1 convolutional kernel. The third feature path provides information on the location, intensity, and spatial distribution of specular reflection components, enabling subsequent layered feature fusion to identify specular remnant regions in the training coarse diffuse image. After completing the three-path parallel feature processing, the first and second features are concatenated along the channel dimension. Channel concatenation preserves the spatial coordinates of the first and second features, combining the original spatial structure retained in the first feature path with the coarse diffuse refinement information extracted from the second feature path. The concatenated result undergoes a 3×3 convolution, utilizing spatial dependencies within the local neighborhood to generate intermediate fusion features, creating a smooth transition between the edges of specular and non-spectral regions and improving the expressive power of local structures. The intermediate fusion features are then concatenated along the channel dimension with the third feature path, allowing the specular reflection component's feature information to participate in the final refinement. The second concatenation result undergoes a 3×3 convolution, completing nonlinear feature transformation and cross-path information fusion, suppressing specular remnants and artifacts in the training coarse diffuse image, resulting in a refined diffuse image for training. The refined diffuse reflection image used for training is calculated according to the following formula: In the formula, Represents the training using a refined diffuse image; Indicates the first-path feature; Indicates the second-path feature; Indicates the third-way feature; This indicates splicing along the channel dimension; This indicates a 3×3 convolutional kernel. Inner channel concatenation and 3×3 convolution are used to fuse the first and second feature paths, forming an intermediate fused feature. Outer channel concatenation and 3×3 convolution are used to fuse the intermediate fused feature and the third feature path, outputting a refined diffuse image for training. The refined diffuse image for training maintains the same spatial dimensions and number of color channels as the original specular image for training. During training, the refined diffuse image for training is compared with the original ground truth image without specular highlights to calculate the image restoration loss. During inference, the corresponding refined diffuse image is output as the reconstructed image after specular removal. Through three parallel feature processing paths, the first path preserves the original spatial structure and intensity distribution, the second path refines the coarse specular removal result, and the third path provides specular reflection component information. Through two-level hierarchical feature fusion, structure preservation, edge smoothing, texture enhancement, and specular residue suppression are gradually achieved. Complete calculation process: First step, read the original specular image for training. Training coarse diffuse reflection image and training images for predicting specular reflections First, confirm that the three types of images have the same spatial dimensions and number of color channels. Second, use the original specular images for training. Input a 1×1 convolutional layer to perform channel dimension alignment and feature compression, and obtain the first-path features according to the first-path feature calculation formula. The third step is to use coarse diffuse images for training. Three 3×3 gated convolutional layers are input sequentially. After the output of each 3×3 gated convolutional layer, linear rectified activation is performed. This process is repeated three times according to the formula for a single gated convolution and linear rectified activation, resulting in the second feature path. The fourth step is to use the training image to predict the specular reflection. Input a 1×1 convolutional layer to unify channel dimensions and enhance cross-channel features, and obtain the third-path features according to the third-path feature calculation formula. The fifth step is to incorporate the first feature... Second-way features The concatenation is performed along the channel dimension, and a 3×3 convolution is applied to the concatenated result to obtain the intermediate fused features. The sixth step involves combining the intermediate fused features with the third-path features. The image is stitched along the channel dimension, and a 3×3 convolution is performed on the stitched result. The refined diffuse reflectance image for training is then obtained according to the formula for calculating the refined diffuse reflectance image for training. The seventh step involves passing the refined diffuse image used for training to the inpainting image loss calculation process. The structure, texture, and color restoration of the refined diffuse image used for training are supervised by the original ground truth image without highlights, thus completing the multi-path gating refinement process.
[0093] The present invention is further configured such that the multi-task combination loss function includes: mask segmentation loss, specular reconstruction loss, and image restoration loss;
[0094] Based on the training specular mask and the manually labeled binary ground truth mask, the binary cross-entropy loss and Dice loss are calculated and combined to obtain the mask segmentation loss;
[0095] Based on the predicted specular reflection image used for training, the real specular reflection image, and the specular mask used for training, the mask-weighted mean square error loss in the specular region and the perceptual loss based on the pre-trained VGG-16 network are calculated and combined to obtain the specular reconstruction loss.
[0096] Based on the refined diffuse reflection image used for training and the original ground truth image without highlights, the perceptual loss and the style loss based on the feature map Gram matrix are calculated and combined to obtain the inpainted image loss.
[0097] A weighted combination of the mask segmentation loss, specular reconstruction loss, and image restoration loss yields a multi-task combined loss function. Specifically, this function is used to simultaneously optimize specular region detection, specular reflection component reconstruction, and specular inpainting. Each training forward propagation outputs a training specular mask, a training predicted specular reflection image, and a training refined diffuse reflection image, which are then compared with a manually labeled binary ground truth mask, a real specular reflection image, and the original specular inpainting ground truth image, respectively. The multi-task combined loss function, composed of the mask segmentation loss, specular reconstruction loss, and image restoration loss, is calculated using the following formula: In the formula, This represents the value of the multi-task combination loss function; Indicates the mask segmentation loss; Indicates loss in highlight reconstruction; Indicates the restoration of image loss; , and These represent the weighted balancing parameters corresponding to the three types of losses, used to balance the gradient magnitudes of the three types of losses and avoid a single training task dominating network parameter updates. To determine the weights of the mask segmentation loss, spectrophotometry reconstruction loss, and image restoration loss, a single-parameter sensitivity analysis was performed using a controlled variable method. Only one loss weight was changed at a time, while the other two loss weights were fixed at 1. Peak signal-to-noise ratio (PSNR) was used as the evaluation metric. The sensitivity analysis results show that the parameters... When the value is 1, the peak signal-to-noise ratio reaches its maximum value within the current scanning range; parameter When the value is 2, the peak signal-to-noise ratio reaches its maximum value within the current scanning range; parameters When the value is 1, the peak signal-to-noise ratio reaches its maximum value within the current scan range. Therefore, the candidate parameter combination obtained from the single-parameter sensitivity analysis is: Set to 1, Set to 2, Setting the weighted average to 1 ensures that the three weighted balancing parameters remain unchanged throughout a single training cycle. To verify the stability of the candidate parameter combination when the highlight reconstruction loss weights are set to 2, in... Under the condition of =2, the parameters were further analyzed through sensitivity analysis. and parameters The analysis results regarding the impact on peak signal-to-noise ratio are consistent with the aforementioned candidate parameter combinations. In this embodiment, the mask segmentation loss weights are applied. Setting it to 1 will adjust the weight of the specular reconstruction loss. Setting it to 2 will restore the weights for image loss. Set to 1. Mask segmentation loss: The specular mask used for training is denoted as... The manually labeled binary true mask is denoted as In the training specular mask, each pixel value represents the confidence level that the corresponding location belongs to a specular region. In the manually labeled binary ground truth mask, a pixel value of 1 represents a specular region, and a pixel value of 0 represents a non-spectral region. The height of the input image is denoted as... The width of the input image is denoted as The masking segmentation loss uses a hybrid loss function that combines the Dice loss and the binary cross-entropy loss. The calculation formula is as follows: The formula for calculating binary cross-entropy loss is: In the formula, Indicates the pixel number. Iterate through all pixels in the image; This represents the manually labeled pixel values of the actual binary mask. This represents the pixel values of the specular mask used for training. The natural logarithm operation is represented; the binary cross-entropy loss is used to constrain the classification results of highlight regions pixel by pixel. The Dice loss calculation formula is: In the formula, The minimum smoothing constant is used to prevent numerical overflow when the sum of the denominators is 0. In this embodiment, the minimum smoothing constant is fixed at 10 to the power of negative 5 and is not adjusted during the entire training process. The Dice loss is used to constrain the overlap between the predicted and actual specular regions and to alleviate training bias caused by the imbalance between the number of specular and non-spectral pixels. Specular reconstruction loss: The predicted specular reflection image used for training is denoted as... The true specular reflection image is denoted as The training images used for predicted specular reflections maintain the same spatial dimensions and number of color channels as the real specular reflection images. The specular reconstruction loss consists of mask-weighted mean square error loss and perceptual loss, calculated using the following formula: The mask-weighted mean square error loss is calculated only within the highlight region, and the calculation formula is as follows: In the formula, This represents element-wise multiplication between pixels with the same pixel position and the same color channel. This represents the pixel-wise L2 norm squared. The training specular mask is multiplied by both the training predicted specular reflection image and the real specular reflection image, making the mask-weighted mean square error loss primarily reflect the reconstruction bias of the specular reflection components within the specular region. The perceptual loss uses a pre-trained visual geometry group of a 16-layer convolutional neural network to extract high-frequency texture and structural features, calculated using the following formula: In the formula, Indicates the convolutional layer number involved in the calculation of perceptual loss; This represents the 16th layer of a pre-trained visual geometry group convolutional neural network. Feature maps output by each convolutional layer; Indicates the first The layer weights corresponding to each convolutional layer; the convolutional layers and layer weights involved in the perceptual loss calculation are fixed before training begins and remain unchanged throughout the entire training process; the pre-trained visual geometry group of sixteen convolutional neural networks is only used for feature extraction and does not participate in the parameter updates of the specular detection and removal network. Image restoration loss: the refined diffuse reflection image used for training is denoted as... The original ground truth image without highlights is denoted as The training image, refined diffuse, maintains the same spatial dimensions and number of color channels as the original ground truth image without highlights. The inpainting loss is a linear combination of perceptual and style losses, calculated as follows: In the formula, This represents the perceptual loss between the refined diffuse image used for training and the original ground truth image without highlights. Indicates style loss; This represents the style loss balancing weight, used to adjust the smoothness of the texture. In this embodiment, the style loss balancing weight is fixed at 0.1 by default. The style loss is calculated based on the Gram matrix, and the calculation formula is as follows: In the formula, This represents the 16th layer of a pre-trained visual geometry group convolutional neural network. Feature maps output by each convolutional layer; Indicates based on the first The Gram matrix is calculated from the feature maps of each convolutional layer; the Gram matrix is used to characterize the correlation between different feature channels and the image texture style. The squared Frobenius norm is used to measure the global difference between two sets of Gram matrices. Perceptual loss is used to maintain structural and semantic consistency, while style loss is used to maintain local texture consistency and reduce over-smoothing. After calculating the three types of losses, the multi-task combined loss function is calculated according to the weighted balancing parameters 1, 2, and 1. Then, the gradients corresponding to the trainable parameters in the specular detection and removal network are calculated through backpropagation, and the Adam optimizer is used to update the parameters of the specular detection and removal network. The training process is performed based on the PyTorch framework. The input images are uniformly adjusted to 256×256 pixels, and the initial learning rate is set to 10 to the power of negative 4. Every 10 training epochs, the current learning rate is divided by 10. The parameters of the pre-trained visual geometry group's sixteen-layer convolutional neural network are kept frozen. The complete calculation process is as follows: First, the training specular mask, manually annotated binary ground truth mask, training predicted specular reflection image, ground truth specular reflection image, training refined diffuse reflection image, and original ground truth image without specular reflection are read from the same training sample. The second step involves calculating the binary cross-entropy loss and Dice loss based on the training specular mask and the manually labeled binary ground truth mask. These losses are then summed to obtain the mask segmentation loss. The third step involves weighting the specular regions of the training predicted specular image and the ground truth specular image using the training specular mask, calculating the mask-weighted mean square error loss. The training predicted specular image and the ground truth specular image are then input into a pre-trained 16-layer convolutional neural network with a visual geometry group to extract preset convolutional layer feature maps and calculate the perceptual loss. The mask-weighted mean square error loss and the perceptual loss are summed to obtain the specular reconstruction loss. The fourth step involves inputting the training refined diffuse image and the original ground truth image without specular highlights into the pre-trained 16-layer convolutional neural network with a visual geometry group to calculate the perceptual loss. The Gram matrix is calculated based on the same convolutional layer feature maps, and the Frobenius norm squared between the two Gram matrices is calculated to obtain the style loss. The perceptual loss is then summed with the style loss multiplied by 0.1 to obtain the restored image loss. Step 5: Multiply the mask segmentation loss by 1, the specular reconstruction loss by 2, and the image restoration loss by 1, then sum the three weighted results to obtain the multi-task combined loss function value. Step 6: Perform backpropagation based on the multi-task combined loss function value, using the Adam optimizer to update the trainable parameters in the specular detection and removal network, and adjust the learning rate according to the rule of reducing it to one-tenth of the current learning rate every 10 training epochs. Step 7: Repeat forward propagation, three-class loss calculation, multi-task combined loss function value calculation, and parameter update for the training samples in the training dataset, completing the preset training epochs to obtain the trained specular detection and removal network.
[0098] The original highlight image to be processed is input into a trained highlight detection and removal network. Highlight detection, coarse highlight removal, and fine highlight removal are performed sequentially to obtain a reconstructed image after highlight removal. The complete calculation process includes: First, loading the parameters of the trained highlight detection and removal network, setting the network to inference mode, and adjusting the original highlight image to 256×256 pixels. Second, inputting the original highlight image into a U-shaped encoder-decoder network to extract the input feature map from the gated dilated spatial pyramid pooling process. Third, inputting the feature map into three gated dilated convolutions with dilation rates of 1, 4, and 8, a 1×1 convolution, and a global pooling process, respectively. The five processing results are concatenated along the channel dimension and fused by convolution, then a highlight mask is generated through the decoding process. The fourth step involves inputting the original specular image and specular mask into the coarse specular removal encoding / decoding network. Before downsampling at each encoder stage, mask convolution mapping, input feature enhancement during the encoding stage, channel concatenation, gated convolution, and residual connections are performed to obtain the fused features for each encoding stage. The fifth step involves upsampling and decoding based on the fused features from each encoding stage to generate the predicted specular reflection image. The sixth step involves subtracting the predicted specular reflection image from the original specular image according to the two-color reflectance model to obtain the coarse diffuse reflection image. The seventh step involves inputting the original specular image into the first 1×1 convolutional processing to obtain the first feature path; inputting the coarse diffuse reflection image into three consecutive gated convolutions and linear rectified activation processing to obtain the second feature path; and inputting the predicted specular reflection image into the third 1×1 convolutional processing to obtain the third feature path. Step 8: Concatenate the first and second feature paths along the channel dimension and perform a 3×3 convolution to obtain the intermediate fused feature; concatenate the intermediate fused feature and the third feature path along the channel dimension and perform a 3×3 convolution to obtain the refined diffuse reflection image. Step 9: Output the refined diffuse reflection image as the reconstructed image after specular removal. The inference stage does not require manual annotation of the binary ground truth mask, the true specular reflection image, the original specular-free ground truth image, or the multi-task combined loss function.
[0099] The present invention is further configured such that S7 includes:
[0100] Pixel-level reconstruction error, peak signal-to-noise ratio, and structural similarity index are calculated based on the reconstructed image after removing highlights and the original ground truth image without highlights.
[0101] When the peak signal-to-noise ratio is greater than the preset first signal-to-noise threshold and the structural similarity index is greater than the preset first similarity threshold, the highlight removal is deemed complete and the repair effect is deemed satisfactory.
[0102] When the peak signal-to-noise ratio is greater than the preset second signal-to-noise threshold and less than the preset first signal-to-noise threshold, and the structural similarity index is greater than the preset second similarity threshold and less than the preset first similarity threshold, it is determined that the highlights have been basically removed.
[0103] When the peak signal-to-noise ratio (PSNR) is less than a preset second SNR threshold or the structural similarity index is less than a preset second similarity threshold, the highlight removal effect is deemed poor and the restoration quality is deemed unqualified. Specifically, the reconstructed image after highlight removal and the original ground truth image without highlights corresponding to the same validation sample are read from the validation dataset. The reconstructed image after highlight removal is output by the trained highlight detection and removal network, and the original ground truth image without highlights is a reference image that does not contain specular highlights. Before calculation, the image size, pixel coordinates, color channel order, and pixel dynamic range of the two images are checked. Both images are uniformly 256×256 pixel three-channel color images, and the pixel values are represented by an eight-bit image from 0 to 255. When the size or pixel position is inconsistent, the two images are processed synchronously according to the size adjustment method used in the training dataset to ensure that the same pixel position has a direct comparison relationship. The pixel-level reconstruction error is used to measure the pixel deviation between the reconstructed image after highlight removal and the original ground truth image without highlights. The smaller the pixel-level reconstruction error, the closer the reconstructed image after highlight removal is to the original ground truth image without highlights. The pixel-level reconstruction error is calculated according to the following formula: In the formula, This indicates pixel-level reconstruction error; This represents the original ground truth image without highlights at the 1st... row and number The pixel value at the column position; This indicates the reconstructed image after specular removal at the [number]th [time]. row and number The pixel value at the column position; Indicates the image height; Indicates the image width; Indicates the pixel row number; This represents the pixel column number. During calculation, the pixel difference at the same pixel position is first calculated, then the pixel difference is squared, and finally the average of the squared differences corresponding to all pixel positions is taken. For a three-channel color image, the pixel-level reconstruction error of the red, green, and blue channels is calculated separately, and then the average of the three calculation results is taken to obtain the pixel-level reconstruction error corresponding to the entire image. Peak signal-to-noise ratio (PSNR) is used to evaluate the degree of pixel distortion of the reconstructed image after highlight removal relative to the original ground truth image without highlights. The higher the PSNR, the lower the pixel distortion. The PSNR is calculated according to the following formula: In the formula, This indicates the peak signal-to-noise ratio, expressed in decibels. The peak amplitude represents the dynamic range of an image pixel. This indicates pixel-level reconstruction error; This represents a base-10 logarithmic operation. This embodiment uses an 8-bit image representation range of 0 to 255, representing the peak amplitude of the image pixel dynamic range. The value is fixed at 255. When the pixel-level reconstruction error is 0, the two images are completely identical at all pixel positions, the peak signal-to-noise ratio is recorded as positive infinity, and the program no longer performs division by zero. The structural similarity index evaluates the visual similarity between the reconstructed image after removing highlights and the original ground truth image without highlights in terms of brightness, contrast, and structure. The closer the structural similarity index is to 1, the better the recovery of brightness, texture, and spatial structure. The structural similarity index corresponding to a local window is calculated according to the following formula: In the formula, This represents the structural similarity index corresponding to a local window; This represents the pixel block within a local window of the original ground truth image without highlights; This represents a pixel block within the same local window in the reconstructed image after highlight removal; Represents pixel blocks The local mean; Represents pixel blocks The local mean; Represents pixel blocks Local variance; Represents pixel blocks Local variance; This represents the local covariance between two pixel blocks; and This represents a stability constant to prevent the denominator from being zero. The structural similarity index is calculated using an existing structural similarity evaluation algorithm. The local window is set to a Gaussian window of 11×11 pixels by default, and the Gaussian standard deviation is set to 1.5 by default. For a pixel dynamic range of 0 to 255, the stability constant... The default value is 6.5025, the stability constant. The default value is 58.5225. Local windows traverse the two images with a one-pixel movement step, and the average of the structural similarity indices corresponding to all local windows is taken. For the three-channel color image, the structural similarity indices for the red, green, and blue channels are calculated separately, and the average of the three results is taken to obtain the structural similarity index for the entire image. The default preset signal-to-noise threshold is 30 dB, the default preset second signal-to-noise threshold is 25 dB, the default preset first similarity threshold is 0.95, and the default preset second similarity threshold is 0.90. When the peak signal-to-noise ratio is greater than 30 dB and the structural similarity index is greater than 0.95, the highlight removal is considered complete and the restoration effect is acceptable; when the peak signal-to-noise ratio is greater than 25 dB and less than 30 dB, and the structural similarity index is greater than 0.90 and less than 0.95, the highlight removal is considered basically complete; when the peak signal-to-noise ratio is less than 25 dB or the structural similarity index is less than 0.90, the highlight removal effect is considered poor and the restoration quality is unacceptable. When the peak signal-to-noise ratio (PSNR) and structural similarity index fall within different judgment ranges, a stricter judgment rule is adopted, selecting the judgment result with lower restoration quality among the two indicators. When the PSNR reaches 30 dB or the structural similarity index reaches 0.95, it is classified as basically removed highlights; when the PSNR reaches 25 dB or the structural similarity index reaches 0.90, it is classified as having poor highlight removal effect and unqualified restoration quality. Pixel-level reconstruction error is used as an auxiliary evaluation parameter and does not participate in the restoration quality judgment alone. Pixel-level reconstruction error, along with PSNR, structural similarity index, and restoration quality judgment results, forms the highlight removal performance evaluation data. Complete calculation process: First, read the reconstructed image after highlight removal and the original ground truth image without highlights corresponding to the same verification sample, unifying the two images to 256×256 pixels, three-channel color format, and a pixel dynamic range of 0 to 255. Second, calculate the pixel difference between the two images according to the same pixel position and the same color channel, square the pixel difference, and take the average of all squared differences to obtain the pixel-level reconstruction error. The third step is to set the peak amplitude of the image pixel dynamic range to 255, and substitute the pixel-level reconstruction error into the peak signal-to-noise ratio (PSNR) calculation formula to obtain the PSNR. When the pixel-level reconstruction error is 0, the PSNR is recorded as positive infinity. The fourth step is to use an 11×11 pixel Gaussian window to traverse both images, calculate the local mean, local variance, and local covariance within each local window, and then substitute them into the structural similarity index calculation formula to obtain the structural similarity index corresponding to each local window. The fifth step is to average the structural similarity indices corresponding to all local windows, and then average the calculation results corresponding to the red, green, and blue channels to obtain the structural similarity index corresponding to the entire image. The sixth step is to compare the PSNR with 30 dB and 25 dB, and compare the structural similarity index with 0.95 and 0.90.Step 7: When the peak signal-to-noise ratio (PSNR) is greater than 30 dB and the structural similarity index is greater than 0.95, the output highlights are thoroughly removed and the restoration effect is acceptable; when the PSNR is greater than 25 dB and less than 30 dB, and the structural similarity index is greater than 0.90 and less than 0.95, the output highlights are basically removed; when the PSNR is less than 25 dB or the structural similarity index is less than 0.90, the output highlights removal effect is poor and the restoration quality is unacceptable; when the two indicators fall into different judgment ranges, the output restoration quality is judged as low. Step 8: Summarize the pixel-level reconstruction error, PSNR, structural similarity index, and restoration quality judgment results to form highlight removal performance evaluation data. To further verify the highlight removal effect of the trained highlight detection and removal network on industrial object images, 183 industrial component images from the HLR dataset were used as test images and input into the trained highlight detection and removal network to obtain the corresponding reconstructed images after highlight removal. The reconstructed images after highlight removal were compared with the corresponding original ground truth images without highlights. The highlight removal comparison results on industrial objects are shown below. Figure 6 As shown. Figure 6 The comparison results are used to compare the specular retention, object structure preservation, and texture restoration status in different processing results. Figure 6 It can be seen that the trained specular highlight detection and removal network can remove specular highlights from industrial part images, and make the reconstructed image after highlight removal close to the corresponding original ground truth image without highlights in terms of subject contours, edges, and textures. Combined with... Figure 5 and Figure 6 The experimental results show that the trained specular highlight detection and removal network can remove specular highlights from the test images and industrial part images in the dataset, while preserving the main structure, edges, and texture information during the highlight removal process, thus verifying the actual highlight removal effect of the network. To verify the highlight removal effect of the trained network on industrial object images, 183 industrial part images from the HLR dataset were used as test images. These images were input into the trained network without retraining or adjusting the network parameters, resulting in reconstructed images after highlight removal. These reconstructed images were then visually compared with the results shown in the figure. The comparison results of highlight removal on industrial objects are as follows: Figure 6 As shown. Figure 6The test results were used to compare the specular retention, object structure preservation, and texture restoration states of different processing results. The results show that the trained specular detection and removal network can maintain its specular removal capability in industrial component scenes, indicating that the specular features learned by the network are not only applicable to fixed objects and fixed imaging conditions in the training data. Further generalization ability tests were conducted using 648 real glass images from the Glass dataset. The Glass dataset was not used in the training and parameter updates of the specular detection and removal network; the real glass specular images were directly input into the trained specular detection and removal network to obtain reconstructed images after specular removal. Further generalization ability tests were conducted using 55 real camera circle images from the Aperture dataset. The Aperture dataset was not used in the training and parameter updates of the specular detection and removal network; the real camera lens specular images were directly input into the trained specular detection and removal network to obtain reconstructed images after specular removal.
[0104] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.
Claims
1. A method for detecting and removing highlights, characterized in that, include: S1: Obtain the original specular image for training, as well as the manually labeled binary ground truth mask, real specular reflection image, and original ground truth image without specular reflection required for network training, and construct the training dataset; S2: Input the original specular image for training into the U-Net network based on gated void spatial pyramid pooling, extract multi-scale features, and output the specular mask for training. S3: Input the original specular image and specular mask for training into the encoding and decoding network, perform mask-guided gating feature fusion processing under the guidance of the specular mask for training, output the predicted specular reflection image for training, and reconstruct the two-color reflection model based on the original specular image and the predicted specular reflection image for training to obtain the coarse diffuse reflection image for training. S4: Based on the original specular image, the predicted specular reflection image, and the coarse diffuse reflection image used for training, and combined with the specular region information provided by the specular mask used for training, multi-path gating refinement processing is performed. Through multi-path parallel feature processing and hierarchical feature fusion, the refined diffuse reflection image used for training is output. S5: Based on the training specular mask, manually labeled binary ground truth mask, training predicted specular reflection image, ground truth specular reflection image, training refined diffuse reflection image and original ground truth image without specular reflection, a multi-task combined loss function is constructed. The specular detection and removal network is iteratively trained using the multi-task combined loss function to obtain the trained specular detection and removal network. S6: Input the original highlight image to be processed into the trained highlight detection and removal network, and perform highlight detection processing, coarse highlight removal processing and fine highlight removal processing in sequence to obtain the reconstructed image after removing highlights. S7: During the verification phase, specular removal performance metrics are calculated based on the reconstructed image after specular removal and the original ground truth image without specular removal. Specular removal performance evaluation results are generated by combining these with preset evaluation rules. The specular removal performance evaluation results include: peak signal-to-noise ratio, structural similarity index, and pixel-level reconstruction error.
2. The method for detecting and removing highlights according to claim 1, characterized in that, S1 includes: The original specular image, manually labeled binary ground truth mask, real specular reflection image, and original ground truth image without specular reflection used for training are sized uniformly. The correspondence between the original specular image, manually labeled binary ground truth mask, real specular reflection image, and original ground truth image without specular reflection used for training is established to form training samples and construct a training dataset.
3. The method for detecting and removing highlights according to claim 1, characterized in that, The training specular mask generated through multi-scale feature extraction includes: Based on the U-Net network, feature extraction is performed on the original specular image used for training to obtain the input feature map of gated hole spatial pyramid pooling processing; The input feature map is subjected to three gated dilated convolutions, a convolution process, and a global pooling process, respectively. The dilation rates of the three gated dilated convolutions are set to 1, 4, and 8, respectively.
4. The method for detecting and removing highlights according to claim 3, characterized in that, The input feature map is processed by three gated dilated convolutions, convolutions, and global pooling: Each gated dilated convolution process is processed through parallel feature transformation and spatial gate weight generation to obtain feature transformation results and spatial gate weights. The spatial gate weights are then multiplied element-wise with the feature transformation results to obtain the gated dilated convolution processing result. The results of the three gated dilated convolutions, the convolution process, and the global pooling process are concatenated along the channel dimension, and the concatenated result is then convolved to generate a specular mask for training.
5. The method for detecting and removing highlights according to claim 1, characterized in that, Mask-guided gating feature fusion processing includes: Mask-guided gating feature fusion processing is set before the downsampling stage of each encoder in the encoding / decoding network; The training of a single channel is convolved using a specular mask to obtain mask features with the same dimension as the input feature channels in the encoding stage. The input features in the encoding stage are subjected to two consecutive convolutions, batch normalization, and activation processing to obtain enhanced features; The mask features and enhancement features are concatenated along the channel dimension to obtain joint features. The joint features are then subjected to gated convolution and connected with the enhancement features through residuals to obtain fused features.
6. The method for detecting and removing highlights according to claim 5, characterized in that, Outputting the predicted specular reflection image for training and obtaining the coarse diffuse reflection image for training includes: Encoding and decoding are performed based on fused features to output a predicted specular reflection image for training. Based on the two-color reflectance model, the original specular image used for training is subtracted from the predicted specular image used for training to obtain the coarse diffuse image used for training.
7. The method for detecting and removing highlights according to claim 1, characterized in that, Multi-path parallel feature processing includes: The original specular image used for training is convolved to perform channel dimension alignment and feature compression, resulting in the first path of features; The training coarse diffuse image is subjected to three consecutive gated convolutions and activations to obtain the second-path features; The training image of the predicted specular reflection is preprocessed by convolution to unify the channel dimensions and enhance cross-channel correlation, resulting in a third-path feature.
8. The method for detecting and removing highlights according to claim 7, characterized in that, The output of the refined diffuse image for training includes: The first and second features are concatenated along the channel dimension and then convolved to obtain the intermediate fused features. The intermediate fusion features and the third-path features are concatenated along the channel dimension and then convolved to obtain a refined diffuse reflection image for training.
9. The method for detecting and removing highlights according to claim 1, characterized in that, The multi-task combined loss function includes: mask segmentation loss, specular reconstruction loss, and image restoration loss; Based on the training specular mask and the manually labeled binary ground truth mask, the binary cross-entropy loss and Dice loss are calculated and combined to obtain the mask segmentation loss; Based on the predicted specular reflection image used for training, the real specular reflection image, and the specular mask used for training, the mask-weighted mean square error loss in the specular region and the perceptual loss based on the pre-trained VGG-16 network are calculated and combined to obtain the specular reconstruction loss. Based on the refined diffuse reflection image used for training and the original ground truth image without highlights, the perceptual loss and the style loss based on the feature map Gram matrix are calculated and combined to obtain the inpainted image loss. The mask segmentation loss, specular reconstruction loss, and image restoration loss are weighted and combined to obtain a multi-task combined loss function.
10. The method for detecting and removing highlights according to claim 1, characterized in that, S7 includes: Pixel-level reconstruction error, peak signal-to-noise ratio, and structural similarity index are calculated based on the reconstructed image after removing highlights and the original ground truth image without highlights. When the peak signal-to-noise ratio is greater than the preset first signal-to-noise threshold and the structural similarity index is greater than the preset first similarity threshold, the highlight removal is deemed complete and the repair effect is deemed satisfactory. When the peak signal-to-noise ratio is greater than the preset second signal-to-noise threshold and less than the preset first signal-to-noise threshold, and the structural similarity index is greater than the preset second similarity threshold and less than the preset first similarity threshold, it is determined that the highlights have been basically removed. When the peak signal-to-noise ratio is less than the preset second signal-to-noise threshold or the structural similarity index is less than the preset second similarity threshold, the highlight removal effect is judged to be poor and the repair quality is unqualified.
