A method for repairing highlight areas on a metal surface based on deep learning
By using a deep learning-based image inpainting method, a mask is generated and a highlight inpainting network is constructed, which solves the limitation of traditional algorithms in handling illumination changes in the highlight areas of metal surfaces, and achieves efficient image quality improvement and feature extraction.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- BEIJING INFORMATION SCI & TECH UNIV
- Filing Date
- 2025-01-14
- Publication Date
- 2026-07-14
AI Technical Summary
Traditional algorithms cannot effectively handle complex lighting changes when processing highlight areas on metal surfaces, resulting in degraded image quality and loss or misalignment of important features.
A deep learning-based approach is adopted to generate a mask using the image's own information, construct a highlight restoration network model, use an encoder and decoder for feature extraction and restoration, and combine a loss function for training to reduce manual annotation and achieve automatic restoration of highlight areas.
It effectively repairs highlight areas on metal surfaces, improves image quality, reduces manual annotation costs, and enhances the accuracy and consistency of feature extraction.
Smart Images
Figure SMS_25 
Figure SMS_50 
Figure FSA0000299943850000011
Abstract
Description
Technical Field
[0001] This invention relates to the field of image restoration, and more specifically, to a method for restoring highlight areas on a metal surface in a color image based on deep learning. Background Technology
[0002] Machine vision technology has been widely used in automated picking tasks. Machine vision systems can capture images of objects with cameras, extract the features of the target objects using image processing algorithms, and guide robotic arms to grasp them. However, some smooth or specularly reflective metal objects are prone to strong reflections, resulting in highlights or reflective spots in local areas, affecting image quality and causing the loss or misalignment of important features in the image.
[0003] Traditional image highlight processing algorithms mostly rely on lighting models, which use simplified mathematical formulas to describe the distribution of light sources, the intensity of reflection, and the material properties of surfaces. However, when applied to scenes with very dynamic lighting conditions and large changes in the intensity and angle of light sources, they exhibit certain limitations and cannot effectively cope with complex lighting changes.
[0004] To address the aforementioned problems, this invention proposes a deep learning-based method for highlight restoration of color images of metal surfaces. This method generates pseudo-labels using the image's own information, specifically masks for the highlighted areas to be restored, the non-highlight areas, and the areas that do not require restoration on the metal surface. Through a designed loss function and network model structure, highlight restoration of the metal surface is achieved without manual annotation, utilizing the image's own information. Summary of the Invention
[0005] To address the aforementioned technical problems, the present invention aims to provide a method for restoring highlight areas on metal surfaces in color images based on deep learning, thereby resolving the issues raised in the above context.
[0006] To achieve the above objectives, the present invention adopts the following technical solution:
[0007] A method for restoring highlight areas on a metal surface based on deep learning, characterized by the following specific steps:
[0008] Step 1: Obtain the dataset of reflective metal;
[0009] Step 2: Construct a deep learning-based specular restoration network model, which includes a mask generation module, an encoder, and a decoder;
[0010] Step 3: Train the constructed highlight restoration network;
[0011] Step 4: Use the image data of the reflective metal to be repaired as input, and apply the trained specular restoration network model to repair the specular areas on the metal surface.
[0012] Further, step 2 specifically involves:
[0013] Step 2.1 For the input image I input Perform color space transformations in HSV and Lab colors respectively to obtain I. hsv and I lab .
[0014] Step 2.2: For I hsv Thresholding is performed on the S and V channels to obtain the domain mask M for the highlight area of the metal surface that needs to be repaired. h And the mask M for pixel areas that do not need repair b ;
[0015] Step 2.3: For I lab K-means clustering was performed on channels a and b to obtain the mask M1 for the non-highlight region of the metal surface.
[0016] Step 2.4: Process the input image Perform normalization to obtain normalized data I; then apply the mask M. b The number of channels has been expanded to 3.
[0017] Step 2.5: Use the PCBA module to process the normalized image, expand the number of channels, update the mask, and then perform batch normalization and LeakyReLU activation function processing to obtain the feature map. and mask M b1 The expression is as follows:
[0018]
[0019] F1 = f LeakyReLU (f BN (F1′));
[0020] in, f represents a partial convolution with kernel k. BN f represents the batch normalization operation. LeakyReLU This represents the LeakyReLU activation function.
[0021] Step 2.6: Use the PCBA module to process the feature map The feature map is obtained through processing. and mask M b2 .
[0022] Step 2.7: Use the PCBA module to process the feature map The feature map is obtained through processing. and mask M b3 .
[0023] Step 2.8: Using the MDPC module, partially convolutional pairs of feature maps with kernels of 3 and dilation rates of 1, 2, and 4. After processing, three feature maps F4 are obtained. d＝1 F4 d＝2 and F4 d＝4 After concatenating along the channel dimension, a convolution with a kernel of 1 is used to reduce the number of channels, followed by batch normalization and LeakyReLU activation to obtain the feature map. And the mask M of the convolution output with a dilation rate of 1. b4 As the updated mask, the expression is as follows:
[0024]
[0025] in f represents a partial convolution with an inflation rate of d. Concat Indicates splicing along the channel dimension, f Conv1×1 f represents a convolution with a kernel of 1. BN f represents the batch normalization operation. LeakyRcLU This represents the LeakyReLU activation function.
[0026] Step 2.9: Simultaneously, the CSA module is used to process feature map F3 to obtain the spatial attention map. Channel attention map Multiply by feature map F3 until the feature map is obtained. The expression is as follows:
[0027] A s =σ(f Conv5×5 (F3));
[0028] A c =σ(f FC (f ReLU( f FC (f AgvPool (F3)+f MaxPool (F3)))));
[0029] F sca =F3⊙A s ⊙A c
[0030] Where f AgvPool For global average pooling, f MaxPool For global max pooling, f FC For a fully connected layer, σ is the Sigmoid activation function.
[0031] Step 2.10: Use the PCBA module to process the feature map The feature map is obtained through processing.
[0032] Step 2.11: Use the nearest neighbor interpolation and GCBA module to process the feature map. The feature map is obtained through processing. The expression is as follows:
[0033] F6 = f LeakyReLU (f BN (f GConv (u nearest (F5))))
[0034] Among them, u nearest f represents nearest neighbor interpolation GConv f represents gated convolution. BN f represents the batch normalization operation. LeakyReLU This represents the LeakyReLU activation function.
[0035] Step 2.12: Use neighbor interpolation to pair feature maps Process it, and then... The feature map is obtained by concatenating along the channel dimension. Then, the GCBA module is used for further processing to obtain the feature map.
[0036] Step 2.13: Process the feature map using nearest neighbor interpolation and the GCBA module.
[0037] Step 2.14: Repeat step 3.13.
[0038] Step 2.15: Repeat step 3.13 to obtain the output feature map.
[0039] Further, step 3 specifically involves inputting two images per round; training terminates when the loss value does not fall below the optimal loss value after 30 rounds or reaches 300 rounds; M is introduced. h M l M b F output By combining I with the output feature map, we obtain the highlight and non-highlight regions of the metal surface, as well as the unrepaired regions in the input and output feature maps. We then calculate the loss function and perform iterative training using gradient descent. The loss function expression is as follows:
[0040]
[0041] Where μ h μ represents the mean value of the highlight region on the metal surface of the output feature map.l σ represents the mean of the non-highlight region of the metal surface in the output feature map. h σ represents the standard deviation of the highlight region on the metal surface of the output feature map. l The standard deviations of the non-highlight regions of the metal surface in the output feature map are represented by C1 = 0.001, C2 = 0.0001, and y. i Indicates the non-repaired region of the input, y′ i λ represents the unrepaired region in the output, and λ represents the weighting coefficient.
[0042] Furthermore, step 4 specifically involves loading model weights, inputting the image to be repaired into the model, and obtaining the output feature map F. output The repaired image I is obtained by performing inverse normalization. output .
[0043] In summary, this invention generates domain masks for the highlight areas that need repair, pixel area masks that do not need repair, and non-highlight area masks for the metal surface through a mask generation module. This reduces manual annotation costs and is incorporated into the loss function calculation, using the image information itself to guide network learning. The encoder effectively masks highlight area features and extracts non-highlight area features through the PCBA module. The MDPC module expands the spatial receptive field, helping the model better understand the relationship between local and global features. The CSA module enhances feature information from different channels and spatial locations. These features are then stitched together in the decoder, resulting in better repair of highlight areas on the metal surface. Attached Figure Description
[0044] The accompanying drawings are provided to further understand the technical solutions of the present invention and constitute a part of the specification. They are used together with the embodiments of this application to explain the technical solutions of the embodiments of the present invention, and do not constitute a limitation on the technical solutions of the embodiments of the present invention.
[0045] Figure 1 This is a schematic diagram of the overall method flow of the present invention;
[0046] Figure 2 This is a schematic diagram of the deep learning-based specular restoration network model of the present invention;
[0047] Figure 3 This is a schematic diagram of the mask generation module;
[0048] Figure 4 This is a schematic diagram of the MDPC module of the present invention;
[0049] Figure 5 This is a schematic diagram of the SCA module of the present invention;
[0050] Figure 6 This is a comparison image of the restored image and the original image in this invention; Detailed Implementation
[0051] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the embodiments of the present invention will be described in detail below with reference to the accompanying drawings. It should be noted that, unless otherwise specified, the embodiments and features described in this application can be arbitrarily combined with each other.
[0052] A method for restoring highlight areas on metal surfaces based on deep learning, characterized in that, as Figure 1 As shown, the specific steps include:
[0053] Step 1: Obtain the dataset of reflective metal;
[0054] Step 2: Construct a deep learning-based specular restoration network model, which includes a mask generation module, an encoder, and a decoder;
[0055] Step 3: Train the constructed highlight restoration network;
[0056] Step 4: Use the image data of the reflective metal to be repaired as input, and apply the trained specular restoration network model to repair the specular areas on the metal surface.
[0057] Further, step 1 specifically involves:
[0058] Using a Hikvision MV-CA060-10GC color camera and a Hikvision MVL-HF0628M-6MP wide-angle lens, 100 color images of magnesium alloy ingots were acquired. The images were cropped to a size of 418×226, and all images were divided into training and test sets at a ratio of 9:1.
[0059] Further examples include: Figure 2 As shown, step 2 specifically involves:
[0060] Step 2.1 For the input image I input Perform color space transformations in HSV and Lab color spaces respectively to obtain I. hsv and I lae .
[0061] Step 2.2: For I hsv Thresholding is performed on the S and V channels. The intersection of regions with values less than 50 in the S channel and values greater than 220 in the V channel is used to obtain the domain mask M for the highlight area of the metal surface that needs to be repaired. h Conversely, the mask M of the pixel region that does not need repair is obtained. b ,like Figure 3 As shown;
[0062] Step 2.3: For Ilab After flattening the data from channels a and b, they are stacked into a two-dimensional array. K-means clustering is used to divide the array into three clusters. The cluster center with the smallest b value is found, yielding the mask M1 for the non-highlight region of the metal surface. Figure 3 As shown.
[0063] Step 2.4: Process the input image Perform normalization to obtain normalized data I; then apply the mask M. b The number of channels has been expanded to 3.
[0064] Step 2.5: Process the normalized image using a PCBA module with a kernel size of 7 to expand the number of channels and update the mask. Then perform batch normalization and LeakyReLU activation to obtain the feature map. and mask M b1 The expression is as follows:
[0065]
[0066] F1 = f LeakyReLU (f BN (F1));
[0067] in, f represents a partial convolution with kernel k. BN f represents the batch normalization operation. LeakyReLU This represents the LeakyReLU activation function.
[0068] Step 2.6: Use a PCBA module with a kernel size of 5 to process the feature map. The process involves a convolution kernel size of 7 to obtain feature maps. and mask M b2 .
[0069] Step 2.7: Use a PCBA module with a kernel size of 3 to process the feature map. The feature map is obtained through processing. and mask M b3 .
[0070] Step 2.8: Use the MDPC module, such as Figure 4 As shown, the feature maps of partial convolution pairs with kernels of 3 and dilation rates of 1, 2, and 4 are shown. After processing, three feature maps F4 are obtained. d＝1 F4 d＝2 and F4 d＝4 After concatenating along the channel dimension, a convolution with a kernel of 1 is used to reduce the number of channels, followed by batch normalization and LeakyReLU activation to obtain the feature map. And the mask M of the convolution output with a dilation rate of 1. b4 As the updated mask, the expression is as follows:
[0071]
[0072] in f represents a partial convolution with an inflation rate of d. Concat Indicates splicing along the channel dimension, f Conv1×1 f represents a convolution with a kernel of 1. BN f represents the batch normalization operation. LeakyReLU This represents the LeakyReLU activation function.
[0073] Step 2.9: Simultaneously, process feature map F3 using the CSA module, such as... Figure 5 As shown, the spatial attention map is obtained. Channel attention map Multiply by feature map F3 until the feature map is obtained. The expression is as follows:
[0074] A s =σ(f Conv5×5 (F3));
[0075] A c =σ(f FC (f ReLU (f FC (f AgvPool (F3)+f MaxPool (F3)))));
[0076] F sca =F3⊙A s ⊙A c
[0077] Where f AgvPool For global average pooling, f MaxPool For global max pooling, f FC For a fully connected layer, σ is the Sigmoid activation function.
[0078] Step 2.10: Use the PCBA module to process the feature map The feature map is obtained through processing.
[0079] Step 2.11: Use the nearest neighbor interpolation and GCBA module to process the feature map. The feature map is obtained through processing. The expression is as follows:
[0080] F6 = f LeakyRcLU (f BN (f GConv(u nearest (F5))))
[0081] Among them, u nearest f represents nearest neighbor interpolation GConv f represents gated convolution. BN f represents the batch normalization operation. LeakyReLU This represents the LeakyReLU activation function.
[0082] Step 2.12: Use neighbor interpolation to pair feature maps Process it, and then... The feature map is obtained by concatenating along the channel dimension. Then, the GCBA module is used for further processing to obtain the feature map.
[0083] Step 2.13: Process the feature map using nearest neighbor interpolation and the GCBA module.
[0084] Step 2.14: Repeat step 3.13.
[0085] Step 2.15: Repeat step 3.13 to obtain the output feature map.
[0086] Further, step 3 specifically involves inputting two images per round; training terminates when the loss value does not fall below the optimal loss value after 30 rounds or reaches 300 rounds; and introduces M... h M l M b F output We obtain the highlight and non-highlight regions of the metal surface in the output feature map, as well as the unrepaired regions in the input and output feature maps, and then calculate the loss function. Iterative training is performed using gradient descent. The loss function expression is as follows:
[0087]
[0088] Where μ h μ represents the mean value of the highlight region on the metal surface of the output feature map. l σ represents the mean of the non-highlight region of the metal surface in the output feature map. h σ represents the standard deviation of the highlight region on the metal surface of the output feature map. l The standard deviations of the non-highlight regions of the metal surface in the output feature map are represented by C1 = 0.001, C2 = 0.0001, and y. i Indicates the non-repaired region of the input, y′ i This represents the unrepaired region of the output, where λ1 = 1.5 and λ2 = 800.
[0089] Furthermore, step 4 specifically involves loading model weights, inputting the image to be repaired into the model, and obtaining the output feature map F. output The repaired image I is obtained by performing inverse normalization. output ,like Figure 6 As shown, the repaired image is compared with the original image, as well as the grayscale histogram.
[0090] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application.
Claims
1. A method for repairing highlight areas on a metal surface based on deep learning, characterized in that, The specific steps include: Step 1: Obtain the dataset of reflective metal; Step 2: Construct a deep learning-based specular restoration network model, which includes a mask generation module, an encoder, and a decoder; Step 3: Train the constructed highlight restoration network; Step 4: Use the image data of the reflective metal to be repaired as input, and apply the trained specular restoration network model to repair the specular areas on the metal surface.
2. The method for restoring highlight regions on a metal surface in a color image based on deep learning according to claim 1, characterized in that, Step 2 includes the following: Step 2.1 For the input image I input Perform color space transformations in HSV and Lab colors respectively to obtain I. hsv and I lab . Step 2.2: For I hsv Thresholding is performed on the S and V channels to obtain the domain mask M for the highlight area of the metal surface that needs to be repaired. h And the mask M for pixel areas that do not need repair b ; Step 2.3: For I lab K-means clustering was performed on channels a and b to obtain the mask M1 for the non-highlight region of the metal surface. Step 2.4: Process the input image Perform normalization to obtain normalized data I; then apply the mask M. b The number of channels has been expanded to 3. Step 2.5: Use the PCBA module to process the normalized image, expand the number of channels, update the mask, and then perform batch normalization and LeakyReLU activation function processing to obtain the feature map. and mask M b1 The expression is as follows: F1=f LeakyReLU (f BN (F1′)); in, f represents a partial convolution with kernel k. BN f represents the batch normalization operation. LeakyReLU This represents the LeakyReLU activation function. Step 2.6: Use the PCBA module to process the feature map The feature map is obtained through processing. and mask M b2 . Step 2.7: Use the PCBA module to process the feature map The feature map is obtained through processing. and mask M b3 . Step 2.8: Using the MDPC module, partially convolutional pairs of feature maps with kernels of 3 and dilation rates of 1, 2, and 4. After processing, three feature maps were obtained. and After concatenating along the channel dimension, a convolution with a kernel of 1 is used to reduce the number of channels, followed by batch normalization and LeakyReLU activation to obtain the feature map. And the mask M of the convolution output with a dilation rate of 1. b4 As the updated mask, the expression is as follows: in f represents a partial convolution with an inflation rate of d. Concat Indicates splicing along the channel dimension, f Conv1×1 f represents a convolution with a kernel of 1. BN f represents the batch normalization operation. LeakyReLU This represents the LeakyReLU activation function. Step 2.9: Simultaneously, the CSA module is used to process feature map F3 to obtain the spatial attention map. Channel attention map Multiply by feature map F3 until the feature map is obtained. The expression is as follows: A s =σ(f Conv5×5 (F3)); A c =σ(f FC (f ReLU (f FC (f AgvPool (F3)+f MaxPool (F3))))); F sca =F3⊙A s ⊙A c Where f AgvPool For global average pooling, f MaxPool For global max pooling, f FC For a fully connected layer, σ is the Sigmoid activation function. Step 2.10: Use the PCBA module to process the feature map The feature map is obtained through processing. Step 2.11: Use neighbor interpolation and the GCBA module to process the feature map. The feature map is obtained through processing. The expression is as follows: F6=f LeakyReLU (f BN (f GConv (u nearest (F5)))) Among them, u nearest f represents nearest neighbor interpolation GConv f represents gated convolution. BN f represents the batch normalization operation. LeakyReLU This represents the LeakyReLU activation function. Step 2.12: Use neighbor interpolation to pair feature maps Process it, and then... By concatenating along the channel dimension, a feature map is obtained. Then, the GCBA module is used for further processing to obtain the feature map. Step 2.13: Process the feature map using nearest neighbor interpolation and the GCBA module. Step 2.14: Repeat step 3.
13. Step 2.15: Repeat step 3.13 to obtain the output feature map.
3. The method for restoring highlight regions on a metal surface in a color image based on deep learning according to claim 1, characterized in that, Step 3 includes the following: Each round inputs two images; training terminates when the loss value does not fall below the optimal loss value after 30 rounds or reaches 300 rounds; introduce M... h M l M b F output By combining I with the output feature map, we obtain the highlight and non-highlight regions of the metal surface, as well as the unrepaired regions in the input and output feature maps. We then calculate the loss function and perform iterative training using gradient descent. The loss function expression is as follows: Where μ h μ represents the mean value of the highlight region on the metal surface of the output feature map. l σ represents the mean of the non-highlight region of the metal surface in the output feature map. h σ represents the standard deviation of the highlight region on the metal surface of the output feature map. l The standard deviations of the non-highlight regions of the metal surface in the output feature map are represented by C1 = 0.001, C2 = 0.0001, and y. i Indicates the non-repaired region of the input, y′ i λ represents the unrepaired region in the output, and λ represents the weighting coefficient.
4. The method for restoring highlight regions on a metal surface in a color image based on deep learning according to claim 1, characterized in that, Step 4 includes the following: loading model weights, inputting the image to be repaired into the model, and obtaining the output feature map F. output The repaired image I is obtained by performing inverse normalization. output .