A high light detection and elimination method based on polarization imaging and deep learning

Abstract

The application belongs to the technical field of highlight detection, and particularly relates to a highlight detection and elimination method based on polarization imaging and deep learning. The method comprises the following processes: image preprocessing of a data set and synthesis of polarization images; training of a Transform generation network, and use of a pre-trained VGG network as a discrimination network to detect the quality of the generated highlight-free image, and feedback of the result to the generation network to improve the effect; a highlight detection method based on the polarization characteristics of highlights, which uses three polarization images to locate the highlight area and obtain a binary image of the highlight area; and fine highlight elimination of a pseudo highlight-free image to obtain a highlight-free image. The method combines polarization imaging and deep learning methods to achieve simple and accurate positioning of highlights on the surface of an object, and effective removal and detail recovery of highlights on the surface of an object.

Description

Technical Field

[0001] This invention belongs to the field of specular detection technology, and particularly relates to a specular detection and elimination method based on polarization imaging and deep learning. Background Technology

[0002] Specular removal is a fundamental technology that has attracted much attention in fields such as computer vision, optical imaging, image processing, and computer graphics. Its main goal is to effectively separate specular highlight components from image data while restoring key information such as the original color and texture details of the specular highlight areas. The development of this technology is of great significance in improving image quality, enhancing visual perception, and achieving accurate image analysis and processing in real-world scenarios.

[0003] On certain special object surfaces, such as optical components, polishing molds, electroplating devices, and metallic materials, specular reflection occurs when detecting objects from specific angles, such as when the detector and illumination source are symmetrical to the plane containing the object's surface normal. The highlights reflected from the object surface cause localized overexposure in the image, obscuring surface details. This makes it difficult for traditional visual inspection methods to accurately measure defects under highlights, leading to distorted detection information and reduced accuracy. Current solutions mostly require multiple camera views, multi-directional projection, or traditional image optimization algorithms and deep learning methods to generate images without highlight areas. However, these methods require significant time investment in data acquisition, making them unsuitable for the practical needs of industrial development. Furthermore, their applicability is limited because data may need to be re-acquired and reprocessed when the object being measured changes. The method proposed in this invention significantly solves these problems. It not only effectively handles highlight surfaces and accurately locates highlight areas but also avoids wasting considerable time in the data acquisition and processing stages, and exhibits excellent stability in complex background detection scenarios. Summary of the Invention

[0004] To address the shortcomings of existing technologies, this invention fully utilizes the polarization characteristics of specular reflection from object surfaces, combining polarization imaging with deep learning-based specular removal techniques to propose a specular detection and removal method based on polarization imaging and deep learning. This method ensures imaging quality in non-spectral areas while restoring object details obscured by specular highlights. Compared to traditional image processing methods, it exhibits superior performance and can complete a single detection in a shorter time.

[0005] The technical solution of this invention is as follows:

[0006] A method for specular detection and removal based on polarization imaging and deep learning, characterized by the following steps:

[0007] S1. Create a training dataset, specifically: acquire images from a public dataset, and synthesize a polarization image dataset as training data using the composition relationship of reflected light from objects and the diffuse reflection image and reflected highlight image provided by the public dataset.

[0008] S2. A Transformer network is used as the generator network to generate images without highlights, and a pre-trained VGG network is used as the discriminator network to detect the quality of the generated images without highlights; the loss function is defined as:

[0009] ,

[0010] in, , and They are 1, 0.1, and 1 respectively. For pixel loss function, The VGG discriminant loss function is... Color loss function:

[0011] ,

[0012] ,

[0013] ,

[0014] in, To output the image, It is true. Set coefficients for the red, green, and blue channels of the image, respectively. , , and They represent gradient in direction, For the VGG network Feature images of the layer, It is the first The coefficients of the feature image of the layer, Representing different color channels Norm;

[0015] The generator network is trained using training data. The VGG network detects the generated images without highlights. Then, backpropagation is used to improve the highlight removal capability of the Transformer generator network, and finally, a well-trained generator network is obtained.

[0016] S3. A specular detection method based on specular polarization characteristics is used to acquire three polarization images to locate the specular region and obtain a pseudo-no-spectrum image with minimal specular highlights. and highlight positioning map ;

[0017] S4. Perform specular removal to obtain a pseudo-highlight-free image. Input the trained generative network to generate an image without highlights.

[0018] Furthermore, after obtaining the images from the public dataset in S1, the following processing is performed:

[0019] Adjust the image size:

[0020] For image size greater than The image, by padding the edges with zeros, yields a size of [size missing]. The image was then cropped into multiple sizes. Image collection ,in Indicates the number of pieces to be cut. , It is an integer;

[0021] For image size smaller The image is directly zero-padded to a size of 1. Or crop the center portion of the image to a size of Then interpolate to a size of , where n is the smaller of the width or height of the original image.

[0022] Furthermore, the images from the publicly available dataset include diffuse images, images with specular reflection, specular images, and specular localization images. The relationship between the polarization image dataset synthesized based on the relationship between the intensity of reflected specular and diffuse light and the polarization angle is as follows:

[0023] ,

[0024] in, It is a diffuse reflection image. and It is the polarized and unpolarized part of the reflected specular highlight synthesized from the image containing specular reflection. Three polarization images are acquired separately. The set of the three synthesized polarization images and their corresponding diffuse reflection images is used as training data, where the polarization images are the input and the diffuse reflection images are the ground truth.

[0025] Furthermore, the specific method of S3 is as follows:

[0026] A polarizing camera was used to photograph an object with highly reflective surfaces, and the polarization angles were obtained as follows: The three images , , Substitute the three polarization images into the Stokes matrix:

[0027] ,

[0028] Ignoring the circularly polarized component in natural scenes, then at any polarization angle... The image is:

[0029] ,

[0030] The algorithm optimizes the pseudo-no-highlight image to obtain the minimum highlight level. : ,

[0031] pass Calculate the image with maximum light intensity :

[0032] ,

[0033] The strongest highlight image With pseudo-no-highlight images Subtraction yields the region of highlight variation. : ,

[0034] Binarize the highlight variation region: Use the OTSU algorithm to obtain the threshold. The intensity is greater than the threshold. The area is considered a highlight area. Intensity less than the threshold value The area is considered a diffuse reflection area. :

[0035] ,

[0036] Highlight area Set to 1, diffuse reflection area Setting to 0 yields the highlight positioning image. :

[0037] .

[0038] Furthermore, the specific method of S4 is as follows:

[0039] The pseudo-no-highlight image to be input into the generator network is cropped as follows: The size of the image is input into a trained generative network to obtain a specular-free image. If C is greater than 1, then the C image slices are stitched together and cropped to the original size.

[0040] The beneficial effects of this invention are as follows:

[0041] This invention proposes a specular detection and removal method based on polarization imaging and deep learning. The method first acquires three polarization images using a polarization camera, and then calculates Stokes parameters using these three images. Using the Stokes parameters, images at any polarization angle can be calculated. Next, an optimization algorithm finds the image with the minimum global intensity, termed the pseudo-no-spectrum image, and the image with the maximum global intensity, termed the strongest specular image. These two images are subtracted, and then adaptive binarization is performed to accurately locate the specular region. Subsequently, the pseudo-no-spectrum image is input into a Transformer network for further refined specular removal. This method enables effective specular detection and removal on object surfaces. Attached Figure Description

[0042] Figure 1 This is a flowchart of a specular detection and removal method based on polarization imaging and deep learning.

[0043] Figure 2 Image acquisition process for polarization camera (hardware part).

[0044] Figure 3 Flowchart for highlight detection and removal in polarized images (software part).

[0045] Figure 4 Images representing all processes involved in a single detection procedure. Detailed Implementation

[0046] The specific embodiments of the present invention will be described in further detail below with reference to the accompanying drawings and examples.

[0047] Example

[0048] Reference Figures 1-3 In this embodiment, to eliminate highlights on object surfaces, polarization imaging and deep learning are combined to achieve simple, accurate, and efficient localization of highlight positions. Polarization imaging is sensitive to light polarization and has no specific requirements on the color of the light source. This method is suitable for scenarios with multi-source or colored light source illumination, solving the prior dependence of traditional image processing methods on the light source. Simultaneously, a Transformer network is used to further refine the highlight elimination, improving the overall highlight removal capability of the method and minimizing the impact on non-highlight areas.

[0049] Please refer to Figure 1 As shown in the flowchart, this embodiment proposes a specular detection and removal method based on polarization imaging and deep learning. The method includes the following steps:

[0050] S1. Image preprocessing and polarization image synthesis from the dataset. First, images from a publicly available dataset were acquired, including diffuse images, images with specular reflection, specular images, and specular localization images. Then, these images were resized using methods such as interpolation, boundary interpolation, and cropping. The standard input size is determined. Next, utilizing the polarization characteristics of the highlight portion, a random polarization angle is selected, and three polarization images are synthesized. Finally, these polarization images are combined with the corresponding diffuse reflection images to form the training set for the Transformer network.

[0051] The specific process of step S1 is as follows:

[0052] S1.1 Obtain images from a public dataset, including diffuse images, images with specular reflection, specular images, and specular localization images.

[0053] S1.2 To accommodate the input size of the Transformer network for training and testing, the image size is resized. For images larger than... For the image, we first pad the edges with zeros to obtain a size of [size missing]. The image was then cropped into multiple sizes. Image collection ,in Indicates the number of cropped slices. For image sizes smaller than [specified size]... For images like this, we can take two processing methods. One method is to directly pad the image with zeros to a size of [size missing]. Another way is to crop the center portion of the image, to a size of... Then interpolate to a size of , where n is the smaller of the width or height of the original image.

[0054] S1.3 Synthesis of Polarization Image Dataset. The polarization image dataset is synthesized based on the relationship between the intensity of reflected specular and diffuse light and the polarization angle, as shown in the following formula:

[0055] ,

[0056] Diffuse reflection images from publicly available datasets are used as diffuse reflection light intensity. Simultaneously, the polarization component of the reflected specular highlights is synthesized using the reflected specular highlight images. Non-polarized part , These are random coefficients. It's a random starting angle. Next, select... Three polarization images were acquired, namely: , , The network uses a set of three polarized images and their corresponding diffuse reflection images synthesized from a publicly available dataset as its training and testing dataset. In this dataset, the polarized images serve as input, while the diffuse reflection images represent the ground truth.

[0057] S2. Train the Transformer generator network and use the pre-trained VGG network as the discriminator network to detect the quality of the generated image without highlights. Feed the results back to the generator network to improve its performance.

[0058] The specific process of step S2 is as follows:

[0059] S2.1 divides the synthesized dataset into training, testing, and validation sets in a 7:2:1 ratio to prepare for training the Transformer generative network.

[0060] S2.2 uses methods such as random rotation, flipping, and adding Gaussian noise to enhance the input polarization image, thereby improving the stability of the network and the generalization ability of the overall specular removal process.

[0061] S2.3 Determine the loss function for network training. In this example, the loss function consists of a pixel loss function, a VGG discriminant loss function, and a color loss function. , and The values ​​are 1, 0.1, and 1 respectively.

[0062] ,

[0063] The pixel loss function is as follows:

[0064] ,

[0065] and They represent The gradient in the direction, the prime loss function can ensure texture details in the generated specular-free image.

[0066] The VGG discriminant loss function is as follows:

[0067] ,

[0068] For the VGG network Feature images of the layer, It is the first The coefficients of the feature image of the layer are selected as the feature function in this example, using the outputs of four layers of the VGG19 network: "conv2_2", "conv3_3", "conv4_4", and "conv5_5". eigenvalues for 1 / 2.56, 1 / 5.12, 1 / 3.5, 1 / 0.17}.

[0069] The color loss function is as follows:

[0070] ,

[0071] Representing different color channels Norm.

[0072] S2.4 Input the training dataset into the network for training, and select the Adam optimizer as the optimization function for network training.

[0073] S2.5 uses a pre-trained VGG network as the discriminator network to detect specular delight in the generated images, and then backpropagates to improve the specular delight removal capability of the Transformer generator network. The network's output image is... The true value is .

[0074] S3 is a highlight detection method based on the polarization characteristics of highlights. It uses three polarization images to locate the highlight region and obtain a binary image of the highlight region.

[0075] The specific process of step S3 is as follows:

[0076] S3.1 Use a polarizing camera to photograph objects with highly reflective surfaces, obtaining polarization angles as follows: The three images , , .

[0077] S3.2 Calculate the pseudo-no-highlight image and the strongest highlight image. Substitute the three polarization images into the Stokes matrix:

[0078] ,

[0079] In natural scenes, the component of circularly polarized light can be ignored, so at any polarization angle... The image is:

[0080] ,

[0081] By optimizing the algorithm, we can find the pseudo-no-highlight image with the minimum highlight.

[0082] ,

[0083] Then through Calculate the image with maximum light intensity :

[0084] ,

[0085] S3.3 will produce the strongest highlight images. With pseudo-no-highlight images Subtract to obtain the area of ​​highlight variation. :

[0086] ,

[0087] S3.4 performs binarization on the highlight variation region. The OTSU algorithm is used to obtain the threshold. The intensity is greater than the threshold. The area is considered a highlight area. Intensity less than the threshold value The area is considered a diffuse reflection area. :

[0088] ,

[0089] Highlight area Set to 1, diffuse reflection area Setting to 0 yields the highlight positioning image. :

[0090] ,

[0091] S4. Refined specular removal from pseudo-highlight-free images to obtain specular-free images. Refined specular removal based on a Transformer network transforms pseudo-highlight-free images... As input to the network, it is fed into the generator network to generate an image without highlights.

[0092] The specific process of step S4 is as follows:

[0093] S4.1 Using the method in step S1.1, crop the pseudo-no-highlight image to be input into the Transformer generator network. The size.

[0094] S4.2 will Input the trained Transformer generator network to obtain a specular-free image. If C is greater than 1, then the image slices of C need to be stitched together and cropped to their original size.

Claims

1. A method for specular detection and removal based on polarization imaging and deep learning, characterized in that, Includes the following steps: S1. Create a training dataset, specifically: acquire images from a public dataset, and synthesize a polarization image dataset as training data using the composition relationship of reflected light from objects and the diffuse reflection image and reflected highlight image provided by the public dataset. S2. A Transformer network is used as the generator network to generate images without highlights, and a pre-trained VGG network is used as the discriminator network to detect the quality of the generated images without highlights; the loss function is defined as: , in, , and They are 1, 0.1, and 1 respectively. For pixel loss function, The VGG discriminant loss function is... Color loss function: , , , in, To output the image, It is true. Set coefficients for the red, green, and blue channels of the image, respectively. , , and They represent gradient in direction, For the VGG network Feature images of the layer, It is the first The coefficients of the feature image of the layer, Representing different color channels Norm; The generator network is trained using training data. The VGG network detects the generated images without highlights. Then, backpropagation is used to improve the highlight removal capability of the Transformer generator network, and finally, a well-trained generator network is obtained. S3. A specular detection method based on specular polarization characteristics is used to acquire three polarization images to locate the specular region and obtain a pseudo-no-spectrum image with minimal specular highlights. and highlight positioning map ; S4. Perform specular removal to obtain a pseudo-highlight-free image. Input the trained generative network to generate an image without highlights.

2. The method for specular detection and removal based on polarization imaging and deep learning according to claim 1, characterized in that, After obtaining images from the public dataset, S1 also performs the following processing: Adjust the image size: For image size greater than The image, by padding the edges with zeros, yields a size of [size missing]. The image was then cropped into multiple sizes. Image collection ,in Indicates the number of pieces to be cut. , It is an integer; For image size smaller The image is directly zero-padded to a size of 1. Or crop the center portion of the image to a size of Then interpolate to a size of , where n is the smaller of the width or height of the original image.

3. The method for specular detection and removal based on polarization imaging and deep learning according to claim 1, characterized in that, Images from publicly available datasets include diffuse images, images with specular reflection, specular images, and specular localization images. The relationship between the polarization image dataset synthesized based on the relationship between the intensity of reflected specular and diffuse light and the polarization angle is as follows: ， in, It is a diffuse reflection image. and It is the polarized and unpolarized part of the reflected specular highlight synthesized from the image containing specular reflection. Three polarization images are acquired separately. The set of the three synthesized polarization images and their corresponding diffuse reflection images is used as training data, where the polarization images are the input and the diffuse reflection images are the ground truth.

4. The method for specular detection and removal based on polarization imaging and deep learning according to claim 3, characterized in that, The specific method for S3 is: A polarizing camera was used to photograph an object with highly reflective surfaces, and the polarization angles were obtained as follows: The three images , , Substitute the three polarization images into the Stokes matrix: , Ignoring the circularly polarized component in natural scenes, then at any polarization angle... The image is: , The algorithm optimizes the pseudo-no-highlight image to obtain the minimum highlight level. : , pass Calculate the image with maximum light intensity : , The strongest highlight image With pseudo-no-highlight images Subtraction yields the region of highlight variation. : , Binarize the highlight variation region: Use the OTSU algorithm to obtain the threshold. The intensity is greater than the threshold. The area is considered a highlight area. Intensity less than the threshold value The area is considered a diffuse reflection area. : , Highlight area Set to 1, diffuse reflection area Setting to 0 yields the highlight positioning image. : 。 5. The method for specular detection and removal based on polarization imaging and deep learning according to claim 4, characterized in that, The specific method for S4 is: The pseudo-no-highlight image to be input into the generator network is cropped as follows: The size of the image is input into a trained generative network to obtain a specular-free image. If C is greater than 1, then the C image slices are stitched together and cropped to the original size.

Images