Scratch defect enhancement method based on multi-modal gradient map fusion

Abstract

The invention provides a scratch defect enhancement method based on multi-modal gradient graph fusion, which can enhance graph features, obtain a fused image with obvious scratch defects, and provide a data basis for improving the detection precision of tiny scratches on the metal surface. An eight-direction light source is used for obtaining an image, specular reflection of the surface of a workpiece can be restrained, and the synthesis quality is closer to a theoretical value; a direction-insensitive gradient enhancement method is constructed, gradient enhancement is performed on X and Y directions of a normal vector, scratch defects in the normal vector are effectively extracted, then gradient enhancement is performed on a reflectivity image, the noise of the obtained enhanced image is obviously reduced, but the performance of the scratch effect is weakened, and finally, fusion is performed on the three images, so that the reflectivity of the image is improved. A multi-modal fusion gradient map with obvious scratches and less noise is obtained, and scratch enhancement and noise suppression are realized by fusing multiple gradient enhancement maps.

Description

Technical Field

[0001] The present invention relates to the technical field of intelligent traffic control, and in particular to a scratch defect enhancement method based on multi-modal gradient map fusion. Background Art

[0002] Vision-based quality inspection is increasingly used in the metal processing industry. In traditional two-dimensional grayscale images, metal surface defect detection is often affected by insufficient contrast, blurred edge details, noise interference, and lack of texture information. In order to improve the contrast of images and improve detection accuracy, some researchers have adopted a variety of image enhancement techniques. For example, Peng Ning et al. combined median filtering, improved wavelet threshold denoising, and DeBlurGANV2 deblurring technology to denoise images; Tang Bo et al. used wavelet transform and homomorphic filtering technology to enhance the contrast of images; Chen Congping et al. based on a zero-reference deep learning network, through a specific loss function, improved the contrast of low-light and detail-missing images; and Luo Hui et al. used Gabor filtering and HSV space transformation algorithms to enhance the key features of defect images. Although these methods have improved the enhancement effect of defect images to a certain extent, two-dimensional graphics cannot reflect the height information of defects, which makes it difficult to identify defects with height as the main feature.

[0003] In recent years, image processing technology combined with depth information has made significant progress in the field of defect detection. Researchers have effectively enhanced the detection of metal surface defects by using three-dimensional feature information. For example, Xiao Tianxing proposed a defect enhancement method combining two-dimensional images and three-dimensional point cloud projection images for two-dimensional image defects of cans. Zong et al. established a mapping relationship between images and point clouds and used a segmentation algorithm to calculate the three-dimensional parameters of defects, thereby improving the recognition of defect features. Obtaining three-dimensional information of target objects is crucial for machine vision, so photometric stereo, which can obtain surface detail information at a low cost, has been widely used. Photometric stereo can obtain three-dimensional information of the surface of an object through images under light sources from different directions, and express it in normal vector maps, depth maps, and reflectivity maps. Compared with traditional two-dimensional image enhancement technology, photometric stereo can more effectively distinguish defects from backgrounds by using three-dimensional feature information of metal surfaces, thereby improving the accuracy and reliability of defect detection.

[0004] However, when performing scratch detection based on the existing photometric stereo method, it was found that there are some tiny defects on the surface of many metal products. When detecting scratch defects, these tiny defects cannot be shown in the depth map because of their shallow depth; at the same time, the pattern and texture of the workpiece itself will also have a negative impact on the detection, ultimately leading to the problem of insufficient accuracy of the detection results. Summary of the invention

[0005] In order to solve the problem of insufficient detection accuracy of tiny defects on metal surfaces when performing scratch inspection on metal products based on the existing photometric stereo method, the present invention provides a scratch defect enhancement method based on multimodal gradient image fusion, which can enhance image features and obtain a fused image with obvious scratch defects, thereby providing a data basis for improving the detection accuracy of tiny scratches on metal surfaces.

[0006] The technical solution of the present invention is as follows: a scratch defect enhancement method based on multi-modal gradient map fusion, characterized in that it comprises the following steps:

[0007] S1: Obtain the normal vector map and reflectivity map of the object to be detected based on the eight-directional photometric stereo method;

[0008] S2: Construct a direction-insensitive gradient enhancement method, which includes the following steps:

[0009] S21: Use Sobe1 operator to perform gradient operation, and derive the image to be enhanced from the x-axis and y-axis directions respectively;

[0010] g x =(z7+2z8+z9)-(z1+2z2+z3);

[0011] g y =(z3+2z6+z9)-(z1+2z4+z7);

[0012] In the formula, the subscripts 1 to 9 represent a 3x3 neighborhood of the target pixel, which are arranged from left to right and from top to bottom. The target pixel corresponds to position 5 in the neighborhood, z represents the grayscale value of the position, and g x Represents the gradient of the target pixel in the x direction, g y Represents the gradient in the y direction of the pixel;

[0013] S22: For g x and g y of fusion;

[0014] g′ M =sgn(g x -g y )×(|g x -g y |+|g x +g y |);

[0015] In the formula, g′ M Represents the gradient value of any pixel in the result image; sgn represents the sign function, which is used to obtain the positive and negative signs corresponding to the pixels;

[0016] S3: Based on the direction-insensitive gradient enhancement algorithm, the normal vector map is gradient enhanced in the X and Y channel directions respectively to effectively extract the scratch defects, and the following are obtained: the enhanced normal vector map in the X channel direction and the enhanced normal vector map in the Y channel direction;

[0017] Performing gradient enhancement on the reflectivity map to obtain: an enhanced reflectivity map;

[0018] S4: fusing the enhanced normal vector map in the X-channel direction, the enhanced normal vector map in the Y-channel direction and the enhanced reflectivity map in the gradient direction to obtain a multi-modal fused gradient map after the corresponding scratch defect feature of the object to be detected is enhanced.

[0019] It is further characterized by:

[0020] In step S1, constructing a collection device for the eight-directional photometric stereo method;

[0021] The acquisition device comprises: a reference platform, a ring light source, a camera and a lens;

[0022] The reference platform is placed horizontally, and the object to be detected is placed on the reference platform; the camera is arranged above the lens, the details and the lens are vertically installed directly above the reference platform, and the annular light source partition is fixed between the camera device and the reference platform;

[0023] The annular light source is divided into eight directions, the intensity of each divided light source is uniform and the center of the light source is equidistant from the center of the reference platform;

[0024] The light source orientations in the annular light source include: 0°, 45°, 90°, 135°, 180°, 225°, 270°, and 315°;

[0025] In step S1, a normal vector map and a reflectivity map of the object to be detected are obtained, which specifically includes the following steps:

[0026] a1: Collect images based on light sources in eight directions to obtain illumination images in eight directions;

[0027] a2: Sort the illumination images in eight directions from small to large according to the azimuth angle of the light source, with serial numbers from 1 to 8;

[0028] a3: Divide the four pictures with even numbers into one group, and the four pictures with odd numbers into one group, to obtain two groups of near-field four-directional photometric stereo models;

[0029] Then one of the corresponding azimuths is: 0°, 90°, 180° and 270°;

[0030] Another set of corresponding azimuth feet are: 45°, 135°, 225° and 315°;

[0031] a4: Construct the basic formula:

[0032] Using the Lambertian surface assumption, the diffuse reflection formula of any incident light in space on the Lambertian surface is given by formula (1);

[0033] The unit light source direction vector is expressed in the form of trigonometric functions of azimuth and elevation angles to obtain formula (2);

[0034] Introduce the surface unit normal vector into the normal vector components p and q to be solved, and obtain formula (3);

[0035] The gray value of the image and the azimuth angle of the image are related to each other, and formula (4) is obtained.

[0036] I i =E c pn·l i (1);

[0037]

[0038] In the formula, the subscript i represents the light source number, I represents the grayscale value of the object illumination image under a certain direction of light source; τ i Represents the angle between the light source direction vector and the surface normal of the object; θ i Indicates the pitch angle of the light source direction vector; Indicates the azimuth of the light source direction vector; n represents the normal vector of the object surface, p and q represent the components of the normal vector in the X and Y directions; 1 represents the incident direction of the light source in polar coordinates; p represents the reflectivity of the object surface; Ec represents the intensity of the light source;

[0039] a5: Substituting the trigonometric function values ​​of the azimuth angles of the two sets of near-field four-directional photometric stereo models into formula (4), the two separate sets of photometric stereo models are derived to obtain formulas (5) and (6) respectively;

[0040]

[0041]

[0042] Wherein, the directions of 0° and 180°, 90° and 270°, and 45° and 225° are in a positional relationship with the directions of 135° and 315°; Ec represents the intensity of the light source, n represents the normal vector of the object surface, l represents the incident direction of the light source in polar coordinates; θ represents the pitch angle of the light source direction vector;

[0043] a6: Using the positional relationship of the light source to construct an eliminable basis, a simplified calculation process is derived to obtain formulas (7) and (8);

[0044]

[0045] In the formula, I p1 ,I q1 ,I p2 and I q2 It is calculated by the grayscale value of the corresponding position of the image collected in eight directions. The trigonometric function value of the pitch angle in the light source direction is determined by the actual layout of the image acquisition device and is calculated through the light source calibration process; sum1 ,I sum2 , p1, p2, q1 and q2 are intermediate calculation results;

[0046] a7: calculated based on formulas (7) and (8): p1, p2, q1 and q2;

[0047]

[0048] a8: Take the average value of p1, p2, q1 and q2 corresponding to the two sets of near-field four-directional photometric stereo models to obtain the normal vector components p and q to be solved;

[0049]

[0050] a9: The calculated normal vector components p and q are normalized to the range of [0, 255], rounded, and mapped to the three channel values ​​of the visualization color space to finally generate the normal vector map of the surface of the object to be detected;

[0051] a10: based on the normal vector map of the surface of the object to be detected, the reflectivity map of the surface of the object to be detected is calculated according to formula (12);

[0052]

[0053] The present application provides a scratch defect enhancement method based on multimodal gradient map fusion. The method uses eight-directional light sources to acquire images, which can curb the specular reflection of the workpiece surface and make the synthesis quality closer to the theoretical value. A direction-insensitive gradient enhancement method is constructed to perform gradient enhancement on the X and Y directions of the normal vector respectively, effectively extract the scratch defects therein, and then perform gradient enhancement on the reflectivity map. The obtained enhanced map has a significant reduction in noise, but the performance of the scratch effect is weakened. Finally, the three are fused to obtain a multimodal fused gradient map with obvious scratches and less noise. By fusing multiple gradient enhancement maps to achieve scratch enhancement and noise suppression, a data basis can be provided for improving the detection accuracy of tiny scratches on metal surfaces. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 It is a schematic diagram of the structure of the eight-directional photometric stereo model;

[0055] Figure 2This is a comparison chart of the eight-directional photometric stereo calculation results;

[0056] Figure 3 Multimodal gradient-enhanced graph fusion process;

[0057] Figure 4 Multimodal gradient enhancement graph fusion results;

[0058] Figure 5 Normal vector calculation time comparison chart;

[0059] Figure 6 Peak signal-to-noise ratio comparison of normal vector graph;

[0060] Figure 7 Peak signal-to-noise ratio comparison of reflectivity images. DETAILED DESCRIPTION

[0061] The present application includes a scratch defect enhancement method based on multi-modal gradient image fusion, which includes the following steps.

[0062] S1: Obtain the normal vector map and reflectivity map of the object to be detected based on the eight-directional photometric stereo method.

[0063] In the application of photometric stereo technology, the common model assumptions include uniform parallel light, Lambertian surface and orthogonal projection. However, in actual industrial environments, objects that meet the Lambertian surface are not common, because the surface of an object usually has both diffuse reflection and specular reflection. In this case, collecting images from more directions can effectively suppress the specular reflection of the object surface. Therefore, the present invention uses eight-directional photometric stereo synthesis to construct a Figure 1 The collection device based on eight-directional photometric stereo method is shown.

[0064] The acquisition device includes: a reference platform, a ring light source, a camera and a lens;

[0065] The reference platform is placed horizontally, and the object to be detected is placed on the reference platform; the camera is set above the lens, and the lens is installed vertically above the reference platform. Figure 1 In the embodiment shown, the camera and the lens form a combined component. The annular light source partition is fixed between the camera device and the reference platform. Figure 1 The object to be detected in is a cylindrical object. represents the surface normal vector of the point, Indicates the incident direction of the eight partitioned light sources.

[0066] The annular light source is divided into eight azimuths, the intensity of each partitioned light source is uniform and the center of the light source is equidistant from the center of the reference platform. In this embodiment, the azimuth angles of the light source in the annular light source include: 0°, 45°, 90°, 135°, 180°, 225°, 270°, and 315°.

[0067] based on Figure 1 The image acquisition process of the acquisition device shown includes the following steps:

[0068] S11: The camera mainly responds to diffuse reflection when the light source is close to the surface of the object, and the influence of specular reflection is reduced by adding the light map to the calculation.

[0069] S12: Calibrate and correct each partition of the ring light source to achieve accurate control of diffuse reflection;

[0070] S13: combining illumination maps from different directions to form a multi-level photometric stereo model, and fusing the calculated three-dimensional information;

[0071] S14: Based on the known prior information such as the azimuth and elevation angle of the light source, the calculation process of the surface normal vector is simplified to achieve a rapid solution to the three-dimensional information.

[0072] In step S1, a normal vector map and a reflectivity map of the object to be detected are obtained based on the eight-directional photometric stereo method, which specifically includes the following steps.

[0073] a1: Collect images based on light sources in eight directions to obtain illumination images in eight directions.

[0074] a2: Sort the illumination images in eight directions from small to large according to the azimuth angle of the light source;

[0075] That is, Figure 1 The light source azimuth angles in the middle ring light source include: 0°, 45°, 90°, 135°, 180°, 225°, 270°, and 315°, which are numbered in sequence from 1 to 8.

[0076] a3: Divide the four even-numbered sheets into one group, and the four odd-numbered sheets into one group, to obtain two groups of near-field four-directional photometric stereo models. The azimuths corresponding to one group are: 0°, 90°, 180° and 270°; the azimuths corresponding to the other group are: 45°, 135°, 225° and 315°.

[0077] a4: Construct the basic formula.

[0078] Using the Lambertian surface assumption, the diffuse reflection formula for any incident light in space shining on the Lambertian surface is given by formula (1); the product of the light source direction and the surface normal vector is the cosine value of the angle between the two. Expand it in the form of vector product to simplify formula (1).

[0079] In order to simplify the eight-directional photometric stereo model, the unit light source direction vector is expressed in the form of trigonometric functions of azimuth and elevation angles, and formula (2) is obtained.

[0080] Introducing the surface unit normal vector into the normal vector components p and q to be solved, we obtain formula (3).

[0081] The relationship between the gray value of the image and the azimuth angle of the image is established to obtain formula (4).

[0082] I i =E c pn·l i (1);

[0083]

[0084] In the formula, the subscript i represents the light source number, I represents the grayscale value of the object illumination image under a certain direction of light source; τ i Represents the angle between the direction vector of the i-th light source and the normal vector of the object surface; θ i Represents the pitch angle of the i-th light source direction vector; Indicates the azimuth of the light source direction vector; n represents the normal vector of the object surface, p and q represent the components of the normal vector in the X and Y directions; 1 represents the incident direction of the light source in polar coordinates; p represents the reflectivity of the object surface; Ec represents the intensity of the light source;

[0085] In the prior art, most of the photometric stereo models based on the classic pseudo-inverse solution use the pseudo-inverse formula to solve formula (1).

[0086] Pseudo-inverse formula: E p ·n i =I i ·L T ·(LL T ) -1 ;

[0087] However, as the light source gets closer to the object, the difference between the light source model and the ideal parallel light assumption increases, and the generalized inverse matrix cannot be directly solved according to the pseudo-inverse formula.

[0088] If the global information of the image is calculated pixel by pixel according to formula (1), the time complexity of the algorithm will be significantly increased and the operating efficiency will be greatly reduced. Therefore, it is necessary to simplify the solution method in combination with the hardware layout of the acquisition device based on the eight-directional photometric stereo method designed by the present invention. By constraining some conditions of the model, the components of p and q can be solved through a simple linear calculation process, replacing the complex pseudo-inverse matrix calculation under multiple light sources, thereby greatly improving the calculation efficiency. In this simplified model, the positions of the eight directional light sources are fixed, and their azimuth angles are They are 0°, 45°, 90°, 135°, 180°, 225°, 270° and 315° respectively; the plane where the light source is located is parallel to the plane of the workpiece, so that the angle θ does not change with the direction of the light source; the intensity of the light source in each direction is equal, and the corresponding value E of each pixel is corrected by grayscale correction. c Similarly, the formula can be greatly simplified on this basis.

[0089] Specifically, the illumination images in the directions of 0°, 90°, 180° and 270° are combined into a first-level near-field four-directional photometric stereo model, and the illumination images in the directions of 45°, 135°, 225° and 315° are combined into another first-level near-field four-directional photometric stereo model.

[0090] a5: Substituting the trigonometric function values ​​of the azimuth angles of the two sets of near-field four-directional photometric stereo models into formula (4), the two separate sets of photometric stereo models are derived to obtain formulas (5) and (6) respectively;

[0091]

[0092]

[0093] Wherein, the directions of 0° and 180°, 90° and 270°, and 45° and 225° are in a correspondence relationship with the directions of 135° and 315°; Ec represents the intensity of the light source, n represents the normal vector of the object surface, and l represents the incident direction of the light source in polar coordinate form.

[0094] a6: To further simplify the calculation process, the difference in the pitch angle of the light source direction of each pixel in the image is ignored, and the positional relationship of the light source is used to construct an eliminable basis, thereby deriving a simplified calculation process and obtaining formulas (8) and (9);

[0095]

[0096] In the formula, I p1 ,I q1 ,I p2 and I q2 It is calculated by the grayscale value of the corresponding position of the image collected in eight directions. The trigonometric function value of the pitch angle in the light source direction is determined by the actual layout of the image acquisition device and is calculated through the light source calibration process; sum1 ,I sum2 , p1, p2, q1 and q2 are intermediate calculation results and have no specific meaning.

[0097] a7: calculated based on formulas (7) and (8): p1, p2, q1 and q2;

[0098]

[0099] a8: Take the average value of p1, p2, q1 and q2 corresponding to the two sets of near-field four-directional photometric stereo models to obtain the normal vector components p and q to be solved;

[0100]

[0101] a9: The calculated normal vector components p and q are normalized to the range of [0, 255], rounded, and mapped to the three channel values ​​of the visualization color space to finally generate the normal vector map of the surface of the object to be detected.

[0102] In this method, formulas (5)(6)(7)(8) are introduced to simplify the calculation process and significantly improve the calculation efficiency.

[0103] a10: based on the normal vector map of the surface of the object to be detected, the reflectivity map of the surface of the object to be detected is calculated according to formula (12);

[0104]

[0105] In this method, eight-directional light sources are used to obtain images, which can curb the specular reflection of the workpiece surface and make the synthesis quality closer to the theoretical value; a simplified formula is derived for photometric stereo synthesis under eight directions, which effectively improves the calculation speed. The comparison effect of the normal vector map and reflectivity map collected and calculated based on the eight-directional photometric stereo method is shown in the figure. Figure 2 shown. Figure 2 The object to be detected is a button battery.

[0106] S2: Construct a direction-insensitive gradient enhancement method, which specifically includes the following steps.

[0107] For scratch defects, gradient features are a good identification method. Scratches usually bring obvious grayscale or color changes, and gradient features can accurately capture these changes, thereby clearly marking the location and shape of the scratches. Since scratches have a high contrast with the surrounding areas, the gradient can identify these high-contrast areas, making scratch extraction more reliable. In addition, the gradient is robust to noise and can highlight the characteristics of scratches in noise. Therefore, the gradient feature can sensitively capture the scratch information in the image. Usually, the gradient is calculated for the x and y directions respectively, and then the effects of the two are combined.

[0108] S21: This method uses the Sobel operator to perform gradient calculations, and derives the image to be enhanced from the x-axis and y-axis directions respectively; the Sobel operator assigns different weights to different positions, which can achieve a smoothing effect;

[0109] g x =(z7+2z8+z9)-(z1+2z2+z3) (13);

[0110] g y =(z3+2z6+z9)-(z1+2z4+z7) (14);

[0111] In the formula, the subscripts 1 to 9 represent a 3x3 neighborhood of the target pixel, which are arranged from left to right and from top to bottom. The target pixel corresponds to position 5 in the neighborhood, z represents the grayscale value of the position, and g x Represents the gradient of the target pixel in the x direction, g y Represents the gradient in the y direction of the pixel. Usually g x With g y Addition or subtraction is used as the fusion method of the two. In this way, the grayscale value of the smooth area is 127, which is gray; one side of the scratch is high grayscale, and the other side is low grayscale. Since the scratch width is narrow, the scratch defect has the highest contrast at this time. However, the addition and subtraction methods have serious directional sensitivity. For example, when the two are added, if the gradients in the two directions of the point are of different signs, even if the gradients in the two directions are large, the calculation result will still be small, and the scratch detection effect in this direction is poor. Therefore, formula (15) is used as the fusion method of the two.

[0112] g M =|g x -g y |+|g x +g y | (15);

[0113] This method can reduce directional sensitivity, but the processed gradients are all positive, the strong contrast on both sides of the scratch no longer exists, and the recognition effect is reduced. To avoid this problem, it is necessary to assign different signs to the two sides of the gradient. Therefore, this experiment uses (g x -g y ) is assigned as shown in formula (16).

[0114] S22: For g x and g y of fusion;

[0115] g′ M =sgn(g x -g y )×(|g x -g y |+|g x +g y |) (16);

[0116] In the formula, g′ m Represents the gradient value of any pixel in the result image; sgn represents the sign function, which is used to obtain the positive and negative signs corresponding to the pixels.

[0117] Through the above strategy, the gradient enhancement method of the present invention can maintain the contrast of scratches while reducing directional sensitivity, and only brings a small increase in the amount of calculation.

[0118] This method explores the implementation of photometric stereo, and generates a normal vector map and a reflectivity map through an eight-directional photometric stereo algorithm. The information of the workpiece surface is expressed in two images. The normal vector map expresses the normal information of each pixel of the workpiece image, which can reflect the tiny bumps and shapes of the surface, so it can effectively display the three-dimensional information of the scratch. The normal vector map is divided into three channels, X, Y, and Z, which respectively represent the gradient values ​​of the workpiece surface changing in three directions. For the detection of scratch defects, the X and Y channels are more important, so the two channels are taken out here for gradient enhancement, so that the gradient information of the scratches in different directions is expressed in different channels, retaining more feature information.

[0119] The reflectivity map expresses the intensity of reflected light at each pixel of the workpiece image, and can represent the properties and texture of the surface material. If the workpiece surface is noisy, such as the surface is brushed or has a complex texture, its concave and convex information will affect the normal vector map, and will be mixed with shallow scratch defects in the map. However, in the reflectivity map, the same texture has similar reflectivity, and its reflectivity value fluctuates smoothly; scratches will affect the roughness of the material, so they will also be expressed in the reflectivity map. Therefore, by gradient enhancement of the reflectivity map, a gradient image with slightly lower scratch contrast but high signal-to-noise ratio can be obtained.

[0120] This method avoids the problem of insensitivity of the classical algorithm in some directions by changing the gradient calculation method; the gradient sign calculated by the classical algorithm is applied to the result of the current algorithm, so that the defect contrast of the current algorithm is equivalent to that of the classical algorithm.

[0121] S3: Based on the direction-insensitive gradient enhancement algorithm, the normal vector map is gradient enhanced in the X and Y channel directions respectively to effectively extract the scratch defects, and the following are obtained: the enhanced normal vector map in the X channel direction and the enhanced normal vector map in the Y channel direction;

[0122] Perform gradient enhancement on the reflectivity map to obtain: enhanced reflectivity map;

[0123] S4: The enhanced normal vector map in the X channel direction, the enhanced normal vector map in the Y channel direction and the enhanced reflectivity map are fused in the gradient direction to obtain a multi-modal fusion gradient map after the corresponding scratch defect feature of the object to be detected is enhanced.

[0124] In this method, the X and Y directions of the normal vector are gradient enhanced respectively to effectively extract the scratch defects; after the reflectivity map is gradient enhanced, the enhanced map obtained has a significant reduction in noise, but the performance of the scratch effect is weakened; (the three are fused to obtain a multimodal fused gradient map with obvious scratches and less noise. For the specific multimodal gradient enhancement map fusion process, please refer to the attached Figure 3 , the fusion effect is as follows Figure 4 As shown in the figure, the image obtained in this way has obvious scratches, rich defect information, and effectively suppressed noise.

[0125] In order to verify the effect of the scratch defect enhancement method proposed in the present invention, the following experiment was designed.

[0126] The experimental environment of the weak defect enhancement algorithm based on multi-view fusion is as follows: 12th Gen Intel(R)Core(TM)i7-12700KF 3.6GHz processor, 32.0GB memory, NVIDIA GeForce RTX3060 graphics card, and Windows10 Home Chinese version operating system host. The experimental image acquisition system is implemented in Python programming, the development platform uses Pycharm 2024 version, and uses external library functions such as OpenCV, Pytorch, and Scikit-Image.

[0127] The specific experimental content is: use an eight-directional photometric stereo acquisition system to collect 198 button batteries, synthesize the normal vector map and reflectivity map of the corresponding number of groups, then perform gradient enhancement and fusion on them, and then verify the effect of each step respectively.

[0128] 1. For the simplified eight-direction photometric stereo method, eight-direction images of 80 groups of batteries were randomly selected, and the image resolution was cropped to 1400×1400 pixels. The simplified eight-direction model proposed in this application and the classical pseudo-inverse photometric stereo model (hereinafter referred to as the classical model or classical algorithm) were used to calculate their normal vectors respectively, and their running time was counted. The results are as follows: Figure 5 The test results show that the average running time for calculating the normal vector of the classic model is about 34 milliseconds, while the calculation time of the simplified model is only 19 milliseconds, which is a significant improvement in calculation speed.

[0129] As for the accuracy of the simplified eight-direction model, the peak signal-to-noise ratio is used here to evaluate the accuracy of the three-dimensional information obtained by the algorithm and the classic photometric stereo model. 40 button batteries were randomly selected for synthesis, and the statistical results of the normal vector and reflectivity were obtained. Figure 6 and Figure 7The experiment shows that the average peak signal-to-noise ratios of the normal map and reflectance map obtained by the present invention are 18.2dB and 13.0dB respectively, while the average peak signal-to-noise ratios of the classic photometric stereo model are only 9.5dB and 7.3dB. Therefore, from the perspective of image quality, the simplified eight-directional photometric stereo method is more accurate than the classic model.

[0130] 2. For the direction-insensitive gradient enhancement method, we select the normal vector map and reflectivity map generated by 198 button batteries, apply the classic gradient enhancement operator and the improved gradient enhancement operator to them, and generate synthetic images as test sets. We use the noise level, edge strength, contrast, clarity, and information entropy indicators for comprehensive comparison. The results are shown in Table 1. All four indicators are higher and better. It can be seen that the improved algorithm has an advantage in edge strength and contrast indicators, but has a significant advantage in clarity and information entropy indicators. In general, the improved algorithm can extract more gradient information, the image has higher clarity, and will not have a significant impact on edge strength and contrast.

[0131] Table 1 Comparison of the effects of gradient enhancement operators

[0132]

[0133] 3. For the fusion of multimodal gradient enhancement images, the normal vector image, gradient image and fusion image generated by 198 button batteries are selected for comparative experiments. The edge strength, contrast, clarity and information entropy indicators are also used. The results are shown in Table 2.

[0134] Table 2 Comparison of multimodal gradient enhancement fusion images

[0135]

[0136] Based on the data in Table 2, it can be seen that as a price for less noise, the various indicators of the reflectivity map are not as good as those of the normal vector map. The fusion map combines the characteristics of the two very well. Compared with the normal vector map, after the introduction of the reflectivity map, the contrast and information entropy indicators do not change significantly, and the edge strength and clarity decrease only slightly, achieving the expected fusion effect.

[0137] For fine scratches, especially those with small contrast between scratches and background and complex background noise, it is difficult to effectively enhance them based on the existing photometric stereo method, which is not conducive to subsequent detection. This method explores the implementation scheme of photometric stereo and uses the eight-directional simplified photometric stereo method to obtain the normal vector map and reflectivity map; then the different manifestations of scratch defects in the two images are studied, and a defect enhancement method is proposed that performs gradient enhancement on different modal images, namely the normal vector map and the reflectivity map, and then fuses them, which effectively improves the contrast between scratches and the background; and the classic gradient calculation method is improved to reduce its directional sensitivity and optimize the gradient map effect, ensuring that it can provide a data basis for improving the detection accuracy of tiny scratches on metal surfaces.

Claims

1. A scratch defect enhancement method based on multi-modal gradient map fusion, characterized in that: It includes the following steps: S1: Obtain the normal vector map and reflectivity map of the object to be detected based on the eight-directional photometric stereo method; S2: Construct a direction-insensitive gradient enhancement method, which includes the following steps: S21: Use the Sobel operator to perform gradient calculation, and derive the image to be enhanced from the x-axis and y-axis directions respectively; g x =(z7+2z8+z9)-(z1+2z2+z3); g y =(z3+2z6+z9)-(z1+2z4+z7); In the formula, the subscripts 1 to 9 represent a 3x3 neighborhood of the target pixel, which are arranged from left to right and from top to bottom. The target pixel corresponds to position 5 in the neighborhood, z represents the grayscale value of the position, and g x Represents the gradient of the target pixel in the x direction, g y Represents the gradient in the y direction of the pixel; S22: For g x and g y of fusion; g′ M =sgn(g x -g y )×(|g x -g y |+|g x +g y |); In the formula, g' M Represents the gradient value of any pixel in the result image; sgn represents the sign function, which is used to obtain the positive and negative signs corresponding to the pixels; S3: Based on the direction-insensitive gradient enhancement algorithm, the normal vector map is gradient enhanced in the X and Y channel directions respectively to effectively extract the scratch defects, and the following are obtained: the enhanced normal vector map in the X channel direction and the enhanced normal vector map in the Y channel direction; Performing gradient enhancement on the reflectivity map to obtain: an enhanced reflectivity map; S4: fusing the enhanced normal vector map in the X-channel direction, the enhanced normal vector map in the Y-channel direction and the enhanced reflectivity map in the gradient direction to obtain a multi-modal fusion gradient map after the corresponding scratch defect feature of the object to be detected is enhanced.

2. The scratch defect enhancement method based on multi-modal gradient map fusion according to claim 1, characterized in that: In step S1, constructing a collection device for the eight-directional photometric stereo method; The acquisition device comprises: a reference platform, a ring light source, a camera and a lens; The reference platform is placed horizontally, and the object to be detected is placed on the reference platform; the camera is arranged above the lens, the details and the lens are vertically installed directly above the reference platform, and the annular light source partition is fixed between the camera device and the reference platform; The annular light source is divided into eight directions, the intensity of each subarea light source is uniform and the center of the light source is equidistant from the center of the reference platform.

3. The scratch defect enhancement method based on multi-modal gradient map fusion according to claim 2, characterized in that: The light source orientations in the annular light source include: 0°, 45°, 90°, 135°, 180°, 225°, 270°, and 315°.

4. The scratch defect enhancement method based on multi-modal gradient map fusion according to claim 1, characterized in that: In step S1, a normal vector map and a reflectivity map of the object to be detected are obtained, which specifically includes the following steps: a1: Collect images based on light sources in eight directions to obtain illumination images in eight directions; a2: Sort the illumination images in eight directions from small to large according to the azimuth angle of the light source, with serial numbers from 1 to 8; a3: Divide the four pictures with even numbers into one group, and the four pictures with odd numbers into one group, to obtain two groups of near-field four-directional photometric stereo models; Then one of the corresponding azimuths is: 0°, 90°, 180° and 270°; Another set of corresponding azimuth feet are: 45°, 135°, 225° and 315°; a4: Construct the basic formula: Using the Lambertian surface assumption, the diffuse reflection formula of any incident light in space on the Lambertian surface is given by formula (1); The unit light source direction vector is expressed in the form of trigonometric functions of azimuth and elevation angles to obtain formula (2); Introduce the surface unit normal vector into the normal vector components p and q to be solved, and obtain formula (3); The gray value of the image and the azimuth angle of the image are related to each other, and formula (4) is obtained. THE i =And c ρn l i (1); In the formula, the subscript i represents the light source number, I represents the grayscale value of the object illumination image under a certain direction of light source; τ i Represents the angle between the light source direction vector and the surface normal of the object; θ i Indicates the pitch angle of the light source direction vector; represents the azimuth of the light source direction vector; n represents the normal vector of the object surface, p and q represent the components of the normal vector in the X and Y directions; l represents the incident direction of the light source in polar coordinates; ρ represents the reflectivity of the object surface; Ec represents the intensity of the light source; a5: Substituting the trigonometric function values ​​of the azimuth angles of the two sets of near-field four-directional photometric stereo models into formula (4), the two separate sets of photometric stereo models are derived to obtain formulas (5) and (6) respectively; Wherein, the directions of 0° and 180°, 90° and 270°, and 45° and 225° are in a positional relationship with the directions of 135° and 315°; Ec represents the intensity of the light source, l represents the incident direction of the light source in polar coordinate form; θ represents the elevation angle of the light source direction vector; a6: Using the positional relationship of the light source to construct an eliminable basis, a simplified calculation process is derived to obtain formulas (7) and (8); In the formula, I p1 ,I q1 ,I p2 and I q2 It is calculated by the grayscale value of the corresponding position of the image collected in eight directions. The trigonometric function value of the pitch angle in the light source direction is determined by the actual layout of the image acquisition device and is calculated through the light source calibration process; sum1 ,I sum2 , p1, p2, q1 and q2 are intermediate calculation results; a7: calculated based on formulas (7) and (8): p1, p2, q1 and q2; a8: Take the average value of p1, p2, q1 and q2 corresponding to the two sets of near-field four-directional photometric stereo models to obtain the normal vector components p and q to be solved; a9: The calculated normal vector components p and q are normalized to the range of [0, 255], rounded, and mapped to the three channel values ​​of the visualization color space to finally generate the normal vector map of the surface of the object to be detected; a10: based on the normal vector map of the surface of the object to be detected, the reflectivity map of the surface of the object to be detected is calculated according to formula (12); Where n represents the surface normal vector of the object.

Images