A method for extracting microstructural features of weak texture surfaces based on a single visible light image

Abstract

The present invention discloses a method for extracting microstructural features of a weak texture surface based on a single visible light image, belonging to the technical field of image processing. In a natural visible light scene, the present invention obtains the micro-geometric concave and convex features of the weak texture surface through steps such as photographing the weak texture surface, gray-scale adjustment, K-Prototypes clustering adaptive grid division, and Gabor kernel extraction of high-frequency detail features, solves the problem that the existing methods cannot extract the inherent features of the weak texture surface area or have low extraction accuracy, achieves the purpose of feature recognition and feature extraction of the micro-structure of the weak texture surface, and can be applied to important fields such as object recognition, VR, AR, surface flaw detection, and medical diagnosis.

Description

Technical Field

[0001] The present invention relates to a method for extracting microstructure features of a weak texture surface based on a single visible light image, and belongs to the technical field of image processing. Background Art

[0002] Feature extraction is an insurmountable step in computer vision for realizing scene understanding and cognitive tasks. Although feature extraction is an old topic and many scholars at home and abroad have made great contributions in this field, the recognition and extraction of features in the weak texture surface area of visible light images are still a major difficulty. The weak texture surface area refers to the area where the visual effects of feature changes such as color, light and shade, and lines are not obvious or invisible. There are a large number of weak texture surface areas in daily life, such as solid color desktops, floors, and walls. Due to the mutual influence of many complex factors, including the real-time changes of the incident angle, reflection angle, and observation angle of light, the interaction between the light wave frequency and intensity and the object surface, and the anisotropy of the micro-geometric structure of the object surface, it is difficult to extract its surface features.

[0003] Due to the rapidity and universality of RGB image acquisition, many region texture feature extraction methods based on such images have emerged. The methods for extracting features of the weak texture surface area of visible light images can be generally summarized into two categories: region direct feature extraction methods and region indirect feature extraction methods. The region direct feature extraction methods can be further divided into traditional feature extraction methods and deep learning methods, and the region indirect feature extraction methods include depth map methods and peripheral assistance methods.

[0004] Traditional feature extraction methods use methods such as mathematical transformation, statistics, and template matching to try to find the mapping relationship between pixels and features, so as to establish feature descriptors. Representative methods include Scale-Invariant Feature Transform (SIFT), Oriented Fast and Rotated BRIEF (ORB), wavelet transform, histogram of gradients, random field method, etc. For the feature extraction of general macroscopic scene objects, traditional methods have been able to effectively solve and have been widely used. However, for the weak texture surface area, since traditional methods rely on the solution strategies of gray difference or gradient, and the gray difference or gradient difference of the weak texture surface is small, when directly solving, it will fall into local extrema and cannot obtain a solution that meets the actual constraint conditions, and cannot accurately calculate to the level that can reflect the microscopic differences, thus resulting in the ineffectiveness of such methods.

[0005] Deep learning methods (such as Cimpoi, Mircea. Deep Filter Banks for Texture Recognition, Description, and Segmentation [J]. International Journal of Computer Vision, 2016, 118(1): 65-94; Mustafa. Ground Texture Classification with Deep Learning [C]. Proc. of IEEE Signal Processing and Communications Applications Conference, 2018: 1-4), especially the Convolutional Neural Network (CNN), convolve the kernel over the entire region of the input image with a small stride through multiple convolutional layers to obtain the feature vector of the texture. Although deep learning methods can learn hidden texture features that cannot be defined by humans, they are highly dependent on sample training. Since the features of the weak texture surface area are not obvious or even unobservable, it is extremely difficult to manually label them, which greatly limits the accuracy and generalization of machine learning.

[0006] Depth map methods (such as Q. Luming, L. Honggao, L. Jieqing, et al. Feature Fusion of ICP-AES, UV-Vis and FT-MIR for Origin Traceability of Boletus delis Mushrooms in Combination with Chemometrics [J]. Sensors, 2018, 18(1): 241) are based on directly measuring the distance between the object surface and the sensor (such as TOF, ultrasonic, laser, etc.) to obtain an approximate topography of the object surface. However, this type of method is only applicable to the extraction of features of relatively macroscopic objects or scene regions and is not applicable to weak texture surface regions that can only be described by minute concavity and convexity changes. Coupled with the interference of noise and light, its accuracy is low or even ineffective.

[0007] The peripheral assistance method (such as Tombari, Federico, A. Franchi, et al. BOLD Features to Detect Texture-Less Objects [C]. Proc. of IEEE International Conference on Computer Vision (ICCV), 2013: 1265-72) uses auxiliary objects such as shadows, labels, and peripheral environmental elements to extract relevant features. Essentially, this type of method uses the peripheral attributes of the surface area to replace the features of the target area.

[0008] It can be seen that the depth map method and the peripheral assistance method are equivalent to describing the features of the image by relying on other factors related to the weakly textured surface, rather than directly extracting the features of this area. These other factors are easily affected by various interferences, resulting in inaccurate feature descriptions. Therefore, the accuracy of the weakly textured surface features extracted using the depth map method and the peripheral assistance method is greatly affected. Summary of the Invention

[0009] To solve the problem of low accuracy of current weakly textured surface feature extraction methods, the present invention provides a method for extracting microstructural features of weakly textured surfaces based on a single visible light image. The technical solution is as follows:

[0010] The first object of the present invention is to provide a method for extracting microstructural features of weakly textured surfaces based on a single visible light image, the method comprising:

[0011] Step 1: In a natural light environment without strong light illumination, use a photographing device to photograph the weakly textured surface to obtain an original image, wherein the lens of the photographing device is perpendicular to the weakly textured surface;

[0012] Step 2: Read the camera sensor EXIF information carried in the original image, and solve the camera response curves of the three RGB channels respectively according to the relationship between the image pixel values, the irradiance of the object surface, and the exposure time of the camera sensor;

[0013] Step 3: Calculate the distance between the photographed weakly textured surface and the center of the lens through the lens focal length;

[0014] Step 4: Adjust the grayscale of the original image to a preset range to obtain a correctly exposed image;

[0015] Step 5: Perform K-Prototypes clustering adaptive grid division according to the grayscale of the correctly exposed image, extract Gabor features for each grid, and solve the surface roughness of each grid;

[0016] Step 6: Solve the surface normal vector of each grid according to the Gabor kernel form of Kirchhoff's equation;

[0017] Step 7: Construct a feature descriptor for each grid as the feature vector of a feature point, and the set of all feature points constitutes the surface concavity and convexity features of the correctly exposed image.

[0018] Optionally, the camera response curve in step 2 reflects the relationship between the image pixel value C, the irradiance R of the object surface, and the exposure time t of the camera sensor, including:

[0019]

[0020] where k is the curve correction coefficient.

[0021] Optionally, the formula for calculating the distance between the captured weakly textured surface and the lens center in step 3 is:

[0022]

[0023] where L is the distance from the surface to the lens center, F is the focal length of the lens, and V represents the distance from the lens center to the camera sensor.

[0024] Optionally, the process of gray level adjustment in step 4 includes:

[0025] Set the average gray level of the original image I 0 as the reference value θ, take the gray level value of the image pixel point x as Gray(x), and the value of Gray(x) satisfies the preset interval range centered on θ. Through threshold constraint, obtain the correctly exposed image I.

[0026] Optionally, the process of K-Prototypes clustering adaptive grid division according to the gray level of the correctly exposed image in step 5 includes:

[0027] Step 51: Randomly select k pixel points in the correctly exposed image, and use the gray level values of the k pixel points as the initial reference prototypes;

[0028] Step 52: Calculate the Euclidean distance between other pixel points in the image and the k initial reference prototypes, and divide the pixel points into the category corresponding to the initial reference prototype closest to it;

[0029] Step 53: After all pixel points are divided, reset the mean value of each current pixel point category as the initial reference prototype of this category; if the difference between the mean values of two category pixel points is less than the preset difference Δ, merge them into one category;

[0030] Step 54: Iterate steps 52 and 53 until no pixel point changes its category;

[0031] Step 55: After clustering, N categories are generated, where N ≤ k. The correctly exposed images are automatically divided into N grid regions according to the classification results. Each grid contains the category center pixel and the remaining pixels of the same category around the center.

[0032] Optionally, the process of extracting Gabor features in step 5 includes:

[0033] Perform Fourier transform on the image pixel points in each grid, and then filter them using the Gabor kernel function:

[0034] r(x) ≈ g(x; m, σ, a)

[0035] where r(x) is the reflectivity of the image pixel point x, and also represents the surface roughness of each pixel point x. g(x; m, σ, a) is the two-dimensional Gabor kernel function, that is:

[0036] g(x; m, σ, a) = G 2D (x; m, σ) · e -i2π(a·x)

[0037] where G 2D (x; m, σ) is the two-dimensional surface distribution function, m is the category center of the grid where the pixel point x belongs, σ is the scale parameter, and a is the plane light wave parameter related to the two-dimensional surface height.

[0038] Optionally, the Gabor kernel form of the Kirchhoff equation in step 6 is:

[0039]

[0040] where ξ is the bidirectional reflection distribution function of the weak texture surface, ω i and ω o are the incident light vector and the reflected light vector, ψ is the vector sum of the projections of ω i and ω o on the two-dimensional plane, λ is the wavelength of light, and S is the area of the correctly exposed image I.

[0041] Optionally, the feature descriptor in step 7 is a multi-dimensional feature vector including the RGB color, roughness value, surface height value, and surface normal vector of the image pixel point x.

[0042] Optionally, the photographing device includes: a consumer-grade mobile phone.

[0043] The second object of the present invention is to provide an image classification method, which uses the above-mentioned method for extracting weak texture surface microstructure features based on a single visible light image to extract features, and then classifies the image according to the extracted features.

[0044] The beneficial effects of the present invention are as follows:

[0045] Through gray-scale adjustment, the present invention minimizes the feature extraction error that may be caused by light changes to the greatest extent; through K-Prototypes clustering and adaptive grid division, pixel points with similar features are grouped into one category, more significantly highlighting the potential weak texture surface features; through Gabor kernel extraction and using Fourier transform, the details of the weak texture surface are captured from the high-frequency space in the complex frequency domain, obtaining the weak texture surface micro-geometric concavo-convex features, solving the problem that the existing methods cannot extract the inherent features of the weak texture surface area or the extraction accuracy is low, and achieving the purpose of feature recognition and feature extraction of the microstructure of the weak texture surface. Description of the Drawings

[0046] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following will briefly introduce the drawings required for the description of the embodiments. Obviously, the following drawings are only some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can be obtained based on these drawings.

[0047] Figure 1 It is a flowchart of the steps of a method for extracting weak texture surface microstructure features based on a single visible light image according to the second embodiment of the present invention.

[0048] Figure 2 It is a picture of a white wall obtained in the second embodiment of the present invention.

[0049] Figure 3 It is a graph of the response curves of the RGB three channels of a camera and the Gamma 2.2 standard reference curve according to the second embodiment of the present invention.

[0050] Figure 4 It is a surface normal vector diagram according to the second embodiment of the present invention.

[0051] Figure 5 It is a surface concavo-convex feature diagram according to the second embodiment of the present invention. Detailed Embodiments

[0052] To make the purpose, technical solutions and advantages of the present invention clearer, the following will further describe the embodiments of the present invention in detail with reference to the drawings.

[0053] Embodiment 1:

[0054] This embodiment provides a method for extracting microstructural features of a weak texture surface based on a single visible light image. The method includes:

[0055] Step 1: In a natural light environment without strong light illumination, use a photographing device to photograph the weak texture surface to obtain an original image. The lens of the photographing device is perpendicular to the weak texture surface;

[0056] Step 2: Read the EXIF information of the camera sensor carried in the original image, and respectively solve the camera response curves of the three RGB channels according to the relationship between the image pixel value, the irradiance of the object surface, and the exposure time of the camera sensor;

[0057] Step 3: Calculate the distance between the photographed weak texture surface and the center of the lens through the lens focal length;

[0058] Step 4: Adjust the grayscale of the original image to a preset range to obtain a correctly exposed image;

[0059] Step 5: Perform K-Prototypes clustering adaptive grid division according to the grayscale of the correctly exposed image, extract Gabor features for each grid, and solve the surface roughness of each grid;

[0060] Step 6: Solve the surface normal vector of each grid according to the Gabor kernel form of the Kirchhoff equation;

[0061] Step 7: Construct a feature descriptor for each grid as the feature vector of a feature point, and the set of all feature points constitutes the surface concavo-convex features of the correctly exposed image.

[0062] Embodiment 2:

[0063] This embodiment provides a method for extracting microstructural features of a weak texture surface based on a single visible light image. Refer to Figure 1 , the method includes:

[0064] Step 1: In a natural light environment without strong light illumination, use a consumer-grade mobile phone to vertically photograph the white wall surface to obtain an original image I 0 , as Figure 2 shown, are two original images obtained in this embodiment.

[0065] Step 2: Read the EXIF (Exchangeable Image File Format) information of the camera sensor carried in the original image I 0 , and respectively solve the camera response curves of the three RGB channels according to the relationship between the image pixel value C, the irradiance R of the object surface, and the exposure time t of the camera sensor.

[0066] The EXIF information includes the horizontal resolution of the sensor, the vertical resolution of the sensor, the image width, the image height, the exposure time, and the lens focal length.

[0067] The relationship between the image pixel value C, the irradiance R of the object surface, and the exposure time t of the camera sensor is:

[0068]

[0069] where k is the curve correction coefficient, and here k = 0.5 2.2 . The camera response curve is as Figure 3 shown.

[0070] Step 3: Calculate the distance L from the weakly textured surface being photographed to the center of the lens through the lens focal length F.

[0071] The calculation method is given by the Gaussian formula, specifically:

[0072]

[0073] where V represents the distance from the center of the lens to the camera sensor. Since V << L, in actual calculation it can be ignored, that is

[0074] Step 4: Adjust the grayscale of the original image to obtain an image with correct exposure.

[0075] The grayscale adjustment process is to set the average grayscale of the entire image as the reference value θ, and take the grayscale value of the image pixel point x as Gray(x). The value of Gray(x) should satisfy within the range of ±90% centered on θ:

[0076]

[0077] When Gray(x) is lower than the minimum value, the pixel point is underexposed and compensated. When Gray(x) is higher than the maximum value, the pixel point is overexposed and attenuated. Through the constraint of the threshold θ, an image I with correct exposure is obtained.

[0078] Step 5: Perform K-Prototypes clustering adaptive grid division according to the grayscale, extract Gabor features for each grid, and solve the surface roughness r of each grid.

[0079] The K-Prototypes clustering includes the following steps:

[0080] (1) Randomly select 512 pixel points and use their grayscale values as the initial reference prototypes;

[0081] (2) For each pixel point in the image, calculate the Euclidean distance between other pixel points and 512 initial reference prototypes, and divide this pixel point into the category corresponding to the center point with the closest distance to it;

[0082] (3) After all pixel points are divided, reset the mean value of each current pixel point category as the initial reference prototype of this category; if the difference between the mean values of two category pixel points is less than the preset difference Δ = 0.01, then merge them into one category;

[0083] (4) Iterate steps (2) and (3) until no samples change their categories.

[0084] The adaptive grid division refers to automatically dividing the entire image into N (N = 128) grid regions according to the number of categories generated by K-Prototypes clustering. Each grid contains the category center pixel point and the remaining pixel points of the same category around this center.

[0085] The Gabor feature specifically refers to performing a Fourier transform on the image pixel points in each grid, and then using a Gabor kernel function to filter them:

[0086] r(x) ≈ g(x; m, σ, a)

[0087] where r(x) is the reflectivity of the image pixel point x, and at the same time represents the surface roughness of each pixel point x (the smoother the surface, the larger the value of the reflectivity, and vice versa), and g(x; m, σ, a) is a two-dimensional Gabor kernel function, that is:

[0088] g(x; m, σ, a) = G 2D (x; m, σ) · e -i2π(a·x )

[0089] where is a normalized two-dimensional isotropic Gaussian distribution function, m is the category center of the grid to which the pixel point x belongs, is the scale parameter, l = |x max - x min | is the grid width, is the plane light wave parameter, H′(x) is the differential of the surface height point represented by the pixel point x, λ is the wavelength of light, and for the RGB three primary colors, 700nm, 546nm, and 436nm are taken respectively.

[0090] Step 6: Solve the surface normal vector of each grid according to the Gabor kernel form of the Kirchhoff equation, see Figure 4 .

[0091] The Gabor kernel form of the Kirchhoff equation is:

[0092]

[0093] Among them is the bidirectional reflectance distribution function of the weak texture surface, approximately reflecting the optical reflection characteristics of the surface, ω i and ω o are the incident light vector and the reflected light vector, ψ is the vector sum of the projections of ω i and ω o on the two-dimensional plane (i.e., the camera sensor), λ is the wavelength of light, F = F 0 +(1 - F 0 )(1 - (n, ω 0 )) 5 is the Fresnel reflection equation, F 0 is the base reflection coefficient of the material, and here the reflection coefficient F 0 of calcium oxide, the main component of the white wall, is taken as 0.43, n is the surface normal vector, S is the area of the entire image I, which is obtained by multiplying the image width and image height in the EXIF information.

[0094] Step 7: Construct the feature descriptor of each grid as the feature vector of a feature point. The set of all feature points constitutes the surface concavity and convexity features of the image I, see Figure 5 .

[0095] The feature descriptor is a 256-dimensional feature vector containing the RGB color, roughness value, surface height value, and surface normal vector of the image pixel point x.

[0096] Some steps in the embodiments of the present invention can be implemented by software, and the corresponding software program can be stored in a readable storage medium, such as an optical disc or a hard disk, etc.

[0097] The above are only the preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for extracting microstructural features of a weak texture surface based on a single visible light image, characterized in that, the method includes: Step 1: In a natural light environment without strong light illumination, use a photographing device to photograph the weak texture surface to obtain an original image, and the lens of the photographing device is perpendicular to the weak texture surface; Step 2: Read the camera sensor EXIF information carried in the original image, and solve the camera response curves of the three RGB channels respectively according to the relationship between the image pixel value, the object surface irradiance, and the camera sensor exposure time; Step 3: Calculate the distance between the photographed weak texture surface and the center of the lens through the lens focal length; Step 4: Adjust the gray level of the original image to within a preset range to obtain a correctly exposed image; Step 5: Perform K-Prototypes clustering adaptive grid division according to the gray level of the correctly exposed image, extract Gabor features for each grid, and solve the surface roughness of each grid; Step 6: Solve the surface normal vector of each grid according to the Gabor kernel form of Kirchhoff's equation; Step 7: Construct a feature descriptor for each grid as the feature vector of a feature point, and the set of all feature points constitutes the surface concavo-convex features of the correctly exposed image.

2. The method for extracting microstructural features of a weak texture surface according to claim 1, characterized in that, the camera response curve in step 2 reflects the relationship between the image pixel value C, the object surface irradiance R, and the camera sensor exposure time t, including: where k is the curve correction coefficient.

3. The method for extracting microstructural features of a weak texture surface according to claim 1, characterized in that, the formula for calculating the distance between the photographed weak texture surface and the center of the lens in step 3 is: where L is the distance between the surface and the center of the lens, F is the lens focal length, and V represents the distance between the center of the lens and the camera sensor.

4. The method for extracting microstructural features of a weak texture surface according to claim 1, characterized in that, the process of gray level adjustment in step 4 includes: Set the average gray level of the original image I 0 as the reference value θ. Take the gray level value of the image pixel point x as Gray(x). The value of Gray(x) satisfies the preset interval range centered on θ. Through threshold constraint, the correctly exposed image I is obtained.

5. The method for extracting microstructural features of a weak texture surface according to claim 1, characterized in that, the process of performing K-Prototypes clustering adaptive grid division according to the gray level of the correctly exposed image in step 5 includes: Step 51: Randomly select k pixel points in the correctly exposed image, and use the gray level values of the k pixel points as the initial reference prototypes; Step 52: Calculate the Euclidean distances between other pixel points in the image and the k initial reference prototypes, and divide the pixel points into the categories corresponding to the initial reference prototypes that are closest to them; Step 53: After all pixel points are divided, reset the mean value of each current pixel point category as the initial reference prototype of this category; if the difference between the mean values of the pixel points in two categories is less than the preset difference Δ, then merge them into one category; Step 54: Iterate steps 52 and 53 until no pixel point changes its category; Step 55: After clustering, N categories are generated, where N ≤ k. The correctly exposed images are automatically divided into N grid regions according to the classification results. Each grid contains the category center pixel and the remaining same-category pixels surrounding the center.

6. The method for extracting weak texture surface microstructure features according to claim 1, characterized in that the process of extracting Gabor features in step 5 includes: Performing Fourier transform on the image pixel points in each grid, and then filtering them using the Gabor kernel function: r(x)≈g(x;m,σ,a) where r(x) is the reflectivity of the image pixel point x, and also represents the surface roughness of each pixel point x. g(x;m,σ,a) is a two-dimensional Gabor kernel function, that is: g(x;m,σ,a) = G 2D (x;m,σ)·e -i2π(a·x) where G 2D (x; m, σ) is a two-dimensional surface distribution function, m is the class center of the grid to which the pixel point x belongs, σ is the scale parameter, and a is the plane light wave parameter related to the two-dimensional surface height.

7. The method for extracting weak texture surface microstructure features according to claim 6, characterized in that the Gabor kernel form of the Kirchhoff equation in step 6 is: Among them, ξ is the bidirectional reflectance distribution function of the weak texture surface, ω i and ω o are the incident light vector and the reflected light vector, ψ is the vector sum of the projections of ω i and ω o on the two-dimensional plane, λ is the wavelength of light, and S is the area of the correctly exposed image I.

8. The method for extracting weak texture surface microstructure features according to claim 1, characterized in that the feature descriptor in step 7 is a multi-dimensional feature vector including the RGB color, roughness value, surface height value, and surface normal vector of the image pixel point x.

9. The method for extracting weak texture surface microstructure features according to claim 1, characterized in that the photographing device includes: a consumer-grade mobile phone.

10. An image classification method, characterized in that the image classification method uses the method for extracting weak texture surface microstructure features based on a single visible light image according to any one of claims 1-9 for feature extraction, and then classifies the image according to the extracted features.

Images