Outlier removal and reflectivity and normal generation method based on multi-angle illumination images

Abstract

The present invention discloses a method for removing outliers and generating reflectivity and normals based on multi-angle illumination images, which belongs to the fields of optics, digital image processing, self-supervised deep learning, etc. First, images under illumination from different directions are obtained; the images are read and normalized to obtain a multi-channel image array; a spatial coordinate system is established according to the position of the light source and the camera, and multiple light sources form a light source direction matrix; a neural network model is built, and a loss function is selected according to the characteristics of the outliers; each channel of the image array and the light source direction matrix are respectively input into the neural network model in pairs, and a back-propagation algorithm is used for iterative optimization to obtain a normal matrix with reflectivity scaling; the normal with reflectivity scaling is obtained by obtaining the vector modulus and normalizing it to obtain the reflectivity and normal of each channel, and the average value of the normal of each channel is the final normal. The present invention can be applied to solving the normal of an object, and can effectively reduce errors such as those caused by highlights, shadows, noise, etc.

Description

Technical Field

[0001] The present invention relates to the fields of optics, digital image processing, self-supervised deep learning, and in particular to a method for removing outliers and generating reflectivity and normals based on multi-angle illumination images. Background Art

[0002] 3D reconstruction technology is a branch of computer vision and has a wide range of applications in the fields of medicine, cultural relics protection, entertainment, etc. Photometric stereo is one of the important means of 3D reconstruction. It uses a camera with a fixed viewing angle to collect target images under different lighting conditions to solve the target object's normal. Photometric stereo was first proposed by Woodham et al. This method has three assumptions: 1. The camera projection is orthogonal; 2. The light emitted by the light source is parallel; 3. The target object is a Lambertian body. Under these three assumptions, the grayscale value of a point on the object surface in the image is It can be expressed as:

[0003] ;

[0004] in, is the reflectivity at that point, is the unit normal vector of the point, is the unit direction vector of the light source. When the grayscale value at that point and the direction of the light source are known, the reflectivity and unit normal can be solved. Under certain hardware conditions, the first two assumptions are relatively easy to meet. However, the third assumption, which addresses the inherent properties of the target object itself, is the most difficult to achieve. Problems caused by the object's non-Lambertian properties are also the most difficult to solve. For example, the reflection of a smooth object is composed of a diffuse reflection (Lambertian reflection) component and a specular reflection component. When the specular reflection component is strong, the grayscale value captured by the camera will be greatly enhanced; or the grayscale value of the occluded area may be close to zero due to occlusion shadows caused by the undulating surface of the object; or high-frequency noise caused by a malfunction of the data acquisition equipment. These situations will greatly affect the image quality, resulting in reduced accuracy of the calculated reflectivity and normal. Summary of the Invention

[0005] In order to overcome the deficiency of existing normal solution technology that is greatly affected by outliers (such as highlights, shadows, etc.), the present invention proposes a method for outlier removal and reflectivity and normal generation based on multi-angle illumination images.

[0006] In order to achieve the above object, the present invention adopts the following technical solutions:

[0007] A method for removing outliers and generating reflectivity and normals based on multi-angle illumination images includes the following steps:

[0008] S1. Acquire target images under different lighting directions;

[0009] S2, reading and normalizing the grayscale values ​​of the acquired multiple target images to obtain a normalized multi-channel image array;

[0010] S3. Establish a spatial coordinate system based on the positions of the light source and the camera. Each light source generates a light source direction vector, and multiple light sources form a light source direction matrix.

[0011] S4. Build a self-supervised learning neural network model and select a loss function based on the distribution characteristics of outliers;

[0012] S5. Input each channel of the multi-channel image array and the light source direction matrix into the self-supervised learning neural network model, and use the back-propagation algorithm to iteratively optimize to obtain the reflectivity-scaled normal matrix;

[0013] S6. Calculate the vector modulus of the normal matrix of the reflectivity scaled pixel by pixel and normalize it to obtain the reflectivity and normal of each channel, and average the normals of each channel to obtain the final normal.

[0014] Preferably, the step of obtaining the target image under lighting from different directions is: making the target object located at the center of the lighting device, the spatial positions of the incident lights are different, and at least ensuring that the spatial positions of the four incident lights are not coplanar, and using a camera with a fixed angle of view to capture the target image.

[0015] Preferably, the step of reading and normalizing the grayscale values ​​of the acquired multiple images to obtain a normalized multi-channel image array is: reading and integrating the images into an image array, and normalizing the grayscale values ​​of all pixels in the image array to be within the range of [0, 1].

[0016] Preferably, the spatial coordinate system is established according to the position of the light source and the camera, each light source generates a light source direction vector, and multiple light sources form a light source matrix. The steps are: the light source direction unit vector corresponding to each image is calculated by measuring the relative position of the light source, the target object and the camera, and the multiple images form a corresponding light source direction matrix.

[0017] Preferably, the steps of building a self-supervised learning neural network model and selecting a suitable loss function according to the distribution characteristics of the outliers are as follows: the neural network includes a linear layer and a nonlinear activation layer, etc., and the loss function is selected according to the specific situation of the outliers. For example, when the outliers are mirror reflection highlights and shadows, the mean absolute error loss is used; when the outliers are high-frequency noise, the total variation loss and the mean absolute error loss are used.

[0018] Preferably, the steps of inputting each channel of the image array and the light source direction matrix into the neural network model respectively, and using the back-propagation algorithm to iteratively optimize to obtain the reflectivity-scaled normal matrix are as follows: splitting the multi-channel image array into multiple single-channel image arrays and inputting them into the neural network in pairs with the light source matrix respectively, and then comparing the output of the neural network with the corresponding channel of the original multi-channel image array, and selecting a suitable loss function to optimize the parameters of the neural network.

[0019] Preferably, the reflectivity-scaled normal matrix is ​​obtained pixel by pixel to obtain the vector modulus and normalized to obtain the reflectivity and normal of each channel, and the normal of each channel is averaged to obtain the final normal. The steps are as follows: for the reflectivity-scaled normal matrix of each channel output by the neural network, its size should be m×n×3, where m and n are the height and width of the image respectively, and 3 represents the three-dimensional normal coordinates of the reflectivity scale, that is, this matrix contains the three-dimensional normal coordinates of all points in the image. The modulus is taken for each three-dimensional coordinate to obtain the reflectivity, and each three-dimensional coordinate is normalized to obtain the normal. The reflectivity of all channels is spliced ​​to obtain the multi-channel reflectivity, and the normal of all channels is averaged to obtain the final normal. The normal can be normalized to the range of [0, 1] for easy visualization.

[0020] The present invention removes outliers in multi-illumination images through the excellent nonlinear fitting characteristics of neural networks and the characteristics of outlier distribution, extracts potential Lambertian reflection components, and then solves high-precision normals.

[0021] Compared with the prior art, the present invention has the following beneficial effects:

[0022] 1. The normals solved by the present invention are less affected by outliers such as highlights and shadows that are difficult to resolve;

[0023] 2. The present invention can reduce the influence of high-frequency noise in the image, and the solved normal map is smoother. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 The figure is a flow chart of a method for removing outliers and generating reflectivity and normals based on multi-angle illumination images according to an embodiment of the present invention.

[0025] Figure 2 Schematic diagram of the results; from left to right and from top to bottom are: (a) outlier image, where the outliers are mainly shadows; (b) the effect after removing shadows using the method of the present invention; (c) the normal solved by the method; (d) the reflectivity solved by the method of the present invention; (e) the normal solved by the least squares method; (f) the reflectivity solved by the least squares method.

[0026] Figure 3Another result diagram is shown in FIG. 1 , where, from left to right and from top to bottom, they are: (a) an outlier image, where the outliers are mainly shadows, highlights, and noise; (b) the effect after removing outliers using the method of the present invention; (c) the normal solved by the method of the present invention; (d) the reflectivity solved by the method of the present invention; (e) the normal solved by the least squares method; and (f) the reflectivity solved by the least squares method. DETAILED DESCRIPTION

[0027] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only intended to illustrate the present invention and are not intended to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below may be combined with each other as long as they do not conflict with each other.

[0028] Example 1:

[0029] like Figure 1 As shown, a method for removing outliers and generating reflectivity and normals based on a multi-angle illumination image according to an embodiment of the present invention includes the following steps:

[0030] S1. Place the target object at the center of the lighting device. Ensure that the spatial positions of the incident lights are different and that the spatial positions of at least four incident lights are not coplanar. Use a camera with a fixed viewing angle to capture the target image.

[0031] S2. Read and integrate the image into an image array, and normalize the grayscale values ​​of all pixels in the image array to the range of [0, 1].

[0032] S3. By measuring the relative positions of the light source, the target object, and the camera, the light source direction unit vector corresponding to each image is calculated, and multiple images form a corresponding light source direction matrix.

[0033] The angle between the projection of the line connecting the light source and the target object on the horizontal plane and the line connecting the camera and the target object is recorded as the azimuth The angle between the projection of the line connecting the light source and the target object on the vertical plane and the line connecting the camera and the target object is recorded as the altitude angle ; The unnormalized light source direction vector is recorded as , Three elements of and azimuth and altitude angle The relationship is:

[0034] ;

[0035] ;

[0036] After the relative relationship is obtained, the normalized light source direction unit vector for:

[0037] ;

[0038] in, represent The mold length.

[0039] S4. Build a self-supervised learning neural network model and select an appropriate loss function based on the distribution characteristics of outliers;

[0040] exist Figure 3 The neural network model is applied in the process. The neural network model consists of a linear layer and a nonlinear activation layer, and the weight matrix is ​​initialized arbitrarily. And enter the light source direction matrix , the output is the reflectivity-scaled normal matrix .for Figure 3 , the outliers are shadows, highlights and high-frequency noise. Due to the sparsity of these outliers and the potential Lambertian component matrix Smoothness, the loss function The definition is as follows:

[0041] ;

[0042] in, , , represents the observed value, Represents the difference between the potential Lambertian component and the observed value, that is, the L1 norm of the outlier. The L1 norm can produce a sparse solution that conforms to the distribution characteristics of the outlier. It is the total variation of the potential Lambertian component, which promotes the smoothness of the Lambertian component and conforms to the surface characteristics of the target object. It is a trade-off parameter and can be adjusted as needed.

[0043] S5. Input each channel of the image array and the light source direction matrix into the neural network model in pairs, and use the back propagation algorithm to iteratively optimize to obtain the normal matrix of reflectivity scaling;

[0044] For the data acquisition device, if it has multiple spectral channels, these spectral channels are separated and input into the neural network model in S4 in pairs with the light source matrix respectively. The normal matrix of the reflectivity scale of each channel is obtained through the iterative back propagation algorithm.

[0045] S6. Calculate the vector modulus of the normal matrix of the reflectivity scaled pixel by pixel and normalize it to obtain the reflectivity and normal of each channel. The normal of each channel is averaged to obtain the final normal.

[0046] For each channel, the reflectivity matrix of that channel It is obtained by the following formula:

[0047] ;

[0048] in, Express The modulus along the normal dimension. Unit normal matrix It is obtained by the following formula:

[0049] ;

[0050] The reflectivity matrix of each channel The multispectral reflectance matrix is ​​obtained by splicing along the channel dimension, and the unit normal matrix of each channel is The final normal matrix is ​​obtained by averaging along the channel dimension. ,because The value range of is within [-1,1], and digital images cannot display negative numbers, so if you directly Visualizing it as a color digital image will result in some information loss. It can be scaled to the range [0, 1] for easier visualization, for example:

[0051] ;

[0052] Among them, the left arrow This is an assignment operation. is a size and unit normal matrix A consistent three-dimensional array with a value range of [0,1] that meets the display range of digital images. Visualization as a digital image does not result in loss of information. After scaling, The XYZ coordinates are assigned to the values ​​of the RGB channels to obtain the visualization of the final normal of the target object.

[0053] like Figure 2 The following is a schematic diagram of the results; from left to right and from top to bottom: (a) outlier image, where outliers are mainly shadows; (b) the effect after removing shadows using the method of the present invention; (c) the normal solved by the method; (d) the reflectivity solved by the method of the present invention; (e) the normal solved by the least squares method; (f) the reflectivity solved by the least squares method. Figure 2 It can be seen that the shadow under the sphere is well removed.

[0054] like Figure 3 Another result diagram is shown, where from left to right and from top to bottom are: (a) outlier image, where outliers are mainly shadows, highlights, and noise; (b) the effect after removing outliers using the method of the present invention; (c) the normal solved by the method of the present invention; (d) the reflectivity solved by the method of the present invention; (e) the normal solved by the least squares method; (f) the reflectivity solved by the least squares method. Figure 3 It can be seen that the shadow under the cone, the specular reflection highlight of the stool and the noise are well removed. In addition, the reflectivity obtained by the least squares method is greatly affected by the specular reflection, while the method of the present invention is less affected by the specular reflection.

[0055] The above embodiments are only preferred embodiments of the present invention and cannot be used to limit the scope of protection of the present invention. Any non-substantial changes and replacements made by technicians in this field on the basis of the present invention fall within the scope of protection required by the present invention.

Claims

1. A method for removing outliers and generating reflectivity and normals based on multi-angle illumination images, characterized in that: The steps include: S1. Acquire target images under different lighting directions; S2, reading and normalizing the grayscale values ​​of the acquired multiple target images to obtain a normalized multi-channel image array; S3. Establish a spatial coordinate system based on the positions of the light source and the camera. Each light source generates a light source direction vector, and multiple light sources form a light source direction matrix. S4. Build a self-supervised learning neural network model and select a loss function based on the distribution characteristics of outliers; S5. Split the multi-channel image array into multiple single-channel image arrays and input them into a self-supervised learning neural network model in pairs with the light source matrix. Use the back-propagation algorithm to iteratively optimize and obtain the normal matrix with reflectivity scaling. S6. Calculate the vector modulus of the normal matrix of the reflectivity scaled pixel by pixel and normalize it to obtain the reflectivity and normal of each channel, and average the normals of each channel to obtain the final normal.

2. The method for removing outliers and generating reflectivity and normal based on multi-angle illumination images according to claim 1, characterized in that: Said S1 comprises: The target object is located at the center of the lighting device. The spatial positions of the incident lights are different, and at least the spatial positions of the four incident lights are not coplanar. A camera with a fixed viewing angle is used to capture the target image.

3. The method for removing outliers and generating reflectivity and normal based on multi-angle illumination images according to claim 1, characterized in that: The S2 includes: Read and integrate the image into an image array, and normalize the grayscale values ​​of all pixels in the image array to the range of [0, 1].

4. The method for removing outliers and generating reflectivity and normal based on multi-angle illumination images according to claim 1, characterized in that: The S3 includes: The light source direction unit vector corresponding to each image is calculated by measuring the relative positions of the light source, the target object and the camera, and multiple images form the corresponding light source direction matrix.

5. The method for removing outliers and generating reflectivity and normal based on multi-angle illumination images according to claim 1, characterized in that: In S4, the self-supervised learning neural network model includes a linear layer and a nonlinear activation layer, and the loss function is selected according to the specific situation of the outlier.

6. The method for removing outliers and generating reflectivity and normal based on multi-angle illumination images according to claim 5, characterized in that: When the outliers are specular highlights or shadows, the mean absolute error loss is used; when the outliers are high-frequency noise, the total variation loss and the mean absolute error loss are used.

7. The method for removing outliers and generating reflectivity and normal based on multi-angle illumination images according to claim 1, characterized in that: The S5 includes: Compare the output of the neural network to the corresponding channels of the original multi-channel image array.

8. The method for removing outliers and generating reflectivity and normal based on multi-angle illumination images according to claim 7, characterized in that: Choose a loss function to optimize the parameters of the neural network.

9. The method for removing outliers and generating reflectivity and normal based on multi-angle illumination images according to claim 1, characterized in that: The S6 includes: For each channel of the neural network output, the reflectivity-scaled normal matrix has a size of m×n×3, where m and n are the height and width of the image, respectively, and 3 represents the three-dimensional normal coordinates of the reflectivity-scaled normal. The m×n×3 matrix contains the three-dimensional normal coordinates of all points in the image.

10. The method for removing outliers and generating reflectivity and normal based on multi-angle illumination images according to claim 9, characterized in that: The modulus length of each three-dimensional coordinate is taken to obtain the reflectivity, and each three-dimensional coordinate is normalized to obtain the normal line; The reflectivity of all channels is spliced ​​to obtain the multi-channel reflectivity, and the normals of all channels are averaged to obtain the final normal.

Images