Non-lambertian surface photometric stereo model and method based on three-dimensional convolution

Abstract

The present application belongs to the technical field of image processing, and particularly relates to a non-Lambertian surface photometric stereo model and method based on three-dimensional convolution. The present application discloses a non-Lambertian surface photometric stereo model based on three-dimensional convolution, which comprises an information fusion layer, an inter-frame feature extraction layer, a spatial feature extraction layer, a maximum pooling layer and a regression layer. The information fusion layer is used for fusing each image and corresponding light source information, so as to ensure one-to-one correspondence between the image and the light source in subsequent processing. The inter-frame feature extraction layer is used for extracting inter-frame information, obtaining information between input image frames, and estimating the normal map. The spatial feature extraction layer extracts structural information inside a single image for normal map recovery. The maximum pooling layer is used for dimension reduction, removal of redundant information, compression of features, simplification of network complexity, reduction of calculation amount and reduction of memory consumption.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of image processing, and particularly relates to a non-Lambertian surface photometric stereo model and method based on three-dimensional convolution. BACKGROUND

[0002] In recent years, with the development of 3D related technologies, three-dimensional digitization technology of non-rigid bodies has attracted extensive attention in the industry. Realizing an efficient, high-precision and inexpensive reconstruction method has been a research focus in the field of three-dimensional digitization.

[0003] Existing methods are all separately using spatial information or inter-frame information to recover the normal vector. Therefore, a multi-dimensional information fusion method is proposed to simultaneously use spatial information and inter-frame information to solve the normal vector, which has good robustness to abnormal value areas (high light and shadow) and can also retain rich texture information. SUMMARY

[0004] The first aspect of the application aims to provide a method for obtaining a non-Lambertian surface three-dimensional image based on three-dimensional convolution.

[0005] To achieve the above technical purposes, the technical solutions adopted by the application are as follows:

[0006] The non-Lambertian surface photometric stereo model based on three-dimensional convolution comprises an information fusion layer, an inter-frame information extraction layer, a spatial information extraction layer, a maximum pooling layer and a regression layer.

[0007] The information fusion layer is used to fuse each image and its corresponding light source information, so as to ensure one-to-one correspondence between the image and the light source in subsequent processing.

[0008] The inter-frame information extraction layer is used to extract inter-frame information to obtain information between input image frames for normal map estimation.

[0009] The spatial information extraction layer extracts the structure information inside a single image for normal map recovery.

[0010] The maximum pooling layer is used for dimension reduction, removal of redundant information, compression of features, simplification of network complexity, reduction of calculation amount and reduction of memory consumption.

[0011] As a preferred scheme of the application, the information fusion layer comprises a three-dimensional convolution layer C1, followed by a LeakyReLU activation function, and the LeakyReLU activation function has an output value dropout ratio of 0.2.

[0012] As a preferred scheme of the present application, the inter-frame information extraction layer includes three three-dimensional convolution layers, which are three-dimensional convolution layer C2, three-dimensional convolution layer C3 and three-dimensional convolution layer C4 respectively, and each three-dimensional convolution layer is followed by a LeakyReLU activation function, and each LeakyReLU activation function has an output value dropout ratio of 0.2.

[0013] As a preferred scheme of the present application, the spatial information extraction layer includes three three-dimensional convolution layers, which are three-dimensional convolution layer C5, three-dimensional convolution layer C6 and three-dimensional convolution layer C7 respectively, and each three-dimensional convolution layer is followed by a LeakyReLU activation function.

[0014] As a preferred scheme of the present application, the regression layer includes three two-dimensional convolution layers, which are two-dimensional convolution layer C8, two-dimensional convolution layer C9 and two-dimensional convolution layer C10 respectively, two-dimensional convolution layer C8 and two-dimensional convolution layer C9 are followed by a LeakyReLU activation function, and two-dimensional convolution layer C10 is followed by an L2 normalization correction function.

[0015] The second aspect of the present application provides a data processing system for reconstructing three-dimensional surface information of a workpiece to be measured from multiple images illuminated from different angles.

[0016] A data processing system includes a data processor, a graphics processor and a data storage, the data processor reads images from the data storage and inputs the images into the graphics processor, the graphics processor is preset with an image recognition network, the image recognition network includes an image input, a feature extractor for obtaining effective features of the input image, a maximum pooling layer and a normal solution unit; characterized in that: the image is input into an initial fusion module, the initial fusion module makes the current image and its illumination information one-to-one correspondence, the image fused with the illumination information output by the initial fusion module is input into the feature extractor, the feature extractor includes an inter-frame information extractor and a spatial information extractor, both the inter-frame information extractor and the spatial information extractor use a three-dimensional convolution kernel, wherein the inter-frame information occupies one dimension and the spatial information occupies two dimensions; the input of the inter-frame information extractor is the output of the initial fusion module, and the input of the spatial information extractor is the output of the inter-frame information extractor; the output of the spatial information extractor is used as the input of the maximum pooling layer.

[0017] Further, the normal solution unit first performs a convolution operation on the output of the maximum pooling layer, and then performs L2 regularization, and the output result of the regularization is a normal map.

[0018] Further, the last convolution operation of the normal solution unit uses 3 channels.

[0019] Further, other convolution operations of the image recognition network use 128 channels.

[0020] Further, the step of fusing the spatial information and the inter-frame information of the object image to be detected comprises:

[0021] The formula of the information fusion module is established:

[0022] wherein, IL input represents the input of the formula, IL input includes each image obtained in each shooting task and the light source intensity and the light source direction, F0 represents the output of the formula, Con 6×1×1 represents a convolution kernel with a size of 6x1x1, Con 6×1×1 (IL input ) represents a convolution operation on IL input , and σ represents a leaky Relu activation function, σ(Con 6×1×1 (IL input ) represents an activation operation on Con 6×1×1 (IL input ). represents a dropout operation, represents a dropout operation on σ(Con 6×1×1 (IL input) ). The convolution operation, the activation operation and the dropout operation belong to the conventional functions or conventional operations of deep learning.

[0023] Each image has three channels of R, G and B, and the light source information map when each image is shot needs three coordinate values of X, Y and Z to represent, and the value of each coordinate direction is stored in a channel; the 3-channel information of each image and the 3-channel information of the corresponding light source information map, and the information of these six channels are operated through a convolution kernel with a size of 6x1x1, thereby fusing the image and the corresponding light source information, so as to prevent the one-to-one correspondence from being disturbed in the subsequent operation.

[0024] Optionally, the inter-frame information is extracted by using an IRFE (inter-frame information extractor),

[0025] The IRFE is composed of a convolution kernel with a size of 5x1x1, a leaky Relu activation function and a dropout layer: · represents the input of the inter-frame information extractor, represents i times of convolution on the input, and the size of the convolution kernel is 5x1x1; represents an activation operation on , represents a dropout operation on .

[0026] Further, the step of extracting the inter-frame information by using the IRFE is:

[0027] The initial input of IRFE is set as F0,

[0028]

[0029] where F0 represents the output of the information fusion module formula; represents the output of the first inter-frame information extractor, IRFE 1 (F0) represents that the first inter-frame information extractor takes F0 as input; K is the number of IRFEs, represents the output of the Kth inter-frame information extractor, represents that the Kth inter-frame information extractor takes the output of the K-1th inter-frame information extractor as input;

[0030] Preferably, K = 3.

[0031] Further, spatial information extraction is performed using an IAFE (spatial information extractor), which is composed of a convolution kernel with a size of 1x3x3 and a leaky Relu activation function:

[0032] where · represents the input of the spatial information extractor, represents that the input is convolved i times, and the size of the convolution kernel is 1x3x3, represents that the output of the i-1th spatial information extractor is activated.

[0033] Further, the initial input of the IAFE is set as

[0034]

[0035]

[0036] wherein, represents the output of the first spatial information extractor, represents the input of the first spatial information extractor, represents the output of the Lth spatial information extractor, represents that the output of the L-1th spatial information extractor is taken as the input of the Lth spatial information extractor.

[0037] Preferably, L = 3.

[0038] Further, the output of the Lth spatial information extractor is taken as the input of the max-pooling,

[0039] MP represents the max-pooling operation, and F MAX ​​Output of max-pooling. The operation of this step extracts the most significant information between different channels, and also fixes the input channels for the following operations.

[0040] Further, F MAX As input, the normal vector is solved:

[0041]

[0042] Where, Con 1×1 represents a convolution of size 1x1, Conv 1×1 (F MAX ) represents a convolution operation on F MAX , the size of the convolution kernel is 1x1, and sigma represents a leaky Relu activation function, represents L2 regularization;

[0043] N ij represents the normal vector of the point with pixel coordinates (i, j), and the normal map formed by the normal vectors of all pixels of the entire image represents the surface three-dimensional information of the object to be measured.

[0044] Advantages of the present application:

[0045] The present application introduces three-dimensional convolution into the field of photometric stereo, and proposes an MT-CNN-PS model; the model fully utilizes the inter-frame information and spatial information of the input image sequence, has good normal vector recovery accuracy; the spatial information has good information compensation effect on highlight points and shadow areas, and the addition of spatial information can improve the robustness of the algorithm to abnormal areas;

[0046] The present application adds a light source image mask during training to reduce the interference of useless points on the result; the robustness of the model to the shadow part of the image is improved; the model of the present application has high normal recovery accuracy compared with existing photometric stereo methods while retaining high operation speed, and has good industrial application prospect. BRIEF DESCRIPTION OF DRAWINGS

[0047] The present application can be further illustrated by the non-limiting examples shown in the accompanying drawings.

[0048] Figure 1 is a network structure diagram of the non-Lambertian surface photometric stereo model based on three-dimensional convolution of the present application.

[0049] Figure 2 is a flowchart of the non-Lambertian surface photometric stereo method based on three-dimensional convolution of the present application.

[0050] Figure 3 is a network structure diagram of the present application embodiment 1-5.

[0051] Figure 4 is a schematic diagram of a network structure of embodiment 6 of the present application.

[0052] Figure 5 is a schematic diagram of a network structure of embodiment 7 of the present application.

[0053] Figure 6 is a schematic diagram of a network structure of embodiment 8 of the present application.

[0054] Figure 7 is a comparison result of rendering image results based on the MERL dataset. DETAILED DESCRIPTION

[0055] In order for those skilled in the art to better understand the present application, the technical solutions of the present application are further described below in conjunction with the drawings and embodiments.

[0056] As shown in Figure 1 , the present application discloses a non-Lambertian surface photometric stereo model based on three-dimensional convolution, which comprises an information fusion layer 1, an inter-frame information extractor 2, a spatial information extractor 3, a maximum pooling layer 4 and a regression layer 5.

[0057] The information fusion layer 1 is used to fuse each image with its corresponding light source information, so as to ensure one-to-one correspondence between the image and the light source in subsequent processing.

[0058] The inter-frame information extractor 2 is used to extract inter-frame information and obtain information between input image frames, which is used for normal map estimation.

[0059] The spatial information extractor 3 extracts structural information inside a single image for normal map recovery.

[0060] The maximum pooling layer 4 is used for dimension reduction, removal of redundant information, compression of features, simplification of network complexity, reduction of calculation amount and reduction of memory consumption.

[0061] The information fusion layer 1 comprises a three-dimensional convolution layer C1, followed by a LeakyReLU activation function, and the output value dropout ratio of the LeakyReLU activation function is 0.2.

[0062] The inter-frame information extractor 2 comprises three three-dimensional convolution layers, which are three-dimensional convolution layer C2, three-dimensional convolution layer C3 and three-dimensional convolution layer C4, respectively, and each three-dimensional convolution layer is followed by a LeakyReLU activation function, and the output value dropout ratio of each LeakyReLU activation function is 0.2.

[0063] The spatial information extractor 3 comprises three three-dimensional convolution layers, which are three-dimensional convolution layer C5, three-dimensional convolution layer 6 and three-dimensional convolution layer C7 respectively, each of which is followed by a LeakyReLU activation function.

[0064] The regression layer 5 comprises three two-dimensional convolution layers, which are two-dimensional convolution layer C8, two-dimensional convolution layer C9 and two-dimensional convolution layer C10 respectively, two-dimensional convolution layer C8 and two-dimensional convolution layer C9 are followed by a LeakyReLU activation function, and two-dimensional convolution layer C10 is followed by an L2 normalization correction function.

[0065] In some embodiments, a data processing system comprises a data processor, a graphics processor and a data storage, the data processor reads an image from the data storage and inputs the image into the graphics processor, the graphics processor is preset with an image recognition network, the image recognition network is a non-Lambertian surface photometric stereo model based on three-dimensional convolution, such as Figure 1 、 2 As shown in the figure, the image recognition network comprises an image input, a feature extractor for obtaining effective features of the input image, a maximum pooling layer and a normal solution unit; the image is input into an initial fusion module, the initial fusion module makes the current image and its illumination information one-to-one correspondence, the image fused with the illumination information output by the initial fusion module is input into the feature extractor, the feature extractor comprises an inter-frame information extractor and a spatial information extractor, both of which use three-dimensional convolution kernels, wherein the inter-frame information occupies one dimension and the spatial information occupies two dimensions; the input of the inter-frame information extractor is the output of the initial fusion module, and the input of the spatial information extractor is the output of the inter-frame information extractor; the output of the spatial information extractor is used as the input of the maximum pooling layer.

[0066] In some embodiments, the normal solution unit first performs a convolution operation on the output of the maximum pooling layer, and then performs L2 regularization, and the result of the regularization output is a normal map. In some embodiments, the last convolution operation of the normal solution unit uses 3 channels. In some embodiments, other convolution operations of the image recognition network use 128 channels. The above-mentioned regression layer serves as the normal vector solution unit.

[0067] As shown in the figure, Figure 1 、 2As shown, in some embodiments, spatial information and inter-frame information are extracted from the image of the object under test. Both spatial information and inter-frame information are represented by three-dimensional convolutions. The three-dimensional convolution includes a two-dimensional spatial dimension and a one-dimensional inter-frame dimension. The value of the one-dimensional inter-frame dimension in the three-dimensional convolution of spatial information is a set value, and the two-dimensional spatial dimension is the spatial information value of the image. The value of the two-dimensional spatial dimension in the three-dimensional convolution of inter-frame information is a set value, and the one-dimensional inter-frame dimension is the inter-frame information value. When performing image information fusion, the inter-frame information is processed first, then the spatial information is processed, and then the inter-frame information and spatial information are fused to obtain a three-dimensional image of the surface texture of the object under test.

[0068] In some embodiments, the step of fusing spatial information and inter-frame information of an image of an object under test includes:

[0069] Formula for establishing an information fusion module:

[0070] Among them, IL input Indicates the input of the formula, IL input This includes each image captured during each shooting mission, along with the light source intensity and direction. F0 represents the output of the formula, and Con... 6×1×1 This represents a convolution kernel of size 6×1×1, Con 6×1×1 (IL input ) indicates IL input Perform a convolution operation, where σ represents the leaky ReLU activation function, σ(Con 6×1×1 (IL input ) indicates that for Con 6×1×1 (IL input Perform the activation operation. This indicates a dropout operation. Indicates that for σ(Con) 6×1×1 (IL input Perform dropout operations. Convolution, activation, and dropout operations are common functions or operations in deep learning.

[0071] Each image has three channels: R, G, and B. The light source information map of each image needs three coordinate values: X, Y, and Z. The value of each coordinate direction is stored in one channel. The three-channel information of each image and the three-channel information of the corresponding light source information map, these six channels are processed through a 6×1×1 convolution kernel, thereby fusing the image with the corresponding light source information and preventing subsequent operations from disrupting the one-to-one correspondence.

[0072] In some embodiments, inter-frame information extraction is performed using an IRFE (Inter-Frame Information Extractor).

[0073] IRFE consists of a 5x1x1 convolution kernel, a leaky Relu activation function and a dropout layer: • represents the input of the inter-frame information extractor, represents the i-th convolution of the input with a 5x1x1 convolution kernel; represents the activation operation on ; represents the dropout operation on .

[0074] In some embodiments, the step of extracting inter-frame information using IRFE is:

[0075] The initial input of IRFE is set as F0,

[0076]

[0077] where F0 represents the output of the information fusion module formula; represents the output of the first inter-frame information extractor, IRFE 1 (F0) represents that the first inter-frame information extractor takes F0 as the input; K is the number of IRFEs, represents the output of the K-th inter-frame information extractor, represents that the K-th inter-frame information extractor takes the output of the K-1-th inter-frame information extractor as the input. In this embodiment, K = 3.

[0078] In some embodiments, spatial information is extracted using IAFE (spatial information extractor), which consists of a 1x3x3 convolution kernel and a leaky Relu activation function: where • represents the input of the spatial information extractor, represents the i-th convolution of the input with a 1x3x3 convolution kernel, represents the activation operation on .

[0079] Further, the initial input of IAFE is set as

[0080]

[0081] where, represents the output of the first spatial information extractor, represents the input of the first spatial information extractor, represents the output of the L-th spatial information extractor, represents taking the output of the (L-1)-th spatial information extractor as the input of the L-th spatial information extractor. In this embodiment, L=3.

[0082] In some embodiments, the input of the maximum pooling is MP represents the maximum pooling operation, F MAX represents the output of the maximum pooling. The operation of this step extracts the most significant information between different channels, and also fixes the input channels for the subsequent operations.

[0083] Further, the normal vector solving is performed as the input of F MAX

[0084]

[0085] where Con 1×1 represents a convolution with a size of 1*1, Con 1×1 (F MAX ) represents a convolution operation on F MAX , the size of the convolution kernel is 1*1, and σ represents a leaky Relu activation function, represents L2 regularization; N ij represents the normal vector of the point with pixel coordinates (i, j), and the normal map formed by the normal vectors of all pixels of the entire image represents the surface three-dimensional information of the object to be measured.

[0086] Specific embodiments 1-5,

[0087] As shown in Figure 3 , the non-Lambertian surface photometric stereo model based on three-dimensional convolution includes an information fusion layer, an inter-frame information extractor, a spatial information extractor, a maximum pooling layer, and a regression layer.

[0088] The inter-frame information extractor includes three three-dimensional convolution layers, which are three-dimensional convolution layer C2, three-dimensional convolution layer C3, and three-dimensional convolution layer C4, respectively. Each three-dimensional convolution layer is followed by a LeakyReLU activation function, and each LeakyReLU activation function has an output value dropout ratio of 0.2. The three-dimensional convolution layer C2, the three-dimensional convolution layer C3, and the three-dimensional convolution layer C4 include IF 1*1 feature maps.

[0089] The spatial information extractor includes three three-dimensional convolution layers, which are three-dimensional convolution layer C5, three-dimensional convolution layer 6, and three-dimensional convolution layer C7, respectively. Each three-dimensional convolution layer is followed by a LeakyReLU activation function. The three-dimensional convolution layer C5, the three-dimensional convolution layer 6, and the three-dimensional convolution layer C7 include 1 S*S feature map.

[0090] ​​In the image processing process, the inter-frame information extractor processing is performed first, and then the inter-frame information extractor processing is performed again.

[0091] In the image processing process, the inter-frame information extractor processing is performed first, and then the inter-frame information extractor processing is performed again.

[0092] In the image processing process, the inter-frame information extractor processing is performed first, and then the inter-frame information extractor processing is performed again.

[0093] In the image processing process, the inter-frame information extractor processing is performed first, and then the inter-frame information extractor processing is performed again.

[0094] In the image processing process, the inter-frame information extractor processing is performed first, and then the inter-frame information extractor processing is performed again.

[0095] In the image processing process, the inter-frame information extractor processing is performed first, and then the inter-frame information extractor processing is performed again.

[0096] In the image processing process, the inter-frame information extractor processing is performed first, and then the inter-frame information extractor processing is performed again.

[0097] As shown in the image processing process, the inter-frame information extractor processing is performed first, and then the inter-frame information extractor processing is performed again. Figure 4 The non-Lambertian surface photometric stereo model based on three-dimensional convolution includes an information fusion layer, an inter-frame information extractor, a spatial information extractor, a maximum pooling layer, and a regression layer.

[0098] The inter-frame information extractor includes three three-dimensional convolution layers, namely three-dimensional convolution layer C2, three-dimensional convolution layer C3, and three-dimensional convolution layer C4, each followed by a LeakyReLU activation function, and each LeakyReLU activation function outputs a value with a dropout ratio of 0.2. The three-dimensional convolution layer C2, the three-dimensional convolution layer C3, and the three-dimensional convolution layer C4 include 1 three-dimensional feature map.

[0099] The spatial information extractor includes three three-dimensional convolution layers, namely three-dimensional convolution layer C5, three-dimensional convolution layer 6, and three-dimensional convolution layer C7, each followed by a LeakyReLU activation function. The three-dimensional convolution layer C5, the three-dimensional convolution layer 6, and the three-dimensional convolution layer C7 include 5 one-dimensional feature maps.

[0100] In the image processing process, the inter-frame information extractor processing is performed first, and then the inter-frame information extractor processing is performed again.

[0101] In the image processing process, the inter-frame information extractor processing is performed first, and then the inter-frame information extractor processing is performed again.

[0102] As shown in the image processing process, the inter-frame information extractor processing is performed first, and then the inter-frame information extractor processing is performed again. Figure 5 The non-Lambertian surface photometric stereo model based on three-dimensional convolution includes an information fusion layer, an inter-frame information extractor, a spatial information extractor, a maximum pooling layer, and a regression layer.

[0103] The inter-frame information extractor includes three three-dimensional convolution layers, namely three-dimensional convolution layer C2, three-dimensional convolution layer C3, and three-dimensional convolution layer C4, each followed by a LeakyReLU activation function, and each LeakyReLU activation function outputs a value with a dropout ratio of 0.2. The three-dimensional convolution layer C2, the three-dimensional convolution layer C3, and the three-dimensional convolution layer C4 include 1 three-dimensional feature map.

[0104] The spatial information extractor includes three three-dimensional convolution layers, which are three-dimensional convolution layer C5, three-dimensional convolution layer 6 and three-dimensional convolution layer C7 respectively, each followed by a LeakyReLU activation function; the three-dimensional convolution layer C5, the three-dimensional convolution layer 6 and the three-dimensional convolution layer C7 include 1 feature map of S*S;

[0105] In the image processing process, the inter-frame information extractor is processed first, and then the inter-frame information extractor is processed;

[0106] Wherein, S = 3, IF = 5;

[0107] Embodiment 8

[0108] As shown in Figure 6 The non-Lambertian surface photometric stereo model based on three-dimensional convolution includes an information fusion layer, a feature extraction layer, a maximum pooling layer and a regression layer.

[0109] The feature extraction layer includes three three-dimensional convolution layers, which are three-dimensional convolution layer C2, three-dimensional convolution layer C3 and three-dimensional convolution layer C4 respectively, each followed by a LeakyReLU activation function, and each LeakyReLU activation function has an output value dropout ratio of 0.2, and the three-dimensional convolution layer C2, the three-dimensional convolution layer C3 and the three-dimensional convolution layer C4 include 5 feature maps of 3*3;

[0110] The convolution data analysis results of embodiments 1-8 are tested for images of different shapes, and the MAE values are shown in Table 1,

[0111]

[0112] Table 1

[0113] As shown in Table 1, referring to the MAE values of each embodiment, the effect of embodiment 1 for each kind of image is relatively good; and it is relatively stable.

[0114] According to the convolution layer data of embodiment 1, an MT-CNN-PS model is established, and this model is compared with other models, and the MAE values are shown in Table 2 for images of different shapes.

[0115]

[0116] Table 2

[0117] As shown in Table 2, the images processed by the MT-CNN-PS model have obvious advantages and disadvantages for different shapes, and have obvious advantages on the bear and buddha samples in the public data set DiLiGenT, so the processed images have high accuracy.

[0118] According to the convolutional layer data of Example 1, the MT-CNN-PS model is established, and the average error of the photometric stereo method of the model and other photometric stereo methods when the number of input pictures is small (DiLiGenT ten object average error) is compared, and the MAE value is shown in Table 3,

[0119]

[0120] Table 3

[0121] The first row of the table represents the input image, the first column represents the method name, and the middle value represents the average angle error.

[0122] The table is a horizontal comparison between the model and the current advanced sparse input photometric stereo method, wherein

[0123] JU-19, CH-18, SI-18 are all deep learning-based methods, and JU-19, CH-18 have specific structures to solve the problem of sparse input. The method is a traditional method. It can be seen that our model does not have a complex structure for sparse input, but only uses an inter-frame spatial extractor to improve information utilization to achieve good results when inputting 16 and 10 pictures.

[0124] According to the convolutional layer data of Example 1, the MT-CNN-PS model is established, and the average error of the photometric stereo method of the model and other photometric stereo methods when the number of input pictures is small (DiLiGenT ten object specific error) is compared, and the MAE value is shown in Table 3,

[0125]

[0126] Table 4

[0127] As shown in Table 3 and Table 4, compared with other photometric stereo methods, the MT-CNN-PS model has relatively good image processing stability.

[0128] The convolutional layer data of Example 1 is used to establish the MT-CNN-PS model, and the images processed by the photometric stereo method of the model and the images processed by the photometric stereo method of the PS-FCN model are compared based on the MERL dataset rendering image results as Figure 7 shown,

[0129] As Figure 7 known, compared with PS-FCN, the MT-CNN-PS model has relatively good image processing stability and better accuracy.

[0130] In conclusion, the application has good normal vector recovery accuracy; the spatial information has good information compensation effect on highlight points and shadow areas, and adding the spatial information can improve the robustness of the algorithm for abnormal areas; the model of the application has high normal recovery accuracy compared with the existing photometric stereo method while retaining high operation speed, and has good industrial application prospect.

[0131] The above embodiments only illustrate the principles and effects of the present application, and are not intended to limit the present application. Any person skilled in the art can modify or change the above embodiments without departing from the spirit and scope of the present application. Therefore, all equivalent modifications or changes made by those skilled in the art without departing from the spirit and technical idea of the present application should be covered by the claims of the present application.

Claims

1. A non-Lambertian surface photometric stereoscopic method based on three-dimensional convolution, characterized in that: The information fusion layer, the inter-frame information extraction layer, the spatial information extraction layer, the maximum pooling layer, and the regression layer are included. The information fusion layer is configured to fuse each image and corresponding light source information, so as to ensure one-to-one correspondence between the image and the light source in subsequent processing. The inter-frame information extraction layer is configured to extract inter-frame information, so as to obtain information between input image frames and estimate a normal map. The spatial information extraction layer is configured to extract structural information inside a single image, so as to restore the normal map. The maximum pooling layer is configured to reduce dimension, remove redundant information, compress features, simplify network complexity, reduce calculation amount, and reduce memory consumption. The image is input to an initial fusion module of the information fusion layer, the initial fusion module ensures one-to-one correspondence between the current image and the light information, the image fused with the light information output by the initial fusion module is input to the feature extraction layer, the inter-frame information extraction layer and the spatial information extraction layer both use a three-dimensional convolution kernel, wherein the inter-frame information occupies one dimension and the spatial information occupies two dimensions; the input of the inter-frame information extraction layer is the output of the initial fusion module, the input of the spatial information extraction layer is the output of the inter-frame information extraction layer; and the output of the spatial information extraction layer is input to the maximum pooling layer.

2. The three-dimensional convolution-based non-Lambertian surface photometric stereo method of claim 1, wherein: The information fusion layer includes a three-dimensional convolution layer C1, followed by a LeakyReLU activation function, and the dropout ratio of the output value of the LeakyReLU activation function is 0.

2.

3. The three-dimensional convolution based non-Lambertian surface photometric stereo method of claim 1, wherein: The inter-frame information extraction layer includes three three-dimensional convolution layers, namely a three-dimensional convolution layer C2, a three-dimensional convolution layer C3, and a three-dimensional convolution layer C4, each followed by a LeakyReLU activation function, and the dropout ratio of the output value of each LeakyReLU activation function is 0.

2.

4. The three-dimensional convolution-based non-Lambertian surface photometric stereo method of claim 1, wherein: The spatial information extraction layer includes three three-dimensional convolution layers, namely a three-dimensional convolution layer C5, a three-dimensional convolution layer C6, and a three-dimensional convolution layer C7, each followed by a LeakyReLU activation function.

5. The three-dimensional convolution-based non-Lambertian surface photometric stereo method of claim 1, wherein: The regression layer includes three two-dimensional convolution layers, namely a two-dimensional convolution layer C8, a two-dimensional convolution layer C9, and a two-dimensional convolution layer C10, the two-dimensional convolution layer C8 and the two-dimensional convolution layer C9 are followed by a LeakyReLU activation function, and the two-dimensional convolution layer C10 is followed by an L2 normalization correction function.

Images