Photometric stereoscopic vision measurement method

Abstract

The invention discloses a photometric stereoscopic vision measurement method which is characterized by comprising the following steps: processing an input image sequence of a measured object based on an illumination information extraction module to obtain an illumination feature tensor; performing diffuse reflection feature extraction on the illumination feature tensor based on a diffuse reflection information analysis module to generate a normal vector information graph of the surface of the measured object; based on a specular reflection information analysis module, specular reflection feature extraction is carried out on the illumination feature tensor so as to determine specular reflection information of the surface of the measured object; based on a normal vector prediction module, performing fusion convolution orientation processing on the normal vector information graph and the mirror reflection information to generate a three-dimensional normal vector of the surface of the measured object; and based on a depth prediction module, performing iterative tensor optimization processing on the normal vector information graph and the mirror reflection information to generate a depth prediction result of the measured object.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a photometric stereo vision measurement method. Background Technology

[0002] Photometric stereo vision technology, as an innovative method in the field of optical measurement, has demonstrated significant advantages in industrial inspection, biometrics, and other fields due to its non-contact and highly efficient 3D reconstruction capabilities. This technology bypasses traditional 3D point cloud computing processes by analyzing image sequences under multiple illumination conditions, directly reconstructing the surface normal vectors of objects. This significantly improves measurement efficiency in complex scenarios, and has broad application potential, especially in high-precision scenarios such as surface defect detection, face recognition, and surface texture analysis.

[0003] Existing photometric stereo vision methods primarily rely on the Lambertian diffuse reflection model to construct photometric stereo equations, achieving reconstruction by solving the mapping relationship between illumination parameters and normal vectors. While these methods perform well on purely diffuse surfaces (such as matte materials), in practical applications, most object surfaces also possess specular reflection components (such as metals and smooth plastics), introducing high-level noise into the image. This makes it difficult for traditional single-model methods to accurately separate geometric features from optical properties, leading to reconstruction results prone to normal vector bias or depth discontinuities. Furthermore, existing technologies utilize multi-scale illumination features in a relatively limited way, typically processing fixed-resolution image sequences using only a single convolutional network. This lack of ability to hierarchically model illumination information at different levels results in insufficient robustness of feature extraction when dealing with high dynamic range illumination or complex curvature surfaces, and reconstruction accuracy fluctuates significantly with changes in illumination conditions. Summary of the Invention

[0004] This application provides a technical solution for a photometric stereo vision measurement method.

[0005] A photometric stereo vision measurement method includes: processing an input image sequence of the object under test based on an illumination information extraction module to obtain an illumination feature tensor; extracting diffuse reflection features from the illumination feature tensor based on a diffuse reflection information analysis module to generate a normal vector information map of the object surface; extracting specular reflection features from the illumination feature tensor based on a specular reflection information analysis module to determine the specular reflection information of the object surface; and performing a fusion convolutional orientation processing on the normal vector information map and the specular reflection information based on a normal vector prediction module to generate a three-dimensional normal vector of the object surface.

[0006] A photometric stereo vision measurement method includes: processing an input image sequence of the object under test using an illumination information extraction module to obtain an illumination feature tensor; extracting diffuse reflection features from the illumination feature tensor using a diffuse reflection information analysis module to generate a normal vector information map of the object's surface; extracting specular reflection features from the illumination feature tensor using a specular reflection information analysis module to determine the specular reflection information of the object's surface; performing a fusion convolutional orientation process on the normal vector information map and specular reflection information using a normal vector prediction module to generate a three-dimensional normal vector of the object's surface; and performing iterative tensor optimization processing on the normal vector information map and specular reflection information using a depth prediction module to generate a depth prediction result for the object. Attached Figure Description

[0007] Figure 1 A flowchart of the photometric stereo vision measurement method provided for the implementation of this application.

[0008] Figure 2 A schematic diagram of the overall network architecture upon which the photometric stereo vision measurement method provided for the implementation of this application depends.

[0009] Figure 3 A schematic diagram of the feature pyramid module provided for the implementation of this application.

[0010] Figure 4 A schematic diagram of the ResNet18 backbone network provided for the implementation of this application.

[0011] Figure 5 A schematic diagram of the diffuse reflection information analysis module provided for the implementation of this application.

[0012] Figure 6 This is a schematic diagram of the specular reflection information analysis module provided in an embodiment of this application.

[0013] Figure 7 This is a schematic diagram of the lighting settings for capturing image sequences provided in an embodiment of this application.

[0014] Figure 8 This is a schematic diagram of a three-dimensional normal vector prediction map provided in an embodiment of this application.

[0015] Figure 9 This is a schematic diagram of the structure of the depth prediction module provided in an embodiment of this application.

[0016] Figure 10.1 This is a structural schematic diagram of the device design provided in the embodiments of this application.

[0017] Figure 10.2 This is a schematic diagram of the actual structure of the main body of the photometric stereoscopic vision device provided in the embodiments of this application.

[0018] Figure 11.1 This is a structural schematic diagram of the calibration plate rotating with the rotary table, as provided in the embodiments of this application.

[0019] Figure 11.2 This is a schematic diagram of the structure for imaging using a camera arm swing calibration plate, as provided in an embodiment of this application.

[0020] Figure 12 This is a schematic diagram of the actual object under test provided in the embodiments of this application.

[0021] Figure 13 This is a schematic diagram of the target feature extraction structure provided in an embodiment of this application.

[0022] Figure 14 This is a schematic diagram of the structure of a real dataset provided in the embodiments of this application.

[0023] Figure 15 This is a schematic diagram of the structure of the publicly available blob dataset provided in an embodiment of this application.

[0024] Figure 16 This is a schematic diagram of the structure of a portion of the virtual dataset images provided in an embodiment of this application.

[0025] Figure 17 This is a schematic diagram of the structure of the virtual dataset image provided in the embodiments of this application.

[0026] Figure 18 This is a schematic diagram of the structure of a virtual dataset of a real object under test provided in an embodiment of this application.

[0027] Figure 19 This is a schematic diagram showing the comparison of the prediction effect of the virtual test object normal vector provided in the embodiments of this application.

[0028] Figure 20 This is a schematic diagram of the structure of the actual object being measured and the predicted normal vector results provided in the embodiments of this application.

[0029] Figure 21 This is a schematic diagram of the structure for depth prediction of a virtual dataset provided in an embodiment of this application.

[0030] Figure 22 This is a schematic diagram of the structure for depth prediction of a real object under test provided in an embodiment of this application.

[0031] Figure 23 This is a schematic diagram of the structure of a specular reflection processing module that is only fused in series, as provided in an embodiment of this application.

[0032] It should be noted that, in order to clearly show and prepare the display technology content, the background color of some of the images in the above-mentioned figures of this application is dark. Detailed Implementation

[0033] like Figure 1 and Figure 2 As shown in the embodiments of this application, the photometric stereo vision measurement method includes: processing an input image sequence of the object under test based on an illumination information extraction module to obtain an illumination feature tensor; extracting diffuse reflection features from the illumination feature tensor based on a diffuse reflection information analysis module to generate a normal vector information map of the surface of the object under test; extracting specular reflection features from the illumination feature tensor based on a specular reflection information analysis module to determine the specular reflection information of the surface of the object under test; performing fusion convolutional orientation processing on the normal vector information map and specular reflection information based on a normal vector prediction module to generate a three-dimensional normal vector of the surface of the object under test; and performing iterative tensor optimization processing on the normal vector information map and specular reflection information based on a depth prediction module to generate a depth prediction result of the object under test.

[0034] Traditional photometric stereo vision technology has two major pain points in the illumination condition calibration process: First, it relies on special markers such as metal balls, making the calibration process cumbersome and susceptible to interference from hardware accuracy and algorithm errors; Second, when analyzing the direction and brightness of incident light based on the calibration results, the process is complex and time-consuming, and multiple reflections in the environment can also lead to errors in light calculation.

[0035] Optionally, in a specific implementation scenario, the image sequence of the object under test can be generated according to the following process: To address the above problems, this solution innovatively integrates neural network learning capabilities with the high-precision motion positioning capabilities of mechanical equipment, generating image sequences through a novel illumination condition association strategy: The number of image sequences for photometric stereo vision measurement is determined (e.g., 18 images). This sequence maintains its order and quantity throughout the prediction process. The zenith angle and azimuth angle of the light source are combined, and the image sequences are grouped for shooting to ensure that the illumination conditions of each image position in the image sequence are completely identical, thereby establishing a strong correlation between environmental illumination condition information and image position. When shooting the image sequences in groups, it is based on the combination of the zenith angle and azimuth angle of the light source, following the principle of "fixed zenith angle, equally spaced azimuth angle changes." That is, the image sequence is first divided into several groups according to the quantity, each group is assigned a fixed zenith angle, and within each group, the light source is rotated and shot at the same azimuth angle interval to ensure consistent illumination conditions within the group; different groups achieve multi-angle illumination coverage by changing the zenith angle. In this way, the lighting conditions of each image in the image sequence are fixed, thereby establishing a strong correlation between ambient lighting conditions and image position. For example, as shown... Figure 7As shown, if the image sequence includes 18 images, the grouping is as follows: Group 1 (images 1-4): With a fixed zenith angle, the light source rotates at 90° intervals to capture 4 images; Group 2 (images 5-10): Adjusting to another fixed zenith angle, the light source rotates at 60° intervals to capture 6 images; Group 3 (images 11-18): Changing the zenith angle again, the light source rotates at 36° intervals to capture 8 images. During the above shooting process, the light source position is precisely moved according to preset rules under different combinations of zenith and azimuth angles, ensuring that the zenith angle remains constant within each group and the azimuth angle rotates at a specified angle; regarding the light source intensity, the equipment maintains a constant light source intensity throughout the entire 18 image shooting process. The camera and the object under test remain in fixed positions throughout the shooting process; only the light source moves according to the rules to complete image acquisition.

[0036] In summary, by fixing the image sequence size and illumination rules and establishing positional illumination mapping relationships, image sequence generation without traditional calibration was achieved, effectively solving the time-consuming and error problems of traditional techniques, and laying an efficient and accurate data foundation for subsequent photometric stereo vision measurements.

[0037] Optionally, based on the illumination information extraction module, the input image sequence of the test object is processed to obtain an illumination feature tensor, including the following steps: performing multi-scale feature extraction on the image sequence of the test object to obtain multiple feature tensors with different resolutions; fusing the multiple feature tensors with different resolutions to obtain multiple fused feature tensors; performing enhancement mapping on each fused feature vector to obtain multiple enhanced feature tensors; and fusing the multiple enhanced feature tensors to obtain an illumination feature tensor. Optionally, the illumination information extraction module includes: a first pyramid convolutional network to perform multi-scale feature extraction on the image sequence of the test object to obtain multiple feature tensors with different resolutions. Optionally, the illumination information extraction module includes: a second pyramid convolutional network to fuse the multiple feature tensors with different resolutions to obtain multiple enhanced feature tensors. Optionally, the illumination information extraction module includes: an enhancement mapping network to perform enhancement mapping on each fused feature vector to obtain multiple enhanced feature tensors. Preferably, in a specific application scenario, the specific implementation of processing the input image sequence of the test object based on the illumination information extraction module to obtain an illumination feature tensor is as follows:

[0038] The illumination information extraction module employs a pyramid illumination information extraction network, based on... Figure 3 Feature Pyramid Network (FPN) and Figure 4 The ResNet18 backbone network enables multi-scale light and shadow feature extraction and fusion of image sequences:

[0039] I. Bottom-up stage: Multi-scale feature extraction (ResNet18 backbone network)

[0040] 1. Input Preprocessing: The input image sequence is cropped and downsampled to a fixed size (e.g., H×W) to serve as the bottom layer input of the feature pyramid. A convolutional layer with a kernel size of 3×3 and a stride of 2 is used to extract the initial feature tensor C1 (size...).

[0041] 2. Multi-resolution feature generation: Through four convolutional blocks of ResNet18 (each block contains two 3×3 convolutional layers + residual connections), feature tensors of different resolutions are extracted layer by layer: Convolutional block 1: Output feature tensor C2 (resolution H / 4×W / 4, number of channels 64); Convolutional block 2: Output feature tensor C3 (resolution H / 8×W / 8, number of channels 128); Convolutional block 3: Output feature tensor C4 (resolution H / 16×W / 16, number of channels 256); Convolutional block 4: Output feature tensor C5 (resolution H / 32×W / 32, number of channels 512). Residual connections: Within each convolutional block, the input and output are fused through skip connections to avoid gradient vanishing and enhance feature propagation efficiency.

[0042] II. Top-down phase: Feature fusion and resolution unification

[0043] 1. High-level feature initialization: Apply a 1×1 convolution (kernel size 1×1, stride 1) to the highest-level feature C5 to generate the initial feature tensor P5 (channel count uniformly set to 64). 2. Layer-by-layer upsampling and lateral connections: Starting from P5, generate features layer by layer through the following operations: 1. Upsampling: Upsample the current layer feature P... i Amplified by 2 times using interpolation, and then compared with the output C of the lower convolutional block. i-1 1. Resolution alignment; 2. Lateral convolution: for lower layer features C i-1 1×1 convolution is used to compress the number of channels to 64; 3. Feature summation: The upsampled P i C after horizontal convolution i-1 Element-wise addition generates the current layer's fused feature P. i-1 Finally, four fused feature tensors P2, P3, P4, and P5 with increasing resolutions (resolutions from H / 4×W / 4 to H / 32×W / 32, with 64 channels each) are generated.

[0044] III. Feature Prediction Stage: Multi-scale Feature Enhancement and Merging

[0045] 1. Single-scale feature enhancement: For each fused feature tensor P iThe standardization process is as follows: Preprocessing: 3×3 convolution + normalization + ReLU activation (shared weights) is used to extract local contextual features; Upsampling: Except for P2, P3, P4, and P5 need to be upsampled to a uniform resolution (e.g., H / 4×W / 4) using interpolation to generate feature tensors S2, S3, S4, and S5 (maintaining 64 channels). 2. Global feature fusion: The four enhanced feature tensors S2-S5 are added element-wise, and then a 1×1 convolution is performed (channel number adjusted to 128) to output the illumination feature tensor of a single image (resolution H / 4×W / 4, 128 channels).

[0046] IV. Multi-image sequence processing: Parallel networks and weight sharing

[0047] 1. Parallel Feature Extraction: For photometric stereo vision image sequences (e.g., N images), N parallel FPN network branches are used. Each branch independently processes a single image and generates the corresponding illumination feature tensor S. 0i (i = 1, 2, ..., N).

[0048] 2. Parameter Sharing Mechanism: To avoid overfitting and reduce computational cost, the weight parameters of all parallel branches of the FPN network are fully shared, ensuring consistent feature extraction logic for different images. The final output is a lighting feature tensor group {S} containing N elements. 01 ,S 02 ,...,S 0N}, used for subsequent normal vector prediction and depth reconstruction.

[0049] In summary, this application combines ResNet18 and FPN to simultaneously capture both low-level texture (high resolution) and high-level semantics (low resolution) of images, thereby enhancing the expressive power of illumination features. By using 1×1 convolutions to compress the number of channels and a shared weight mechanism, computational complexity is reduced while maintaining accuracy. Residual connections and normalization operations effectively alleviate gradient vanishing, improve network training stability, and adapt to image inputs under different illumination conditions.

[0050] Optionally, based on the diffuse reflection information analysis module, diffuse reflection features are extracted from the illumination feature tensor to generate a normal vector information map of the surface of the object under test. This includes: pooling the illumination feature tensor to obtain a fused feature map; convolving the fused feature map to generate a preliminary convolutional feature tensor containing diffuse reflection information; reducing the channels of the preliminary convolutional feature tensor to obtain a channel-reduced feature tensor; and mapping the number of channels of the channel-reduced feature tensor to the three dimensions of x, y, and z to generate a normal vector information map of the surface of the object under test.

[0051] Preferably, see Figure 5Based on the diffuse reflection information analysis module, the specific steps for extracting diffuse reflection features from the illumination feature tensor to generate a normal vector information map of the surface of the measured object are as follows: When performing pooling processing on the illumination feature tensor to obtain a fused feature map, for example, using max pooling, all illumination feature tensors corresponding to the input image sequence are fused, retaining the maximum value of each region, thus obtaining a fused feature map with unchanged resolution and 128 channels, with the same size as the image sequence. When performing convolution processing on the fused feature map to generate a preliminary convolutional feature tensor containing diffuse reflection information, convolution operations are performed using two convolution kernels with a stride of 1 to extract diffuse reflection-related features to fit the diffuse reflection function of the measured object, thereby generating a preliminary convolutional feature tensor containing diffuse reflection information. At this time, the number of channels remains unchanged. When performing channel reduction on the initial convolutional feature tensor to obtain the channel-reduced feature tensor, a convolutional layer with a stride of 1 and a kernel size of 33 is used to reduce the number of feature channels of the initial convolutional feature tensor containing diffuse information from 128 to 64, resulting in a channel-reduced feature tensor N with 64 channels. L This reduces the number of channels, lowers the data dimensionality, and removes redundant information. When mapping the number of channels in the channel reduction feature tensor to the x / y / z dimensions to generate the normal vector information map of the tested object's surface, a convolutional layer with a stride of 1 and a kernel size of 33 is used. This maps the number of channels from 64 to the x, y, and z dimensions in three-dimensional space. The three channel values ​​of each pixel correspond to the components of the normal vector in the three dimensions, thus adjusting to 3 channels to fit and obtain the preliminary normal vector information map N0 of the tested object's surface.

[0052] In summary, this application achieves the transformation from illumination feature tensors to normal vector information maps through four core steps: max pooling fusion, multi-level convolution, channel reduction, and dimension mapping. Max pooling integrates lighting features from multiple frames, multi-level convolution fits the diffuse reflection function, channel reduction optimizes the data dimension, and finally maps the channels to three-dimensional space, outputting a feature map that conforms to the surface normal vector expression, providing crucial data support for subsequent normal vector prediction.

[0053] Optionally, based on the specular reflection information analysis module, specular reflection features are extracted from the illumination feature tensor to determine the specular reflection information of the surface of the object under test. This includes: extracting pixel-level illumination direction information from the illumination feature tensor; generating an image specular reflection map based on the extracted pixel-level illumination direction information; fusing the image specular reflection map and the image sequence to obtain a multi-channel feature map that fuses the specular reflection intensity on the image specular reflection map with the texture in the image sequence; and performing convolution processing on the multi-channel feature map to determine the specular reflection information of the surface of the object under test. Preferably, the specular reflection information analysis module includes an illumination direction extraction sub-network to perform multi-level channel compression on the illumination feature tensor to obtain pixel-level illumination direction information. Preferably, the specular reflection information analysis module includes a physics-driven calculation module to calculate the specular reflection amount based on the normal vector information map, the camera viewing direction, and the pixel-level illumination direction information to generate an image specular reflection map reflecting the specular reflection intensity. Preferably, the specular reflection information analysis module includes a channel stitching module to stitch the image specular reflection map and the image sequence along the channel dimension, forming a multi-channel feature map containing specular reflection intensity and image texture. Preferably, the specular reflection information analysis module includes a specular reflection fitting convolutional network to extract features layer by layer from the multi-channel feature map and fit the relationship between specular reflection components and imaging, thereby outputting the specular reflection information of the surface of the object under test.

[0054] Preferably, see Figure 6 Based on the specular reflection information analysis module, specular reflection feature extraction is performed on the illumination feature tensor to determine the specular reflection information of the surface of the measured object. The implementation is as follows: When extracting pixel-level illumination direction information from the illumination feature tensor, two convolutional layers with a stride of 1 and a kernel size of 33 are used. The number of channels in the illumination feature tensor is adjusted sequentially from 128 to 64 and then to 3. The final feature tensor is then normalized. Specifically: The first convolutional layer compresses the number of channels in the illumination feature tensor from 128 to 64, initially extracting illumination direction-related features to obtain the output feature map; the second convolutional layer further compresses the number of channels in the output feature map of the first convolutional layer to 3, focusing on pixel-level illumination direction information. Normalization is performed to ensure that the numerical distribution of the feature tensor meets the requirements of subsequent calculations, resulting in a pixel-level illumination direction information tensor with 3 channels for outputting pixel-level illumination direction information. Figure 6 In the diagram, S0 represents the illumination feature tensor. N0 represents the normal vector information map of the surface of the object under test generated by the diffuse reflection information analysis module. R iLThis refers to the specular reflection information of the tested object's surface output by the specular reflection information analysis module. When generating the image specular reflection map based on the extracted pixel-level illumination direction information, the normalized pixel-level illumination direction information is used as input. Combined with the target surface normal vector obtained from diffuse reflection component analysis and the camera's viewing direction (vertically upward, i.e., v = [0,0,1]), and based on the constructed specular reflection model, the specular reflection amount is calculated in the network's physical driving stage to generate the image specular reflection map. Specifically, using the physical driving calculation module, based on r... xi =v xi T (2(n0(x) T l xi )·n0(x)-l xi The mathematical and physical relationship between the target surface normal vector, the incident ray direction (pixel-level illumination direction information), and the camera's viewing direction is used to calculate the specular reflection amount pixel by pixel, generating a specular reflection map that reflects the intensity of specular reflection. The closer the specular reflection ray is to the camera's viewing direction, the higher the pixel value at the corresponding position in the specular reflection map, reflecting the greater contribution of the specular reflection component at that position.

[0055] In the above formula: r xi : Represents the amount of specular reflection at position x in the image. It is the result of this formula and is used to generate a specular reflection map that reflects the intensity of specular reflection. The larger the value, the stronger the specular reflection at the corresponding position. The camera's observation direction v at the corresponding position xi The transpose of converts a column vector into a row vector for vector dot product operations. In this paper, the camera's viewing direction is vertically upward, i.e., v = [0, 0, 1]. n0(x): The normal vector at point x on the target surface, obtained from diffuse reflection component analysis, reflects the orientation of the target surface at that point and plays a crucial role in calculating specular reflection. xi The incident light direction at the corresponding location, i.e., pixel-level illumination direction information, determines the angle of incidence of the light. n0(x) T : The transpose of the normal vector n0(x) at point x on the target surface, used for vector dot product operations. Through n0(x) T l xi The dot product of the normal vector and the incident light direction is calculated to obtain information related to the angle between the two, which is then used to calculate the amount of specular reflection.

[0056] When fusing an image specular reflection map and an image sequence to obtain a multi-channel feature map that integrates specular reflection intensity and image texture, the generated image specular reflection map (1 channel) and the original image sequence (e.g., RGB 3 channels) are stitched together. Three convolutional layers with a stride of 1 and a kernel size of 3×3 are used to extract features from the stitched image to generate a multi-channel feature map containing both specular reflection intensity and image texture. When performing convolution processing on the multi-channel feature map to determine the specular reflection information of the object's surface, a specular reflection fitting convolutional network is used to extract features layer by layer from the multi-channel feature map, fitting the relationship between the specular reflection component and the imaging, and outputting the specular reflection information of the object's surface. Specifically, the specular reflection fitting convolutional network consists of three convolutional layers with a stride of 1 and a kernel size of 3×3, extracting features from the multi-channel feature map layer by layer to output the final specular reflection information. Each pixel value represents the contribution of the specular reflection component at that location. To address this, this application employs a deep learning network incorporating a physical model, utilizing a process of "pixel-level illumination direction extraction, physical model calculation, multi-channel feature fusion, and convolutional fitting of specular reflection information" to achieve precise extraction from illumination feature tensors to specular reflection information. Each step has a clear division of labor, and the physical model and convolutional network work synergistically to effectively capture the specular highlight phenomenon, providing crucial data support for subsequent 3D reconstruction.

[0057] Optionally, based on the normal vector prediction module, the normal vector information map and specular reflection information are fused and convolutionally oriented to generate a 3D normal vector prediction map of the surface of the object under test. This includes: stitching and fusing the normal vector information map and specular reflection information to obtain a stitched feature map; performing convolution processing on the stitched feature map to obtain a primary feature map; performing deconvolution operation on the primary feature map to increase the feature resolution and obtain a full-size feature map; performing convolution processing on the full-size feature map to predict the normal vector information of the surface of the object under test; and normalizing the predicted normal vector information to output the 3D normal vector map of the object under test. Optionally, the normal vector prediction module includes: an information fusion unit configured to stitch the normal vector information map and specular reflection information along the channel dimension to generate a stitched feature map. Optionally, the normal vector prediction module includes: a primary convolution unit containing two cascaded convolution kernels configured to perform convolution processing on the stitched feature map to generate a primary feature map. Optionally, the normal vector prediction module includes: a resolution restoration unit, including a deconvolution layer, configured to expand the spatial resolution of the primary feature map to the size of the input image sequence, generating a full-size feature map. Optionally, the normal vector prediction module includes: a normal vector prediction unit, containing two cascaded convolutional kernels, configured to perform regression prediction on the full-size feature map, generating predicted surface normal vector information. Optionally, the normal vector prediction module includes: a vector normalization unit, configured to normalize the predicted surface normal vector information, generating a three-dimensional normal vector map.

[0058] Optionally, see Figure 8The specific implementation principle of generating a 3D normal vector prediction map N of the surface of the measured object by fusing and convolutionally oriented processing of the normal vector information map and specular reflection information based on the normal vector prediction module is explained as follows: When splicing and fusing the normal vector information map and specular reflection information to obtain the spliced ​​feature map, the specular reflection information formed by processing all image sequences is spliced ​​and fused with the initial normal vector information of the measured object along the channel dimension. The normal vector information map is generated by the diffuse reflection information analysis module and reflects the preliminary normal vector distribution of the surface of the measured object. The specular reflection information is extracted from the illumination feature tensor by the specular reflection information analysis module and includes key information such as specular reflection intensity. Specifically, the normal vector information map and specular reflection information are merged along the channel dimension to form a spliced ​​feature map containing the features of both, realizing the integration of multi-source information, thus obtaining the spliced ​​feature map, which fuses the normal vector and specular reflection information, and the number of channels is the sum of the number of channels of both. When performing convolution processing on the stitched feature map to obtain the primary feature map, two cascaded convolutional layers with a stride of 1 and a kernel size of 3×3 are used to perform convolution operations on the stitched feature map. This enables feature extraction and transformation of the stitched feature map, mining the potential features after the fusion of normal vectors and specular reflection information, enhancing the feature representation capability, and thus obtaining the primary feature map. This further refines the effective information in the stitched feature map, preparing for subsequent processing. When performing deconvolution operations on the primary feature map to upsample the feature resolution and obtain the full-size feature map, a deconvolutional layer with a stride of 2 and a kernel size of 3×3 is used. This upsampling of the primary feature map increases the feature resolution to the same size as the input image sequence, restoring image details and making the feature map size suitable for subsequent prediction requirements, resulting in a full-size feature map with the same spatial resolution as the input image sequence, facilitating pixel-by-pixel normal vector prediction. When performing convolution processing on the full-size feature map to predict the normal vector information of the surface of the object under test, two cascaded convolutional layers with a stride of 1 and a kernel size of 3×3 are used. By performing convolution processing on the full-size feature map, regression prediction is achieved to predict the normal vector information of each pixel on the surface of the object under test. Each pixel corresponds to a three-dimensional normal vector. When normalizing the predicted normal vector information and outputting the three-dimensional normal vector map of the object under test, the vector normalization unit normalizes each normal vector to ensure that the magnitude of the vector is 1, so that the normal vector conforms to the physical definition and obtains a standard three-dimensional normal vector to form the three-dimensional normal vector map of the object under test. The normal vector of each pixel is a normalized unit vector, which accurately reflects the directional characteristics of the surface.

[0059] In summary, this application's normal vector prediction module employs five core steps—information fusion, convolution, feature extraction, resolution recovery, regression, and vector normalization—to deeply fuse the normal vector information map with specular reflection information. By combining convolution and deconvolution operations, it achieves the transformation from multi-source information to an accurate 3D normal vector prediction map. The clear division of labor and synergistic effect of each unit effectively improves the accuracy and reliability of normal vector prediction, providing a crucial foundation for subsequent tasks such as depth prediction.

[0060] Optionally, based on the depth prediction module, iterative tensor optimization processing is performed on the normal vector information map and specular reflection information to generate a depth prediction result map of the measured object. This includes: concatenating and fusing the normal vector information map and specular reflection information to obtain a fused feature representation map; performing convolution processing on the fused feature data to obtain a fused feature map, and then fusing it in parallel with the primary feature map along the channel direction to obtain a first parallel feature map; performing convolution processing on the first parallel feature map to obtain an enhanced feature map, and then concatenating it in parallel with the full-size feature map along the channel direction to obtain a second parallel feature map; performing deconvolution layer processing on the second parallel feature map to scale the resolution to the same size as the image sequence, and then concatenating it in parallel with the predicted surface normal vector information along the channel direction to obtain a third parallel feature map; performing convolution calculation on the third parallel feature map to obtain a depth feature representation map; and performing convolution processing on the depth feature representation map to obtain the depth prediction result map of the measured object. Optionally, the depth prediction module includes an information fusion unit configured to concatenate the normal vector information map and specular reflection information along the channel dimension to generate a fused feature representation map. Optionally, the depth prediction module includes a first-level convolutional unit containing a convolutional layer, configured to perform convolution processing on the fused feature data to generate a fused feature map, and then fuse it in parallel with the primary feature map along the channel direction to obtain a first parallel feature map. Optionally, the depth prediction module includes a second-level convolutional unit containing a convolutional layer, configured to perform convolution processing on the first parallel feature map to generate an enhanced feature map, and then fuse it in parallel with the full-size feature map along the channel direction to obtain a second parallel feature map. Optionally, the depth prediction module includes a resolution adjustment and fusion unit containing a deconvolutional layer, configured to perform deconvolution processing on the second parallel feature map, scaling the resolution to the same size as the image sequence, and then concatenating it in parallel with the predicted surface normal vector information along the channel direction to obtain a third parallel feature map. Optionally, the depth prediction module includes a third-level convolutional unit containing a convolutional layer, configured to perform convolution calculation on the third parallel feature map to generate a depth feature representation map. Optionally, the depth prediction module includes a depth output unit containing a convolutional layer configured to perform convolutional processing on the depth feature representation map to generate a depth prediction result map of the object being measured.

[0061] Preferably, in a specific scenario, such as Figure 9As shown, the specific implementation details of generating the depth prediction result map D of the measured object by iterative tensor optimization processing of the normal vector information map and specular reflection information based on the depth prediction module are explained below. When splicing and fusing the normal vector and specular reflection information, the normal vector information map and specular reflection information are spliced ​​along the channel dimension to generate a fused feature representation map. For example, the normal vector information map is generated by diffuse reflection analysis and has 3 channels; the specular reflection information is generated by specular reflection analysis and has one or more channels. By splicing along the channel dimension, a fused feature representation map containing geometric and optical properties is formed (e.g., the number of channels is 3+1=4). In the first convolution process and parallel connection with the primary features, the fused feature representation map is subjected to convolution calculation with a stride of 1 and a convolution kernel size of 3×3 to obtain a feature tensor with 128 channels. This tensor is then connected in parallel along the channels with the first step process tensor of normal vector prediction (the primary feature map, which comes from the primary convolution unit output of the normal vector prediction module and has 64 channels, for example) to obtain the first parallel feature map (128+64=192 channels). When performing the second convolutional processing and paralleling with the full-size feature map, the first parallel feature map is convolved with a stride of 1 and a kernel size of 33 to obtain an enhanced feature map with 128 channels. This enhanced feature map is then convolved in parallel with the second-step process tensor of normal vector prediction (full-size feature map, from the deconvolution layer output of the normal vector prediction module, with 64 channels) to obtain the second parallel feature map (128 + 64 = 192 channels). When performing deconvolution processing and paralleling with the normal vector prediction result, the second parallel feature map is deconvolved with a stride of 2 and a kernel size of 33 to scale the resolution to the input image size, reducing the number of channels to 64. This is then convolved in parallel with the third-step process tensor of normal vector prediction (predicting surface normal vector information, from the output of the normal vector prediction unit, with 3 channels) to achieve resolution adjustment and fusion unit restoring spatial resolution through deconvolution, matching the input image size, and strengthening the guiding role of geometric constraints on depth, thus obtaining the third parallel feature map (64 + 3 = 67 channels). During the third convolution process and depth feature extraction, the third parallel feature map is subjected to convolution with a stride of 1 and a kernel size of 33. By compressing the number of channels through convolution, the core features directly related to depth (such as depth cues related to surface curvature and reflection intensity) are extracted, resulting in a depth feature representation map with 64 channels.

[0062] In summary, the depth prediction module, through multi-stage feature fusion and iterative tensor optimization, progressively integrates multi-level features (primary feature map → full-size feature map → predicted normal vector information) in the normal vector prediction process, combined with specular reflection information, to achieve a gradual improvement in depth details. Its advantages include: 1. Cross-stage feature parallelism: fusing intermediate results of normal vector prediction at different resolution levels avoids the information limitations of single-stage features; 2. Physical constraint guidance: gradually incorporating geometric priors into depth regression through the gradual introduction of normal vector prediction results, improving the physical rationality of the results; 3. Resolution adaptive adjustment: dynamically matching the input image size using deconvolution, balancing computational efficiency and detail accuracy.

[0063] The above explanation uses the example of applying the model for inference after model training is complete. To effectively ensure the accuracy of the model's prediction results, this application specifically designs a loss function to guide the model training, as detailed below.

[0064] Regarding the design of the network loss function, this application uses the Mean Squared Error (MSE) loss function as shown in Equation 1.1 to calculate the accuracy of the network in predicting the normal vector.

[0065]

[0066] In the formula N i pred For network output, N i gt For the actual optimization target, n is the number of pixels in the image.

[0067] In addition, this application also constrains the output of the diffuse reflection information analysis module in the network according to the mean square error loss function shown in Equation 1.2, so as to ensure that the network specular reflection information analysis module of this application can be driven correctly according to the BlinnPhong model constraints.

[0068]

[0069] n: Represents the number of pixels in the image, used to average the error of all pixels, making the loss value more representative. This represents the predicted normal vector corresponding to the i-th pixel in the image, output by the diffuse reflection information analysis module. This is the ground truth vector corresponding to the i-th pixel in the image, used as a reference standard to evaluate the accuracy of the prediction result. ||·||2: This is the L2 norm, used here to calculate the predicted normal vector. With the true normal vector The Euclidean distance between them measures the degree of difference between the two. The square of the difference is then calculated. It can amplify the error gap, making the loss function more sensitive to larger errors. This represents the summation of the normal vector errors of all n pixels in the image, resulting in the total error of all pixels. This refers to the mean square error loss value output by the diffuse reflection information analysis module. By minimizing this loss value, the parameters of the diffuse reflection information analysis module can be optimized, making its output normal vector closer to the true value.

[0070] In addition, this application uses the root mean square error (RMSE) loss function as shown in Equation 1.3 to calculate the accuracy of the network for depth prediction.

[0071]

[0072] n: Represents the number of pixels in the image. It is used to average the depth error of all pixels, so that the loss value can reflect the overall depth prediction bias. This represents the predicted depth value corresponding to the i-th pixel in the image, as output by the network. This is the ground truth value corresponding to the i-th pixel in the image, used as a reference standard to measure the accuracy of the network's depth prediction. ||·||2: This is the L2 norm, used here to calculate the predicted depth. With true depth Calculate the Euclidean distance between them and assess the degree of difference. First, square the difference. It can amplify the error gap, making the loss function more sensitive to larger errors. This represents the summation of the depth errors of all n pixels in the image, yielding the total depth error of all pixels. D This refers to the root mean square error loss value. By minimizing this loss value, the parameters related to depth prediction in the network can be optimized, making the depth value output by the network closer to the true depth. The square root operation makes the penalty for outliers (large errors) in this loss function more moderate than that for the root mean square error (MSE).

[0073] Considering the unequal magnitude of the errors between the normal vector map and the depth map, after multiple experimental tests, this application finally adopted the weighted loss function shown in Equation 1.4 to assist the neural network in training and optimization.

[0074]

[0075] In addition, in order to form the input image sequence, such as Figure 10.1 and 10.2This application also provides a device design drawing and actual structure, which mainly consists of a camera, a light source, a central rotary table, and four motors and their reduction mechanisms that control the movement of each component.

[0076] First, to meet the requirements of photometric stereo vision for changes in the zenith and azimuth angles of incident and outgoing rays, the camera is positioned at the center of the outermost swing arm, with its shooting direction perpendicular to the arm and pointing inwards. Motor 1 controls the camera swing arm to swing back and forth, thereby controlling the camera's shooting angle, i.e., the zenith angle of the outgoing rays. The light source is located at the end of the light source cantilever arm on the inner side of the equipment, with its illumination direction perpendicular to the arm and pointing inwards. Motor 2 controls the light source cantilever arm to swing back and forth. Simultaneously, motor 3 controls motor 2 and the light source cantilever arm as a whole to rotate around the center of the equipment. By controlling the movement of the light source cantilever arm, the illumination angle of the equipment's light source, i.e., the zenith and azimuth angles of the incident rays, can be controlled. The equipment's central rotary table is used to place the object being measured. Motor 4 controls the rotary table to rotate around the center of the equipment. Rotating the central rotary table can simultaneously change the illumination angle of the light source and the camera's shooting angle towards the target, i.e., the azimuth angles of the incident and outgoing rays. This establishes all four degrees of freedom of the photometric stereoscopic vision measurement device, as well as the zenith angle and azimuth angle of the incident and outgoing rays of the observed object.

[0077] Based on this, to ensure independent control of each angle, during the assembly of the main body of the equipment, the rotation center of the camera arm controlled by motor 1 and the rotation center of the cantilever controlled by motor 2 must be aligned coaxially; similarly, the rotation center of the cantilever controlled by motor 3 and the rotation center of the central rotary table controlled by motor 4 must also be aligned coaxially. This ensures that only the zenith and azimuth angle information controlled by each axis changes during the movement of each axis, preventing angle information coupling.

[0078] Finally, regarding the device size design, due to the limitations of the experimental site, the shooting distance D between the camera and the target in this application is approximately...

[0079] 500mm. Furthermore, considering that the algorithm in this application requires near-field point illumination, the light source illumination distance is set to approximately d.

[0080] The diameter of the central rotary table is 250mm, the size of the object being measured is approximately 50mm, the diameter of the central rotary table is 200mm, and the height of the central rotary table can be manually adjusted ±20mm via the scissor mechanism.

[0081] Regarding the selection of camera and lens, this application uses the Hikvision MVCA05012UC camera with a resolution of 2592×2048 pixels and a camera pixel size of 3.45μm. To ensure sufficient margin for the placement of the test object during the data set construction process, the camera's field of view L is set to 100mm. According to Equation 1.5, the ideal camera lens focal length is approximately 35.33mm. Therefore, the Hikvision MVLHF3524M10MP camera lens with a focal length of 35mm is selected.

[0082]

[0083] f: Represents the focal length of the camera lens, in mm (millimeters). It is the target value in this formula and is used to determine a suitable camera lens. D: Represents the shooting distance between the camera and the target, in mm (millimeters). The text mentions that the experimental setting for the shooting distance D between the camera and the target is approximately 500mm. 2048: This is one dimension (2592×2048 pixels) of the selected camera's resolution (Hikvision MVCA05012UC camera). This value is used in the calculation based on the camera's imaging principle and the relationship between parameters such as field of view size and focal length. 3.45: Camera pixel size, in μm (micrometers). It reflects the size of the camera's pixel unit and is an inherent attribute parameter of the camera. L: Camera field of view size, in mm (millimeters). In the text, it is set to 100mm to ensure sufficient margin for the placement of the test object during dataset construction. 1000: This is a unit conversion factor used to convert the pixel size unit from μm to mm, ensuring consistency of units in the formula and guaranteeing calculation accuracy.

[0084] Regarding the selection of the light source, this application uses the Lubang Optoelectronics LBL2164WDG white point light source with an illuminance range of 1700-2300 Klux. The illuminance of the light source is controlled by a constant current controller, and the divergence angle θ of the light source is 15°. According to Equation 1.6, the effective illumination range diameter of the light source on the central rotary table under the working state of the equipment is about 135mm, which is larger than the set camera field of view L, and meets the experimental requirements.

[0085] D l = 2d·tan(θ) (1.6)

[0086] In formula D l =2d·tan(θ) in: D l: This represents the diameter of the effective illumination range of the light source on the central rotary table when the equipment is operating, in mm (millimeters). It is the result of the formula and is used to evaluate whether the illumination range of the light source meets the experimental requirements. d: This represents the illumination distance of the light source, in mm (millimeters). The text mentions that the illumination distance of the light source in the experiment was set to approximately d = 250 mm. θ: This is the divergence angle of the light source, in degrees (°). The divergence angle θ of the Lubang Optoelectronics LBL2164WDG white point light source used in the text is 15°.

[0087] Regarding motor selection, this application first sets and measures the technical parameters of each motion axis of the photometric stereo vision measurement device, and the results are shown in Table 1.1.

[0088] Table 1.1 Technical parameters of each motion axis of the equipment

[0089]

[0090] Regarding the camera arm axis, the magnitude of its angular acceleration during acceleration is shown in Equation 1.7.

[0091]

[0092] At this point, the motor needs to provide torque as shown in Equation 1.8.

[0093] M1=Ta=1.375·17.453=23.998N·m (1.8)

[0094] Considering the effect of gravity during the motion, we take the distance d0 from the center of gravity of the swing arm shaft to the rotation center of the swing arm shaft in the extreme state as 0.5m. At this time, the rotation of the swing arm shaft requires the motor to overcome the torque of gravity as shown in Equation 1.9.

[0095] M2=mg·d0=5·9.8·0.5=24.5N·m (1.9)

[0096] Based on the above calculation results, this application selects Leadshine ACM1H0808BMS10B motor for the swing arm shaft of the equipment. Assuming the motor transmission coefficient η = 0.9, the output torque provided by the motor during the movement is as shown in Equation 1.10.

[0097] M=M0·ε·η=2.39·30·0.9=64.53N·m (1.10)

[0098] The rated torque of the motor is greater than the sum of the torques required for the movement of the swing arm shaft, and the motor parameters meet the requirements.

[0099] Formula (1.7)a: Angular acceleration during the acceleration process of the camera arm axis, in rad / s². 2 (radians / second) 2), used to measure the rate of change of angular velocity during the acceleration of the camera arm axis. 100: The speed limit value of the camera arm axis, in ° / s (degrees / second), representing the maximum allowable rotational speed of the arm axis. Here, it is converted to radians for calculating angular acceleration. 2π: Used to convert the angle unit from degrees to radians, because 360° corresponds to 2π radians. 360: The degree of one revolution in the angle system, used to help convert the speed limit value from degrees / second to radians / second. 0.1: The uniform acceleration time of the camera arm axis, in s (seconds), representing the time it takes for the arm axis to accelerate from rest to the speed limit value. Formula (1.8) M1: The torque required by the motor to accelerate the camera arm axis, in N·m (Newton-meter). T: A parameter related to the camera arm axis, calculated to be 1.375, which is the moment of inertia parameter. a: The angular acceleration during the acceleration process of the camera arm axis, in rad / s 2 (radians / second) 2 The result is derived from formula (1.7). Formula (1.9) M2: The gravitational torque that the motor needs to overcome when the swing arm shaft rotates, in N·m (Newton-meter). m: The load mass of the swing arm shaft, in kg (kilogram), taken as 5 kg in this paper. g: Gravitational acceleration, taken as 9.8 m / s². 2 d0: The distance from the center of gravity of the swing arm shaft to the center of rotation of the swing arm shaft under extreme conditions, in meters (m), with a value of 0.5m. Formula (1.10) M: The torque output by the motor through the transmission system during motion, in Newton-meters (N·m). M0: The rated torque of the motor, with a value of 2.39. ε: A parameter related to the motor transmission, with a value of 30 in this paper. η: The motor transmission coefficient, dimensionless, with a value of 0.9 in this paper, used to consider the efficiency factor in the motor transmission process.

[0100] Regarding the cantilever shaft of the light source, the magnitude of its angular acceleration during acceleration is shown in Equation 1.11.

[0101]

[0102] At this point, the motor needs to provide torque as shown in Equation 1.12.

[0103] M1=Ta=0.156·17.453=2.723N·m(1.12)

[0104] Considering the effect of gravity during the motion, we take the distance d1 from the center of gravity of the cantilever shaft to the rotation center of the cantilever shaft in the extreme state as 0.25m. At this time, the rotation of the cantilever shaft requires the motor to overcome the torque of gravity as shown in Equation 1.13.

[0105] M2=mg·d1=2·9.8·0.25=4.9N·m(1.13)

[0106] Based on the above calculation results, this application selects Leadshine ACM1H0401BMS10B motor for the cantilever shaft of the equipment. Assume that the motor transmission coefficient η1=η2=0.9, and the rated torque provided by the motor during the movement is as shown in Equation 1.14.

[0107] M=M0·ε·η1·η2=0.32·150·0.9·0.9=38.88N·m(1.14)

[0108] The rated torque of the motor is greater than the sum of the torques required for the movement of the cantilever shaft, and the motor parameters meet the requirements.

[0109] Regarding the rotation axis of the light source, the magnitude of its angular acceleration during acceleration is shown in Equation 1.15.

[0110]

[0111] At this point, the motor needs to provide torque as shown in Equation 1.16.

[0112] M1=Ta=0.244·62.832=15.331N·m(1.16)

[0113] Since the axis of motion is parallel to the direction of gravity during the motion process, the influence of the equipment's gravity on the motor's torque is not considered.

[0114] a: Angular acceleration during the acceleration process of the light source cantilever shaft, in rad / s². 2 (radians / second) 2 ), used to measure the rate of change of angular velocity during the acceleration of the cantilever shaft. 100: The speed limit value of the light source cantilever shaft, in degrees / second (° / s), representing the maximum allowable rotational speed of the cantilever shaft. Here, it is converted to radians for calculating angular acceleration. 2π: Used to convert the angle unit from degrees to radians, since 360° corresponds to 2π radians. 360: The degree measure of one revolution in the angle system, used to help convert the speed limit value from degrees / second to radians / second. 0.1: The uniform acceleration time of the light source cantilever shaft, in seconds (s), representing the time it takes for the cantilever shaft to accelerate from rest to the speed limit value. M1: The torque required by the motor to accelerate the light source cantilever shaft, in Newton-meters (N·m). T: A parameter related to the light source cantilever shaft, which is 0.156. a: The angular acceleration during the acceleration process of the light source cantilever shaft, in rad / s. 2 (radians / second) 2 The values ​​are derived from the calculation results of formula (1.11). M2: The gravitational torque that the motor needs to overcome when the cantilever shaft rotates, in N·m (Newton-meter). m: The load mass of the cantilever shaft, in kg (kilograms), taken as 2 kg. g: Gravitational acceleration, taken as 9.8 m / s². 2d1: The distance from the center of gravity of the cantilever shaft to the center of rotation of the cantilever shaft under extreme conditions, in meters (m), the value taken in this paper is 0.25m. M: The output torque provided by the motor during the motion, in Newton-meters (N·m). M0: Basic parameter, the value taken in this paper is 0.32. ε: Parameter related to motor transmission, the value taken in this paper is 150, not specified. η1, η2: Motor transmission coefficients, both taken as 0.9, used to consider factors such as efficiency in the motor transmission process. a: Angular acceleration during the acceleration process of the light source rotation shaft, in rad / s. 2 (radians / second) 2 ), used to measure the rate of change of angular velocity during the acceleration of the rotating shaft. 360: The speed limit value of the light source rotating shaft, in degrees per second (° / s), representing the maximum allowable rotational speed of the rotating shaft. Here, it is converted to radians for calculating angular acceleration. 2π: Used to convert the angle unit from degrees to radians, because 360° corresponds to 2π radians. 360: The degree measure of one revolution in the angle system, used to help convert the speed limit value from degrees / second to radians / second. 0.1: The uniform acceleration time of the light source rotating shaft, in seconds (s), representing the time it takes for the rotating shaft to accelerate from rest to the speed limit value. M1: The torque required by the motor to accelerate the light source rotating shaft, in Newton-meters (N·m). T: A parameter related to the light source rotating shaft, which is 0.244. a: The angular acceleration during the acceleration process of the light source rotating shaft, in rad / s. 2 (radians / second) 2 ), which is the result of the calculation from formula (1.15).

[0115] Based on the above calculation results, this application selects Leadshine ACM1H0604BMS10B motor for the rotating shaft of the equipment light source. Assuming the motor transmission coefficient η = 0.9, the rated torque provided by the motor during the movement is as shown in Equation 1.17.

[0116] M=M0·ε·η=1.27·30·0.9=34.29N·m(1.17)

[0117] The motor's rated torque is greater than the sum of the torques required for the rotation of the light source's shaft, and the motor parameters meet the requirements.

[0118] Regarding the central rotary table axis, the magnitude of its angular acceleration during acceleration is shown in Equation 1.18.

[0119]

[0120] At this point, the motor needs to provide torque as shown in Equation 1.19.

[0121] M1=Ta=0.0675·62.832=4.241N·m(1.19)

[0122] Since the axis of motion is parallel to the direction of gravity during the motion process, the influence of the equipment's gravity on the motor's torque is not considered. However, considering the influence of the frictional force generated by the object being measured during the rotation of the central rotary table, this application takes the static friction coefficient of the rough aluminum rotary table surface as μ = 0.6, and the distance from the friction point to the center of the rotary table as d3 = 0.1m under the extreme condition. The torque generated by the frictional force at this time is shown in Equation 1.20.

[0123] M2=μmg·d3=0.6·10·9.8·0.1=5.88N·m(1.20) M: Output torque provided by the light source rotary shaft motor during motion, in N·m (Newton-meter). M0: Rated torque of the motor, taken as 1.27. ε: A parameter related to motor transmission, taken as 30 in the text, with no clear physical meaning. η: Motor transmission coefficient, dimensionless, taken as 0.9 in the text, used to consider factors such as efficiency during motor transmission. a: Angular acceleration during the acceleration process of the central rotary table shaft, in rad / s². 2 (radians / second) 2 ), used to measure the rate of change of angular velocity during the acceleration of the rotary table axis. 360: The speed limit value of the center rotary table axis, in degrees per second (° / s), representing the maximum allowable rotational speed of the rotary table axis. Here, it is converted to radians for calculating angular acceleration. 2π: Used to convert the angle unit from degrees to radians, since 360° corresponds to 2π radians. 0.1: The uniform acceleration time of the center rotary table axis, in seconds (s), representing the time it takes for the rotary table axis to accelerate from rest to the speed limit value. M1: The torque required by the motor to accelerate the center rotary table axis, in Newton-meters (N·m). T: A parameter related to the center rotary table axis, 0.0675. a: The angular acceleration during the acceleration process of the center rotary table axis, in rad / s. 2 (radians / second) 2 The values ​​are derived from the calculation results of formula (1.18). M2: The torque generated by friction when the central rotary table shaft rotates, in N·m (Newton-meter). μ: The static friction coefficient of the rough aluminum rotary table surface, dimensionless, taken as 0.6 in this paper, reflecting the friction characteristics of the contact surface. m: The mass of the central rotary table and the object under test, etc., in kg (kilogram), taken as 10 kg in this paper. g: Gravitational acceleration, taken as 9.8 m / s^{2}, used to calculate the magnitude of gravity. d3: The distance from the point of friction force application to the center of the rotary table under the ultimate condition, in m (meter), taken as 0.1 m in this paper. Based on the above calculation results, this application selects the Leadshine ACM1H0604BMS10B motor for the central rotary table shaft of the equipment. Assuming the motor transmission coefficient η = 0.9, the rated torque provided by the motor during the movement is shown in formula 1.21.

[0124] M=M0·ε·η=1.27·30·0.9=34.29N·m (1.21)

[0125] The rated torque of the motor is greater than the sum of the torques required for the movement of the central rotary table shaft, and the motor parameters meet the requirements.

[0126] Thus, the selection of key components for the photometric stereo vision measurement equipment has been completed. Based on this, the overall equipment adopts a darkroom design to prevent the influence of external light, and uses a 485 communication interface to enable the host computer to control the equipment's four degrees of freedom motion and camera shooting actions.

[0127] The photometric stereo vision device designed in this application mainly comprises three coordinate systems: the central rotary table coordinate system, the device coordinate system, and the camera coordinate system. The central rotary table coordinate system is used to assist in the localization of the real object being measured and serves as the medium for constructing the real dataset. The device coordinate system is used for the overall design and manufacturing of the device; the positioning, assembly, and movement of each axis, as well as the subsequent construction of the virtual dataset, are all based on the device coordinate system. The camera coordinate system is used to calculate and represent the surface normal vector and depth information of the object being measured, and is the representation benchmark for the final prediction results of the photometric stereo vision technology. Since the photometric stereo vision technology uses the camera coordinate system as the benchmark for prediction output, before constructing the dataset, it is necessary to calibrate the transformation relationship between the coordinate system containing the dataset and the camera coordinate system. This allows for the calculation of the true normal vector and depth information of the object being measured in the camera coordinate system, enabling the training and accuracy testing of the deep learning network. Therefore, this application focuses on the calibration process between the rotary table coordinate system, the device coordinate system, and the camera coordinate system. The rotation axis of the central rotary table is defined as the z-axis of the equipment coordinate system, pointing vertically upwards. The rotation axis of the camera arm is defined as the x-axis of the equipment coordinate system, pointing horizontally towards the motor. According to the right-hand rule, the y-axis of the equipment coordinate system is determined to be horizontally pointing inwards towards the equipment, and the origin of the equipment coordinate system is determined to be the intersection of the z-axis and the x-axis. It should be noted that since the rotary table moves vertically, the origin of the equipment coordinate system may not coincide with the center of the central rotary table. Using the photometric stereo vision measurement equipment employed in this application, this application first uses a camera calibration plate as an intermediate medium to calculate the coordinate transformation relationship between the central rotary table coordinate system and the camera coordinate system. The central rotary table coordinate system uses the rotation axis of the rotary table as the z-axis, and its x and y axes are consistent with the x and y axes of the camera coordinate system, with the center of the central rotary table surface as the origin. To further establish the relationship between the equipment rotary table coordinate system and the camera coordinate system, this application places a calibration plate of appropriate size on the equipment rotary table. By rotating the turntable of the equipment, a sequence of calibration plate images rotating around the z-axis of the equipment can be obtained, such as... Figure 11.1 As shown. Using a camera calibration correlation algorithm, this application can calculate the coordinates of each calibration board corner point in the image sequence in the camera coordinate system.

[0128] After obtaining the coordinates of all calibration plate corner points, this application first uses the least squares method to fit and calculate the planar information of all corner points. This application assumes that the plane equation of the target plate in space is as shown in Equation 1.22.

[0129] Ax + By + Cz + D = 0 (1.22)

[0130] Since the calibration plate plane is in the camera coordinate system space and satisfies C≠0, the above formula can be rewritten as shown in Equation 1.23.

[0131] z = a0x + a1y + a2 (1.23)

[0132] in

[0133]

[0134] When there are more than three spatial points that need to be fitted, due to the existence of identification and calculation errors, these spatial points are not completely in the same plane. At this time, the least squares objective function can be obtained, as shown in Equation 1.25.

[0135]

[0136] When the objective function is satisfied, we have

[0137]

[0138] Substituting the values, we obtain the parameter solution equation for the target plane, as shown in Equation 1.27.

[0139]

[0140] After solving for the plane parameters, the normal vector of the rotary table plane can be obtained as shown in Equation 1.27.

[0141] n h =[a0 a1 -1] T (1.27)

[0142] Based on this, during the rotation of the calibration plate, the motion position of each corner point of the calibration plate is fitted by least squares to calculate the rotation center of the corner point.

[0143] A, B, C, D: Coefficients of the plane equation Ax + By + Cz + D = 0, used to determine the position and orientation of the plane in space. These coefficients describe the calibration plate plane in the camera coordinate system and establish the relationship between the plane and coordinate points. x, y, z: Three-dimensional coordinate components in the camera coordinate system, representing the position of a point in space in three dimensions. z: The coordinate value of a point in the camera coordinate system along the z-axis. x, y: The coordinate values ​​of a point in the camera coordinate system along the x-axis and y-axis. a1, a2: Coefficients of the transformed plane equation, obtained by... The coefficients A, B, C, and D of the original equation are transformed to represent the plane equation of the calibration plate in another form. a0: Coefficients calculated from the coefficients A and C of the original plane equation. The transformed plane equation z = a0x + a1y + a2 reflects the influence weight of the x-coordinate on the z-coordinate. a1: A coefficient calculated from the original plane equation coefficients B and C. The transformed plane equation reflects the influence weight of the y-coordinate on the z-coordinate. a2: Coefficients calculated from the original plane equation coefficients D and C. In the transformed plane equation, it is a constant term that affects the plane's intercept along the z-axis. `min`: represents the operation of finding the minimum value, that is, by adjusting the values ​​of a0, a1, and a2, the objective function S is minimized. `S`: the objective function value of the least squares method, calculated... This reflects the degree of error between the fitted plane and the actual corner coordinates. n: The number of calibration plate corner points involved in the fitting. x i y i z i Let be the 3D coordinates of the i-th calibration board corner point in the camera coordinate system. By minimizing the objective function, find the values ​​a0, a1, and a2 that best fit the planar information of these corner points. The partial derivative of the objective function S with respect to the coefficient a0. A partial derivative of 0 indicates that the objective function S reaches an extremum in the direction of a0. The value of a0 that minimizes the objective function is determined by solving this equation. The partial derivative of the objective function S with respect to the coefficient a1. Similarly, the value of a1 that minimizes the objective function is determined by solving this equation.

[0144] The partial derivative of the objective function S with respect to the coefficient a². The value of a² that minimizes the objective function is determined by solving this equation. The sum of the squares of the x-coordinates of all calibration plate corner points reflects the distribution of x-coordinates in the data and is used to calculate the plane equation coefficient a0. ∑x i y i The sum of the products of the x and y coordinates of all calibration plate corner points reflects the relationship between the x and y coordinates and plays a role in calculating the coefficients a0 and a1 of the plane equation. ∑xi The sum of the x-coordinates of all calibration plate corner points is used in the calculation of the plane equation coefficients a0 and a2. The sum of the squares of the y-coordinates of all calibration plate corner points reflects the distribution of y-coordinates in the data and is used to calculate the coefficients a1 of the plane equation. ∑y i : The sum of the y-coordinates of all calibration plate corner points, used in the calculation of the plane equation coefficients a1 and a2. n: The number of calibration plate corner points involved in the fitting, which plays a role in calculating the plane equation coefficient a2. a0, a1, a2: The coefficients of the plane equation z = a0x + a1y + a2, obtained by solving this matrix equation. ∑x i z i The sum of the products of the x and z coordinates of all calibration plate corner points is related to the coefficient a0 of the plane equation. ∑y i z i The sum of the products of the y-coordinates and z-coordinates of all calibration plate corner points is related to the calculation of the plane equation coefficients a1. ∑z i The sum of the z-coordinates of all calibration plate corner points is related to the coefficients a2 of the plane equation. h : The normal vector of the rotary table plane, used to describe the direction of the rotary table plane. a0, a1: Coefficients of the plane equation z = a0x + a1y + a2, reflecting the tilt of the plane in the camera coordinate system, and also components of the normal vector. -1: The component of the normal vector in the z-direction, which, together with a0 and a1, determines the direction of the normal vector. [a0 a1 -1] T : Represents the transpose of a vector, converting a row vector into a column vector; it is the matrix representation of the normal vector. Assume the center c of the circle at the corner of the calibration plate during the rotation process. h =[c hx c hy c hz ] T .

[0145] At this point, taking any two coordinates of the same corner point during its rotation, we know that these two points are theoretically located on a circle in space with center C. Let the two coordinates of the corner point be p1 = [x1 y1 z1]. T p2 = [x2 y2 z2] T .

[0146] The line vector connecting the two points can be obtained as shown in Equation 1.28.

[0147] p 12 = [x2-x1 y2-y1 z2-z1] T (1.28)

[0148] Simultaneously, the vectors connecting the center of the circle to the midpoints of the two points can be obtained, as shown in Equation 1.29.

[0149]

[0150] According to the properties of a circle, vector p 12 With vector p 12,c Vertical, that is, satisfying equation 1.30.

[0151]

[0152] In the above formula, c h : Represents the coordinate vector of the center of the circle formed by the corner points of the calibration plate during rotation; it is a three-dimensional vector. hx : The coordinates of the center of the circle along the x-axis in the camera coordinate system. hy : The coordinates of the center of the circle along the y-axis in the camera coordinate system. hz : The coordinates of the center of the circle along the z-axis in the camera coordinate system. p1, p2: Represent the coordinate vectors of two different positions of the same calibration plate corner point during rotation, both being three-dimensional vectors. x1, y1, z1: ​​The coordinates of the first position of the corner point along the x, y, and z axes in the camera coordinate system. x2, y2, z2: The coordinates of the second position of the corner point along the x, y, and z axes in the camera coordinate system. p 12 : is a three-dimensional vector representing the line connecting two points determined by the coordinates p1 and p2 of the corner point. It reflects the relative positional relationship between these two points in space. Its components are the differences in the coordinate values ​​of the two points along the x, y, and z axes, respectively. p 12,c The vector connecting the center of the circle to the midpoints of the two points represents the spatial relationship between the center and these two midpoints; it is a three-dimensional vector. Its components are the differences between half the sum of the coordinates of the two points on their respective coordinate axes and the corresponding coordinates of the center. In the above formula, vector p... 12 With vector p 12,c The dot product of two vectors is 0. According to the property of perpendicular vectors, when the dot product of two vectors is 0, they are perpendicular. Here, we use the geometric property of a circle—that the line connecting the center of the circle to the midpoint of any two points on the circle is perpendicular to the line connecting those two points—to establish an equation to solve for the coordinates of the circle's center.

[0153] Based on the above rules, for a single corner point that has undergone multiple rotations, an overdetermined set of equations can be obtained as shown in Equation 1.31.

[0154] Δx (n-1)n ·x h +Δy (n-1)n ·y h +Δz (n-1)n ·z h -l (n-1)n =0 (1.31)

[0155] In the formula

[0156] Δx (n-1)n=x n -x n-1

[0157] Δy (n-1)n =y n -y n-1

[0158] Δz (n-1)n =z n -z n-1

[0159]

[0160] Meanwhile, based on the prior information that the corner points of the calibration plate lie on the same plane during rotation, the center c of the circle that needs to be solved is... h It lies within the plane of the calibration plate. Therefore, by combining Equation 1.32, we can obtain Equation 1.33.

[0161] z h =a0x h +a1y h +a2(1.33)

[0162] Right now

[0163] n h T ·c h = -a2(1.34)

[0164] Let b = [Δx] (n-1)n Δy (n-1)n Δz (n-1)n ], L=l (n-1)n

[0165] Equation 1.35 can be obtained.

[0166] b·c h =L (1.35)

[0167] Δx (n-1)n Δy (n-1)n Δz (n-1)n : These are the differences in the coordinates of the corner points of the calibration plate in the x, y, and z axes after the nth and (n-1)th rotations, respectively, used to describe the change in the corner point's position. h y h z h : The coordinates of the center of the circle to be solved in the x, y, and z axes of the camera coordinate system. (n-1)n : is a calculated value related to the coordinates of different rotational positions of the corner point, obtained through formula (1.32), and plays a constraint role in the overdetermined system of equations. n y n z n: The coordinates of the calibration plate corner point in the camera coordinate system along the x, y, and z axes after the nth rotation. n-1 y n-1 z n-1 The coordinates of the calibration plate corner point after the (n-1)th rotation in the camera coordinate system along the x, y, and z axes. These coordinate values ​​are used to calculate l. (n-1)n This reflects the relationship between different positions of the corner points. h : The coordinates of the center of the circle to be solved in the z-axis direction of the camera coordinate system. x h y h : The coordinates of the center of the circle to be solved in the x and y directions of the camera coordinate system. a0, a1, a2: The coefficients of the calibration plate plane equation z = a0x + a1y + a2, used to establish the relationship between the center coordinates and the plane equation, since the center of the circle is located in the calibration plate plane. Rotary table plane normal vector n h The transpose of converts a column vector into a row vector, which is used for vector dot product operations. h The coordinate vector of the center of the circle to be solved [x] h y h z h ] T By combining the dot product with the normal vector and the coefficients of the plane equation, the position of the circle center is further constrained. -a2: The constant term in the plane equation is negative, which is the result of the dot product operation and is used to establish the equation to solve for the coordinates of the circle center.

[0168] Combining equations 4.31 and 4.32, and introducing the Lagrange operator λ, the objective function is obtained as shown in equation 1.36.

[0169]

[0170] Differentiating with respect to and respectively, we obtain the solution equations for the center and the Lagrange operator as shown in Equation 1.37.

[0171]

[0172] This allows us to determine the coordinates of the center of the trajectory formed by a single corner point on the calibration plate after the calibration plate completes its rotation. This application iteratively solves for the trajectories formed by all corner points on the calibration plate, obtaining a series of center positions fitted to all corner points. The application then calculates the average value of these center coordinates and uses this average value as the rotation center during the calibration plate's rotation process.

[0173] min: indicates the operation of finding the minimum value, that is, by adjusting c. h The values ​​of λ and c are used to minimize the objective function C. C: The objective function value after introducing the Lagrange operator. c: Here, c represents the coordinate vector of the circle's center. hλ is one of the variables to be optimized. λ: Lagrange operator, used to apply constraints (the center of the circle lies in the calibration plate plane, i.e.) Incorporate it into the objective function. ||bc h -L|| 2 : Represents vector bc h The square of the L2 norm of -L. Where b is determined by Δx (n-1)n Δy (n-1)n Δz (n-1)n The matrix formed, c h It is the coordinate vector of the center of the circle, and L is formed by l. (n-1)n The vector formed by this term reflects the fitting error of the overdetermined system of equations. Rotary table plane normal vector n h The transpose of the vector converts the column vector into a row vector, which is used to intersect the center coordinate vector c. h Perform the dot product operation. a2: The constant term in the equation z = a0x + a1y + a2 of the calibration plate plane, which plays a role in the constraints. b T b: The product of the transpose of matrix b and itself is a square matrix, used in solving the equation to construct the coordinate vector c around the center of the circle. h The relevant coefficient matrix part. Rotary table plane normal vector n h The transpose of is used as part of the coefficient matrix in the equation to reflect the constraint that the center of the circle lies in the plane of the calibration plate. h The coordinate vector of the center of the circle to be solved [x] h y h z h ] T λ: The Lagrange operator, one of the variables to be solved. T L: The product of the transpose of matrix b and vector L, which is part of the vector on the right side of the equation. -a2: The negative of the constant term in the equation of the calibration plate plane, which is part of the vector on the right side of the equation, reflecting the constraint condition that the center of the circle lies in the calibration plate plane.

[0174] Thus, this application has calculated the normal vector n of the equipment rotary table plane. h And the center point c of the rotary table's rotation. h =[c hx c hy c hz ] T .

[0175] For convenience, this application sets the x-axis direction of the equipment rotary table coordinate system to be consistent with the camera direction; therefore, the x-axis vector of the equipment rotary table is denoted as x. h =[1 0 0] T .

[0176] After normalizing the plane normal vector of the equipment turntable, we obtain the z-axis vector of the equipment turntable coordinate system. The cross product of this vector yields the y-axis vector of the equipment turntable, and thus the unit direction vectors of the x, y, and z axes of the equipment turntable coordinate system in the camera coordinate system, denoted as xi, yj, and zz, respectively. h y h z h .

[0177] Considering that the calibration plate itself has a thickness of 5mm, the transformation matrix from the equipment rotary table coordinate system to the camera coordinate system is finally obtained as shown in Equation 1.38.

[0178]

[0179] Regarding n h n h The normal vector of the rotary table plane describes the direction of the rotary table plane and plays a crucial role in determining the z-axis direction of the rotary table coordinate system. Regarding c... h =[c hx c hy c hz ] T c h : The coordinate vector of the rotation center point of the rotary table, representing the position of the rotary table's rotation center in the camera coordinate system. hx c hy c hz These are the coordinates of the center point along the x, y, and z axes of the camera coordinate system, respectively. Regarding x... h =

[100] T x h : The x-axis vector of the rotary table coordinate system, whose direction is set to be consistent with the camera direction, is the unit direction vector of the x-axis of the rotary table coordinate system. Regarding x... h y h z h , which are the unit direction vectors of the x, y, and z axes of the equipment rotary table coordinate system in the camera coordinate system. Where z h It is formed by the normal vector n of the rotary table plane. h After normalization, we get y h Through x h With z h The cross product yields the coordinates, which together determine the orientation of the rotary table coordinate system. Regarding formula (1.38)T... hc T hc : The transformation matrix from the equipment rotary table coordinate system to the camera coordinate system, used to achieve coordinate transformation between the two coordinate systems. hx x hy x hz Unit direction vector x along the x-axis of the rotary table coordinate system hComponents along the x, y, and z axes of the camera coordinate system. y hx y hy y hz : Unit direction vector of the y-axis in the rotary table coordinate system h Components in the x, y, and z axes of the camera coordinate system. hx z hy z hz Unit direction vector z along the z-axis of the rotary table coordinate system h Components in the x, y, and z axes of the camera coordinate system. hx c hy c hz : Center point of rotation of the rotary table c h The coordinate values ​​along the x, y, and z axes in the camera coordinate system. In the third row and third column of the matrix, c hz +5 is an adjustment to the z-axis coordinate considering the calibration plate thickness of 5mm. The fourth row of the matrix [0 0 0 1] is the standard form of homogeneous coordinates, used to ensure the homogeneity of coordinate transformations.

[0180] The device coordinate system is the reference coordinate system for constructing a photometric stereo vision device. In the subsequent construction of the virtual synthetic dataset, this device coordinate system is needed as a reference to determine camera capture, light source illumination, and the pose of the object being measured. Therefore, the transformation relationship between the device coordinate system and the camera coordinate system needs to be calibrated to meet the requirements of subsequent virtual dataset construction. In defining the central rotary table coordinate system, the z-axis of both the central rotary table coordinate system and the device coordinate system is set as the rotation center of the rotary table. Therefore, this application directly uses the plane normal vector as the z-axis coordinate vector of the device rotary table. However, since the device rotary table can be adjusted vertically, and the x-axis is defined to be consistent with the camera x-axis, while the origin and x-axis of the device coordinate system are defined through the motion relationships between the axes, the origin and x-axis of the device coordinate system need further calculation. This application still places the calibration plate on the device rotary table, keeps other motion axes of the device stationary, rotates the camera axis, and obtains a sequence of calibration plate images captured by the camera rotating around the x-axis of the device coordinate system, such as... Figure 11.2 As shown. At this point, the coordinate information of the calibration board corner points in each image of the image sequence in the camera coordinate system is calculated, and the calibration board coordinate system is constructed. This allows for the reverse calculation of the coordinate sequence p of the camera coordinate system origin in the calibration board coordinate system for each image. n =[x n y n z n ] T .

[0181] Similarly, this application uses the least squares method to fit the coordinates of the origin of the camera coordinate system in the calibration board coordinate system of the image sequence, to obtain the rotation center point p0 of the camera motion and the motion plane normal vector n0 in the calibration board coordinate system. Based on this, this application records the transformation relationship between the calibration board coordinate system and the camera coordinate system under the zero-return state of each axis of the device, transforming the camera motion rotation center point and motion plane normal vector in the calibration board coordinate system to the current camera coordinate system, and then using the transformation result of the rotation center point p0 as the x-axis origin c of the device coordinate system in the camera coordinate system. c =[c cx c cy c cz ] T The transformation result of the plane normal vector n0 is used as the x-axis direction vector x in the device coordinate system under the camera coordinate system. m Based on the above calculation results, and according to the device coordinate system z-axis direction vector n obtained in this application... h Combined with the unit direction vector x of the device coordinate system x-axis obtained in this application m This application normalizes the two direction vectors and then performs a cross product to obtain the unit direction vector y along the y-axis in the device coordinate system under the camera coordinate system. m Furthermore, considering potential calculation errors, the device in the camera coordinate system is positioned with respect to the rotation center c of the rotary table. h =[c hx c hy c hz ] T The z-axis line with the origin intersects with the camera rotation center c. c =[c cx c cy c cz ] T The x-axis line, serving as the origin, may be two non-intersecting lines in space. In such cases, the common perpendicular segment between the z-axis and x-axis is further calculated, and the origin of the device coordinate system in the camera coordinate system is updated to the midpoint c of this common perpendicular segment. m =[c mx c my c mz ] T Thus, all information about the origin, x, y, and z axes of the device coordinate system in the camera coordinate system is obtained. Therefore, the transformation matrix between the device coordinate system and the camera coordinate system can be calculated as shown in Equation 4.36, realizing the coordinate transformation between the two coordinate systems.

[0182]

[0183] Regarding c h =[c hx c hy chz ] T c h The coordinate vector of the rotation center of the equipment's rotary table in the camera coordinate system is used as the coordinate vector in the equipment coordinate system.

[0184] The origin, with the rotary table as the reference point. hx c hy c hz These are the coordinates of the rotary table's rotation center along the x, y, and z axes of the camera coordinate system. Regarding c... c =[c cx c cy c cz ] T Chinese: c c : The coordinate vector of the camera's rotation center in the camera coordinate system, which is an important reference point in the camera coordinate system. cx c cy c cz These are the coordinates of the camera's rotation center along the x, y, and z axes of the camera coordinate system, respectively. Regarding c... m =[c mx c my c mz ] T Chinese: c m The origin of the device coordinate system after updating to the camera coordinate system is the midpoint of the common perpendicular segment between the z-axis line of the rotary table and the x-axis line of the camera. mx c my c mz : These are the updated coordinates of the origin along the x, y, and z axes of the camera coordinate system. Regarding formula (1.39)T mc Chinese: T mc : The transformation matrix between the device coordinate system and the camera coordinate system, used to achieve accurate coordinate transformation between the two coordinate systems. x mx x my x mz : The components of the unit direction vector of the device coordinate system along the x, y, and z axes in the camera coordinate system. mx y my y mz : The components of the unit direction vector of the y-axis in the device coordinate system along the x, y, and z axes in the camera coordinate system. z mx z my z mz : The components of the unit direction vector of the device coordinate system along the z-axis in the camera coordinate system along the x, y, and z-axis directions. mx c my c mz: The updated coordinates of the origin of the device coordinate system in the x, y, and z axes of the camera coordinate system. The fourth row of the matrix

[0001] is the standard form of homogeneous coordinates, ensuring the homogeneity of the coordinate transformation.

[0185] Considering the influence of unstable factors such as the nature of potential targets, lighting effects, and shooting conditions in real measurement environments, based on the experimental equipment designed in this chapter, this application divides the dataset used for network training into two main parts: a real dataset and a virtual dataset, and introduces the construction process of the two datasets respectively.

[0186] like Figure 12 As shown, for a real dataset, this application designs more than ten types of test pieces with known morphological information. These test pieces can be fixed at specific positions on the equipment's turntable, and measurement input images of the test pieces can be captured by a camera. Based on the known morphology of the test pieces and the transformation relationship between the equipment coordinate system and the camera coordinate system, this application first processes the captured test piece images using image preprocessing and feature extraction algorithms to obtain the feature positions of the test pieces. Then, based on the feature positions and coordinate system relationship information of the test pieces, the pose of the test pieces in the camera coordinate system is calculated. Finally, based on the morphological features and pose information of the test pieces, the normal vector and depth information of the test pieces in the camera coordinate system are calculated, ultimately completing the construction of the real dataset. The virtual dataset mainly includes existing publicly available datasets and synthetic datasets. The synthetic dataset is based on the real dataset of this application and is obtained by simulating the physical environment through simulation software. In the process of constructing the virtual dataset, this application keeps the physical parameters such as illumination, shooting angle, and distance consistent with the parameters of the real device, while focusing on modifying the optical properties of the object being tested, such as color, metallicity, and reflectivity, thereby improving the accuracy of the neural network in predicting the normal vector and depth of objects with different physical properties.

[0187] In industrial settings, environmental disturbances, background interference, and instability of transmission and storage devices can all introduce noise into captured images. This noise significantly impacts the accuracy of computer-generated image feature point extraction, interfering with image analysis results. Based on the construction process of the real dataset and actual captured images used in this application, the application first performs distortion correction on the calibrated camera images according to an image distortion correction process. Then, it primarily employs median filtering and low-pass filtering algorithms for image preprocessing.

[0188] After obtaining the distortion-corrected image, this application first uses Equation 1.40 to convert the three-channel color image captured by the camera into a grayscale image.

[0189] I=0.299R+0.587G+0.114B (1.40)

[0190] In the formula, RGB represents the pixel values ​​of the three channels of a color image.

[0191] After acquiring the grayscale image, median filtering is used to process any noise that may be present in the image. Median filtering is a non-linear smoothing filtering algorithm. When the pixel value of a certain pixel differs significantly from the pixel values ​​of a certain number of pixels within the template range, that pixel is considered a noise point, and the median value of the pixels within the template range is taken as the new pixel value, thus achieving the filtering purpose. The main steps of median filtering are: moving the template in the image so that a certain pixel in the image coincides with the center of the template; if the pixel value of this pixel meets the conditions for median filtering, then reading the pixel values ​​of the surrounding pixels corresponding to the template positions, finding the median value among these pixel values, and assigning this median value to the pixel at the center of the template. In the images captured by the device in this application, due to the metal material of the device's turntable and the sandblasting process, a small number of high-reflectivity, high-brightness noise points may appear under the illumination conditions of the device's light source. For the above noise, median filtering can achieve a good filtering effect. Based on this, this application further uses low-pass filtering for image processing. Low-pass filtering is a frequency domain filtering method. By treating an image as a two-dimensional signal and performing a Fourier transform, its frequency domain characteristics can be obtained. Typically, edges, noise, and other information in the image are transformed into the high-frequency components of the spectrum, while larger, less variable image features are transformed into the low-frequency components. Based on the aforementioned principle of image Fourier transform, this application employs low-pass filtering to further filter the median filtering result of the captured image. By controlling the low-pass filtering parameters, the texture information of the object in the image is removed while retaining the boundary information of the test object. This eliminates redundant information interference such as background noise points and the texture of the test object itself for subsequent feature extraction steps. This application captures images of the test object under the device's illumination conditions. The resulting image after preprocessing operations such as image distortion correction and image filtering is shown below. Figure 13 As shown in (a). Based on this, this application uses the Canny operator to perform boundary detection on the tested part to obtain the target boundary image. Then, a connected component detection algorithm is used to filter the boundary length to obtain a target boundary of moderate length that belongs to the boundary features of the tested object, such as... Figure 13 As shown in (b). Finally, the Hough algorithm is used to further filter the target boundary, obtaining the center c of the circular boundary of the measured part in the camera image. ILocation information. Since the true normal vector information of the measured object needs to be constructed in the camera coordinate system, this application divides the calculation process of the true normal vector of the measured object into the following steps. First, the transformation relationship between the workpiece coordinate system and the camera coordinate system needs to be calculated, transforming the point and surface coordinate information of the measured object located in the workpiece coordinate system into the camera coordinate system. Then, the normal vector and depth information of each surface of the measured object are calculated in the camera coordinate system. Finally, through physical constraints in the real environment, the correspondence between the surface normal vectors and depths in the camera coordinate system and each pixel in the image coordinate system is constructed, and the three-dimensional normal vectors and depth information are transferred into a two-dimensional image corresponding to the image captured of the measured object for storage. This application obtains the center c of the circular base of the measured object. I That is, the coordinates of the origin of the workpiece coordinate system in the image coordinate system. Based on these coordinates and combined with the camera calibration parameters, the point can be reprojected into the three-dimensional camera coordinate system, and the reprojection vector of the origin of the workpiece coordinate system is obtained as shown in Equation 1.41.

[0192] p cI =c c -c I (1.41)

[0193] In the formula c c p represents the camera optical center coordinates in the image coordinate system. cI This is the reprojection vector of the workpiece coordinate system origin. According to the camera pinhole imaging model, this reprojection vector can be directly applied to the camera coordinate system, pointing to the position of the workpiece coordinate system origin under the camera coordinate system.

[0194] By combining the equations of the rotary table plane in the camera coordinate system obtained from the solution, we can find the intersection point P of the reprojection vector and the rotary table plane. I That is, the origin coordinates of the workpiece coordinate system under the camera coordinate system. Considering the physical constraints of the workpiece being placed on the equipment rotary table and the structural constraints of the measured object being a body of revolution, it can be determined that the z-axis direction of the workpiece coordinate system is consistent with the direction of the normal vector of the equipment rotary table plane. At the same time, it can be assumed that the x-axis direction of the workpiece coordinate system is consistent with the x-axis direction of the camera coordinate system. At this time, the method of determining the transformation relationship matrix from the equipment rotary table coordinate system to the camera coordinate system in this application can obtain the coordinate transformation matrix from the workpiece coordinate system to the camera coordinate system, as shown in Equation 1.42.

[0195]

[0196] In the formula x I y I z I These are the unit direction vectors of the three axes of the workpiece coordinate system in the camera coordinate system, P. I =[p Ix p Iyp Iz ] T Let T be the coordinates of the origin of the workpiece coordinate system in the camera coordinate system. Regarding formula (1.42)T... Ic T Ic : The coordinate transformation matrix from the workpiece coordinate system to the camera coordinate system. This matrix enables coordinate transformation between the two coordinate systems. Regarding x... Ix x Iy x Iz These are the unit direction vectors x and x' of the workpiece coordinate system, respectively. I The components along the x, y, and z axes of the camera coordinate system describe the direction of the x-axis of the workpiece coordinate system in the camera coordinate system. Regarding y... Ix y Iy y Iz These are the unit direction vectors of the y-axis in the workpiece coordinate system. I The components along the x, y, and z axes of the camera coordinate system describe the direction of the y-axis of the workpiece coordinate system in the camera coordinate system. Regarding the z-axis... Ix z Iy z Iz These are the unit direction vectors z and z along the z-axis of the workpiece coordinate system, respectively. I The components along the x, y, and z axes of the camera coordinate system describe the direction of the z-axis of the workpiece coordinate system in the camera coordinate system. Since the z-axis direction of the workpiece coordinate system coincides with the direction of the normal vector of the rotary table plane, these components are related to the projection of the rotary table plane normal vector into the camera coordinate system. Regarding p... Ix p Iy p Iz These are the origin P of the workpiece coordinate system. I The coordinate values ​​along the x, y, and z axes of the camera coordinate system determine the position of the origin of the workpiece coordinate system within the camera coordinate system.

[0197] Based on the existing surface model of the real object being measured, the object can be completely digitized, yielding information such as the surface and vertex coordinates of the model. Assuming the vertex coordinates in the model are P, the vertex coordinates P′ transformed into the camera coordinate system are shown in Equation 1.43.

[0198]

[0199] After obtaining the coordinates of each vertex of a facet in the camera coordinate system, this application performs a cross product of the boundary vectors formed by these vertices to obtain the normal vector information of that facet in the camera coordinate system. After obtaining the normal vector information of the object under test in the camera coordinate system, it is necessary to calculate the correspondence between the normal vector information and the pixels of the captured image. The image pixels of the object under test after distortion correction are connected to the optical center of the camera to form the pixel reprojection direction vector. Then, it is necessary to find the facets that intersect with this direction vector. The specific implementation process is as follows: First, the equation of the pixel reprojection direction vector in the camera coordinate system is constructed, as shown in Equation 1.44.

[0200] P = O + tD (1.44)

[0201] In the formula, O is the optical center of the camera in the camera coordinate system, D is the reprojection direction vector from the camera pixel to the optical center of the camera, and P is a point on the direction vector.

[0202] Then, the surface information of the object under test in the camera coordinate system is traversed, and the plane equation in the camera coordinate system is constructed based on the three vertices V1, V2, and V3 of the triangular surface, as shown in Equation 1.45. P=(1-uv)V1+uV2+vV3 (1.45), In the formula, u and v are equation parameters. When the equation parameters satisfy u≥0, v≥0, and u+v≤1, point P on the plane is located inside the triangular facet.

[0203] Combining the above equation, the equation for solving the intersection point P0 between the projection direction vector and the triangular facet is shown in Equation 1.46.

[0204] (1-uv)V1+uV2+vV3=O+tD (1.46)

[0205] By rearranging Equation 1.46, we can obtain the following system of linear equations.

[0206]

[0207] Where E1 = V2 - V1, E2 = V3 - V1, T = O - V1.

[0208] Solving the linear equations using Cramer's rule yields the parameter values ​​shown in the following equations.

[0209]

[0210] Based on this, according to the vector mixed product formula, the calculation results of each parameter value are shown in the following formula.

[0211]

[0212] After traversing all faces of the model of the object under test and solving for the parameters corresponding to the intersection point P0, the parameters are filtered to determine the face that makes point P0 satisfy the conditions u≥0, v≥0, u+v≤1. This allows the determination of the face position of the object under test corresponding to the pixel in the camera-captured image. Furthermore, since the reprojection direction vector of a specific pixel may intersect with the object under test multiple times in the camera coordinate system, based on the actual shooting physical environment, this application selects the normal vector of the object under test face that satisfies the conditions and is closest to the camera's optical center as the normal vector corresponding to that pixel.

[0213] In the normal vector map construction stage, this application uses the RGB information contained in the three-channel image to represent the XYZ direction information of the normal vector in the corresponding space. First, the normal vector is normalized to obtain the normal vector information [xy z] corresponding to the pixel. T As can be seen from the properties of unit vectors, the normal vector numerically satisfies...

[0214]

[0215] Therefore, a normalization formula can be used to normalize the value of the target's normal vector, and the result is shown in Equation 1.50.

[0216]

[0217] Based on this, the normalized The numerical values ​​are assigned to the three channels of the color image to obtain a color normal vector image containing the normal vector information of the measured object. Furthermore, in the depth map construction stage, since this application has already obtained various parameters related to P0 during the process of solving for intersection point P0, where parameter t represents the distance of intersection point P0 from the camera's optical center, and since the target depth information in this application is all less than 1m, this application uses meters as the unit, multiplies it by 65535 and rounds it down, uses the calculation result as the pixel value, and assigns it to a single-channel image in a 16-bit storage format to construct a grayscale depth image containing the depth information of the measured object.

[0218] like Figure 14 This application presents a sequence of images of the frustum, the actual object being tested, captured under the equipment and lighting conditions designed in this application. Based on this sequence, the application employs image preprocessing and feature extraction to obtain the boundary information of the object being tested, and constructs a mask image of the object based on this information, as shown below. Figure 14 As shown in (d). After obtaining the position information of the object under test in the camera image, this application calculates the coordinates of the object under test in the camera coordinate system according to the dataset construction process of this application, and forms the surface normal vector map and depth map of the object under test, as shown in (d). Figure 14 (e) Figure 14As shown in (f), the construction of the real dataset of the tested object was completed. Figure 14 In (g), this application uses a heat map to further clarify the depth information of the object under test.

[0219] This application designs ten real-world test objects with known shapes. In each shooting process, a single test object is placed at five different distances from the center of the device's turntable. After each position is determined, the turntable is rotated 18° and then brought to a stop. The direction of the light source is changed to acquire a sequence of images of the test object. After one full rotation of the turntable, twenty datasets are obtained. Thus, a total of 1000 real-world datasets of specific test objects are obtained. These real-world datasets can be used to train the network to adapt to phenomena such as noise and light attenuation that may exist in real-world environments, improving the algorithm's robustness.

[0220] The virtual dataset mainly consists of two parts: a publicly available dataset and a virtual dataset based on real test objects. This application primarily uses the blob dataset, which contains, for example, the following publicly available datasets: Figure 15 The example shown comprises ten complex surface patch models of varying shapes. To create a virtual dataset, this application imports the blob dataset into the Blender simulation software. The virtual camera shooting distance and virtual light source illumination position are identical to the actual equipment design, and the model coordinate system is consistent with the scene coordinate system. For each model, before each image capture, it is rotated 36° around the x-axis, y-axis, or z-axis of the workpiece coordinate system, and its dimensions are randomly transformed in three directions. This allows for the generation of 1000 different virtual datasets for each model. Furthermore, before each image capture, this application randomly defines the surface optical properties of the model, such as color, metallicity, and roughness, thereby expanding the BRDF parameter samples of the measured object. Some model images are shown below. Figure 16 As shown. Therefore, this application obtained a total of 10,000 virtual datasets generated by optical simulation. For example... Figure 17 As shown, this application selects a portion of the dataset and displays virtual imaging images of the targets in the dataset, along with their corresponding normal vector maps and depth maps.

[0221] To construct a virtual dataset based on a real test object, this application first requires calculating the pose of the real test object according to visual recognition and localization, and the transformation relationships between various coordinate systems. In the virtual simulation environment, the illumination and shooting conditions of the light source and camera are all based on the device coordinate system, and the simulation environment coordinate system is consistent with the device coordinate system in the actual environment. Therefore, it is necessary to calculate the pose of the real test object in the device coordinate system to construct the virtual dataset. This application has already obtained the transformation matrix T between the device coordinate system and the camera coordinate system. mcMeanwhile, the coordinate transformation matrix T from the workpiece coordinate system to the camera coordinate system has been obtained in this application. Ic According to the coordinate transformation relationship, the transformation matrix from the workpiece coordinate system to the equipment coordinate system is shown in Equation 1.51.

[0222] T Im =T mc -1 ·T Ic (1.51)

[0223] Based on the above transformation matrix, the expressions of each axis and origin of the workpiece coordinate system in the device coordinate system can be obtained, the pose of the real measured object in the device coordinate system can be obtained, and the construction of a virtual dataset based on the real measured object can be realized.

[0224] This application also expands the BRDF sample parameters of the virtual dataset based on the real test object by changing the optical properties such as color, metallicity, and roughness of the virtual test object, thereby ensuring that the deep learning network can achieve high-precision measurement of objects with different optical properties. Some images of the virtual dataset based on the real test object are shown below. Figure 18 As shown in the figure, in order to further clarify the target depth information, this application uses a heatmap to assign different colors to the target depth information. Figure 18 (d) shows the distance between different positions of the measured object and the optical center of the camera. The distribution of depth information of the measured object is displayed intuitively through the correspondence between color and numerical value.

[0225] To further verify the effectiveness of the photometric stereo vision reconstruction algorithm proposed in this application, this application first constructs and trains the algorithm according to the photometric stereo vision algorithm proposed in Chapter 2, and then verifies the network prediction effect using virtual datasets and real test objects respectively. In addition, this chapter also designs an ablation experiment to demonstrate the effectiveness of the physical model-driven part in the neural network module designed in this application.

[0226] To train the target surface normal vector and depth prediction network for photometric stereo vision technology proposed in this application, a machine equipped with a Core i9-13900KF CPU (5.8GHz), 64GB RAM, and an NVIDIA GeForce RTX 4090 graphics card was used to train the neural network. Considering the hardware limitations, the dataset image size designed in this application is 256×256. For real datasets larger than this size, this application first symmetrically crops the images to squares, and then uses bilinear interpolation to downsample the image size to 256×256 before inputting it into the network for training. Before network training, the virtual dataset images only contain lighting information at the pixel positions corresponding to the tested object, while other pixel positions, although containing pixel values, are all interference information that does not contain the lighting information of the tested object. According to the algorithm scheme in Chapter 2, this application uses tensors to replace position and orientation to describe the lighting information in the image sequence. The network analyzes the lighting information of the entire image domain and analyzes the above interference information as the lighting information of the tested object. Therefore, the pixel values ​​of all pixels in the image will affect the lighting information. Therefore, before using the virtual dataset images for network training, this application creates an image mask using the target normal vector image from the dataset to obtain the image region where the test object is located. This mask is then applied to all images in the dataset to remove interference information that does not contain the lighting and shadow information of the test object. The filtered images are then input into the neural network for training. During network training, this application uses a random sampling method to shuffle the input order of the training dataset to prevent interference from similar shapes in adjacent datasets. Furthermore, this application uses random noise addition to the input dataset each time it reads the dataset for training to perform data augmentation, thereby improving the network's learning ability. The dataset designed in this application contains approximately 13,000 sets, with the ratio of virtual datasets to real datasets approximately 6:1, the training set accounting for 99%, and the test set accounting for 1%. The network training environment of this application is a Python environment, using the PyTorch deep learning open-source library, and the network uses Adam optimization technology to update network parameters. During network training, this application addresses the two major challenges of determining illumination information and calculating target depth by maintaining a constant ambient light intensity in the dataset. However, since the light intensity is not entirely consistent when constructing the virtual and real datasets respectively, the network parameters should have slight differences when predicting the virtual and real datasets. Therefore, this application divides network training into two stages: pre-training and fine-tuning. The pre-training stage consists of 30 iterations, and the fine-tuning stage consists of 10 iterations. In the fine-tuning stage, two sets of parameters are trained using the virtual and real datasets respectively, and then applied to the prediction of the virtual and real test objects respectively.Both training phases use 8 datasets as a training batch. The initial learning rate of the network is set to 10. -4 During the pre-training phase, the learning rate is lowered at the 2nd, 5th, 10th, 15th, 20th, and 25th training iterations, respectively. In the fine-tuning phase, the learning rate is set to half of the learning rate at the end of the pre-training phase. The network prediction performance is tested in two parts: normal vector prediction verification and depth prediction verification. After network training, the network's normal vector prediction performance on a virtual dataset is first compared horizontally with that of a classic photometric stereo vision algorithm using the trained parameters, and the network's depth prediction performance is then tested. Subsequently, real-world images are used to test the network's normal vector prediction and depth prediction capabilities in a real measurement environment. The network prediction performance of this application further reflects the rationality of the photometric stereo vision device design.

[0227] To verify the effectiveness of the proposed algorithm in normal vector prediction, this application first uses the prediction results of a partial virtual dataset to compare the proposed network with the FS-FCN work presented at the 2018 CVPR top machine vision conference. The normal vector test results are as follows: Figure 19 As shown in the figure, by comparing the normal vector prediction results obtained by the method of this application with the normal vector prediction results of PS-FCN, it can be seen that the algorithm of this application provides more refined surface normal vector prediction results for objects with complex shapes and more accurate prediction of the overall shape.

[0228] This application further calculated the mean error of the algorithm's prediction results. In the test results on 90 sets of virtual datasets, the mean error of the network's prediction of the surface normal vector of the measured object ranged from 2.528° to 5.128°, with an overall mean error of 3.643°. This demonstrates relatively accurate prediction of the surface normal vector of the measured object. The mean error of the FS-FCN network's surface normal vector prediction ranged from 10.550° to 20.762°, with an overall mean error of 15.173°. This application shows a significant improvement in the accuracy of normal vector prediction on virtual datasets compared to previous work. To further test the network performance, this application used actual products with relatively complex surface morphology, and conducted photography and normal vector prediction in a device-defined environment. The prediction results are as follows: Figure 20 As shown. When faced with a complex object to be measured, traditional algorithms struggle to locate the target by extracting image features, making it difficult to calculate the true surface normal vector of the object using only a single camera. The photometric stereo vision algorithm proposed in this application can predict the surface normal vector information of an object at an unknown location with high accuracy, thus obtaining the surface morphology information of the object.

[0229] Based on the above results, it can be demonstrated that the photometric stereo vision algorithm model proposed in this application can meet the surface morphology measurement requirements of complex objects and can realize photometric stereo vision measurement of targets.

[0230] This application first uses a virtual dataset to verify the depth prediction performance of deep learning networks. For example... Figure 21 The figure shows the prediction effect of the depth prediction module of this application on a virtual dataset. The first column on the far left of the figure is the image of the virtual object being measured, the second column is the true depth map of the object being measured, the third column is the depth prediction result of the network, and the fourth column is the depth prediction error map. For ease of observation, this application plots the depth map and error map as a heatmap, with the data range represented by the color shown on the right side of the figure. The units for depth and error in the figure are both mm.

[0231] Table 5.4 presents detailed test results for the seven sets of test datasets. This application uses the minimum distance between the object being tested and the camera's optical center to represent the shooting distance of the object, and provides the root mean square error and mean error between the predicted depth and the true depth. The data in the table shows that the depth prediction module of this application's deep learning network has good robustness to the shooting distance of the virtual object being tested. When the shooting distance of the object changes, the depth prediction accuracy remains within 10mm. Therefore, with the help of this application's deep learning network, the shooting distance of the object being tested can be estimated using only a single camera, and the prediction effect is good.

[0232] Table 5.4 Experimental Results of the Virtual Dataset

[0233]

[0234] In addition, such as Figure 22 As shown, this application uses real-world images to verify the network's depth prediction capability in a real-world environment. The imaging of the circular stepped structure component used in this application is as follows: Figure 22 As shown in (a), the true depth maps obtained through target localization calculation and network prediction are respectively as follows: Figure 22 As shown in (b) and 22(c), 22(d) is the prediction error map. The units of depth and error in the map are mm.

[0235] As shown in Table 5.5, this application also provides detailed depth prediction results of the network for the four types of objects in the figure.

[0236] Table 5.5 Experimental Results of Real Test Subjects

[0237]

[0238] Experimental results for both virtual and real test objects show that the algorithm in this application can generally predict the depth information between the test object and the camera. In the experiments, due to the better consistency of surface material and texture in the virtual dataset, the depth prediction accuracy is higher, with errors mainly occurring in areas where the depth of the test object changes abruptly. The real test object, due to its more complex texture information, has a slightly larger depth prediction error compared to the virtual test object, with an overall root mean square error of approximately 13.02 mm. However, it can still make relatively accurate predictions of the overall depth information of the test object, thus demonstrating that the method in this application has a certain target position prediction capability. To verify the effectiveness of the fused physical imaging model in the photometric stereo vision reconstruction algorithm of this application, an ablation experiment was designed for the specular reflection information analysis module in the proposed deep learning network architecture. For example... Figure 23 As shown, the ablation model in this application directly concatenates the above information along the channel dimension and then performs convolution processing. The number of network parameters in this ablation model is basically the same as that in the original network structure, which can effectively verify the driving effect of the physical formula in the original network. In addition, this application also designs an ablation model that removes the specular reflection module. This network structure completely removes the specular reflection information analysis module for predicting the normal vector and depth of the object under test.

[0239] This application used a completely consistent dataset and hyperparameters to train and validate the original network architecture and the two ablation models mentioned above. The mean error of normal vector prediction and the root mean square error of depth prediction of the virtual dataset are shown in Table 5.6.

[0240] Table 5.6 Results of the ablation experiment on the specular reflection module

[0241]

[0242] The test results show that the physical model-driven specular reflection information analysis module proposed in this application can effectively improve the network's prediction of the normal vector and depth of the measured object.

[0243] The above description is merely an embodiment of this application and is not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principle of this application should be included within the scope of the claims of this application.

Claims

1. A photometric stereo vision measurement method, characterized in that, include: Based on the illumination information extraction module, the image sequence of the input test object is processed to obtain the illumination feature tensor; Based on the diffuse reflection information analysis module, diffuse reflection features are extracted from the illumination feature tensor to generate a normal vector information map of the surface of the object under test. Based on the specular reflection information analysis module, specular reflection features are extracted from the illumination feature tensor to determine the specular reflection information of the surface of the object under test. The following steps are used to determine the 3D normal vector and / or depth prediction results of the surface of the object being measured: Based on the normal vector prediction module, the normal vector information map and specular reflection information are fused and convolutionally oriented to generate the three-dimensional normal vector of the surface of the object being measured. Based on the depth prediction module, iterative tensor optimization processing is performed on the normal vector information map and specular reflection information to generate the depth prediction result of the measured object.

2. The method according to claim 1, characterized in that, Based on the specular reflection information analysis module, specular reflection features are extracted from the illumination feature tensor to determine the specular reflection information of the surface of the object under test, including: Pixel-level illumination direction information is extracted from the illumination feature tensor; Based on the extracted pixel-level illumination direction information, a specular reflection map of the image is generated; By fusing the image specular reflection map and the image sequence, a multi-channel feature map is obtained that combines the specular reflection intensity on the image specular reflection map with the texture in the image sequence. Convolution processing is performed on the multi-channel feature maps to determine the specular reflection information of the surface of the object being measured.

3. The method according to claim 7, characterized in that, The specular reflection information analysis module includes a light direction extraction sub-network to perform multi-level channel compression on the light feature tensor to obtain pixel-level light direction information.

4. The method according to claim 7, characterized in that, The specular reflection information analysis module includes a physical drive calculation module to calculate the specular reflection amount based on the normal vector information map, the camera viewing direction, and pixel-level illumination direction information, so as to generate an image specular reflection map that reflects the intensity of specular reflection.

5. The method according to claim 7, characterized in that, The specular reflection information analysis module includes a channel stitching module to stitch the image specular reflection map and the image sequence along the channel dimension to form a multi-channel feature map containing specular reflection intensity and image texture.

6. The method according to claim 7, characterized in that, The specular reflection information analysis module includes a specular reflection fitting convolutional network to extract features layer by layer from the multi-channel feature map and fit the relationship between the specular reflection component and the imaging, so as to output the specular reflection information of the surface of the object being measured.

7. The method according to claim 1, characterized in that, Based on the depth prediction module, iterative tensor optimization processing is performed on the normal vector information map and specular reflection information to generate a depth prediction result map of the measured object, including: The normal vector information map and the specular reflection information are spliced ​​and fused to obtain a fused feature representation map; The fused feature data is convolved to obtain a fused feature map, which is then fused in parallel with the primary feature map along the channel direction to obtain the first parallel feature map. The first parallel feature map is convolved to obtain an enhanced feature map, which is then convolved in parallel with the full-size feature map along the channel direction to obtain a second parallel feature map. The second parallel feature map is processed by a deconvolution layer to scale the resolution to the same size as the image sequence, and then concatenated in parallel with the predicted surface normal vector information along the channel direction to obtain the third parallel feature map. Convolution is performed on the third parallel feature map to obtain the depth feature representation map; The depth feature representation map is convolved to obtain the depth prediction result map of the object being measured.

8. The method according to claim 7, characterized in that, The depth prediction module includes an information fusion unit, configured to stitch together the normal vector information map and the specular reflection information along the channel dimension to generate a fused feature representation map.

9. The method according to claim 7, characterized in that, The depth prediction module includes a third-level convolutional unit containing a convolutional layer, configured to perform convolution calculations on the third parallel feature map to generate a depth feature representation map.

10. The method according to claim 7, characterized in that, The depth prediction module includes a depth output unit containing a convolutional layer configured to perform convolutional processing on the depth feature representation map to generate a depth prediction result map of the object being measured.

Images