Dataset creation method and verification system based on photometric stereo surface reconstruction

Abstract

A method for producing a defect detection dataset based on photometric stereo surface reconstruction includes the following steps: (1) selecting four types of objects to generate a dataset; (2) acquiring multiple images under different lighting conditions; sequentially stimulating light sources at different positions for each object to illuminate the object, and acquiring a set of photometric images of the object's surface area; (3) acquiring multiple images under different lighting conditions; sequentially stimulating light sources at different positions for each object to illuminate the object, and acquiring a set of photometric images of the object's surface area; (4) constructing a defect detection dataset based on photometric stereo surface reconstruction based on the normal vector map obtained in step (3). The present invention also provides a verification system for a method for producing a defect detection dataset based on photometric stereo surface reconstruction. The invention can avoid information loss due to surface reflection by acquiring multiple images under different light source conditions, and photometric stereo vision can acquire a surface normal vector map; and a verification system is designed to verify the overall solution, thereby achieving effective identification of metal surface defects.

Description

Technical Field

[0001] The invention relates to a data set preparation method and a verification system based on photometric stereo surface reconstruction. Background Art

[0002] Currently, most companies use manual visual inspection to test product surface quality. This method has several problems: the factory production environment is relatively harsh, especially in stamping workshops, where there is high vibration and noise, making it unsuitable for workers to work for long periods of time; workers' visual inspection is subject to subjective factors; and prolonged inspection can cause eye fatigue and harm physical health. Automated inspection has become a trend in corporate development. Automated inspection can standardize testing standards, save labor costs, reduce worker workload, and improve inspection efficiency.

[0003] The development of computer vision has greatly alleviated this problem. Image-based machine vision inspection technology enables non-contact, non-destructive surface testing. It also combines non-destructive testing with automation and intelligence, achieving highly efficient and accurate non-contact defect detection. However, using machine vision algorithms for defect detection requires the ability to extract morphological information about defects from images. However, the highly reflective nature of metal surfaces can cause overexposure in some areas of the captured image, making it difficult to capture all defect information from a single image. Adjusting the direction and angle of light is necessary to ensure that defects are not located in glossy or shadowed areas. However, the large size of product parts and the high inspection accuracy required make it difficult to capture image information of the entire surface in a single image. Product size varies, and the curved surfaces of cavities make it difficult to obtain high-quality surface information directly above the surface. Furthermore, two-dimensional image detection is difficult to handle when depth information is required for defect detection.

[0004] At the same time, photometric stereo has garnered significant attention in the advanced manufacturing sector in recent years. This technology utilizes images captured under different lighting conditions at a fixed viewing angle in a known light source environment to determine the surface normal information based on the object's surface reflection patterns, thereby reflecting the three-dimensional characteristics of the object's surface. Since most actual inspection objects are non-Lambertian materials, the complex reflection patterns of non-Lambertian surfaces make accurate modeling difficult using ideal assumptions. Therefore, non-Lambertian photometric stereo has become a major research area in photometric stereo algorithms.

[0005] Both defect detection methods and photometric stereo algorithms have achieved excellent performance on their respective benchmarks. However, there are few defect detection methods based on photometric stereo. This is because most current defect detection data is captured under a single illumination, which makes it difficult to solve normals and thus unsuitable for photometric stereo. Furthermore, existing photometric stereo datasets lack defect samples. Summary of the Invention

[0006] To overcome the above problems, the present invention provides a method for preparing a defect detection dataset based on photometric stereo surface reconstruction, comprising the following steps:

[0007] (1) Construct a defect sample set, which consists of multiple samples. The sample attributes include color, material, shape, and defect type. Each sample has at least one attribute that is different from other samples.

[0008] (2) stimulating light sources at different positions of each object in turn to illuminate the object, obtaining an image by stimulating the light source at each position, and forming a set of photometric images with at least three images;

[0009] (3) Inputting the photometric image into the photometric stereo network model, using the trained photometric stereo network model to perform model inference based on the photometric image to estimate the surface normal vector at each pixel point, and obtain the object surface normal vector map; training the photometric stereo network model includes the following steps:

[0010] 3.1) Establish an imaging environment model for the acquisition system and calibrate the light source information of the photometric stereo imaging device;

[0011] 3.2) Build a 3D convolution-based photometric stereo network model and train it using a synthetic photometric stereo dataset until the model converges on the training dataset. The synthetic photometric stereo datasets are the Bloddy and Sculpture datasets, and the real-world object dataset used for verification is the DiLiGenT dataset.

[0012] 3.3) Design evaluation metrics for the photometric stereo network model. The performance of the photometric stereo model is evaluated by the surface normal estimation accuracy. The surface normal estimation accuracy is calculated as the average angular error between the predicted normal vector and the true normal vector. The calculation formula is:

[0013]

[0014] Among them, N k Represents the normal vector at the kth pixel in the predicted normal vector map, Represents the normal vector at the kth pixel in the true normal vector map, and K represents the total number of pixels in the normal vector map. The smaller the MAE value, the higher the estimation accuracy of the normal vector.

[0015] (4) Based on the normal vector map obtained in step (3), a defect detection dataset based on photometric stereo surface reconstruction is constructed;

[0016] 4.1) Use the Python annotation script LabelImg to mark the defect target area with a rectangular box;

[0017] 4.2) Read the annotation file for each image, obtain the location and category information of each defect in the current image, set a sliding window, and traverse the image with the sliding window starting from the upper left corner of the current image;

[0018] 4.3) Determine whether there is a defective target within the current sliding window. If there is no defective target, discard the image cropped by the window; if there is a defect, save the information of the defective area that intersects with the sliding window;

[0019] 4.4) Determine whether the window of the regional information saved in step (4.3) contains an incomplete defect. If not, convert the position information of the defect to the image coordinate system of the window, save it and generate a new annotation file; when an incomplete defect area exists, calculate the ratio of the area of ​​the defect in the window to the area of ​​the actual annotated defect area, and judge by a threshold. When the ratio is greater than 0.75, adjust the annotation information of the defect and use the window boundary as the new boundary. When the ratio is less than 0.75, discard the defect, save the window image and generate a new annotation file; after completion, obtain a defect sample image;

[0020] 4.5) Divide the defect sample images obtained in step 4.6) into a training set, a validation set, and a test set.

[0021] Furthermore, the defect sample set includes at least four types of defect samples; the four types include yellow hemisphere, silver hemisphere, yellow cone and silver cone of stainless steel, and each sample has defects on its surface by simulating the impact of the production process, and the types of defects include at least point pits, deformation, scratches and abrasions.

[0022] Furthermore, the light source calibration includes the following steps: light source direction calibration; using a chrome-plated metal ball as a calibration ball, first obtaining the center of the calibration ball, solving the position of the highlight point on the surface of the steel ball through image processing, and then obtaining the surface normal vector at this point;

[0023] Get the parameter center coordinates (x C ,y C ) and radius r, process the highlight point image, and obtain the coordinates of the highlight point P (x P ,y P ), calculate the spherical normal vector at that point; according to the calibration sphere three-view image, the relative coordinates of the highlight point compared to the center of the circle are the x and y components of the normal vector, and the z component is calculated according to the Pythagorean theorem. N is expressed as:

[0024]

[0025] Given the reflection direction vector and the normal direction vector, the incident direction vector can be obtained by the following method:

[0026] First, translate the reflection direction vector and intersect it with the extended normal direction vector S. According to the law of reflection of light, vectors L, R, and S form an isosceles triangle; the incident direction vector L can be expressed as:

[0027] L=RS (2)

[0028] Vector S has the same direction as the normal vector and can be represented by vectors R and N:

[0029]

[0030] Combining the above formula, we can get:

[0031] L=R-2(R·N)N (4)

[0032] 3.1.2) Light source intensity calibration: Use a standard whiteboard and treat it as an ideal Lambertian surface. The shape of the standard whiteboard is an ideal plane, and the normal vectors of each point are consistent. The plane normal vector n of the whiteboard is std =(0,0,1) T ; The photometric stereo formula for calibrating a whiteboard is as follows:

[0033]

[0034] Among them, the pixel response value of the whiteboard surface Light source direction l, whiteboard plane normal vector All are known parameters, kc is the camera internal parameter, Elight is the light intensity, T is the exposure time, ρ white is the reflectivity of the white board; rearranging the above formula yields:

[0035]

[0036] The left side of the above formula is a constant. By calculating the ratio of the constants between the light sources and then normalizing them, the relative intensity factors of the light sources are obtained.

[0037] Furthermore, building a photometric stereo network model based on 3D convolution specifically includes the following steps:

[0038] 3.2.1) Design the network model input for the photometric stereo network model. The input data is the photometric matrix and the calibration matrix of the light source direction and light source intensity. The size of each photometric image frame is 3×W×H, and the corresponding light source direction and light source intensity calibration parameters are 3×1 vectors, where the image width is W and the image height is H. The number of image pixels p representing the object can be expressed as p=W×H. The light source direction parameters and the one-dimensional light source direction vector are replicated according to the spatial dimension to generate a 3×W×H light source direction map. This is concatenated with the corresponding photometric image to generate a 6×W×H matrix. Then, the q photometric images are concatenated in the spatial dimension to generate a (6×q)×W×H three-dimensional matrix as the model input.

[0039] 3.2.2) Design the initial feature extraction module of the photometric stereo network model. This module performs a convolution operation on the input (6×q)×W×H data matrix to obtain the feature map of each frame, preparing for the subsequent extraction of spatiotemporal features using 3D convolution. The initial feature module consists of a 3D convolution layer, an activation function layer, and a dropout layer. First, a 6×1×1 3D convolution kernel is used to convolve the original matrix. The spatial dimensions of the convolution kernel are the same as the spatial dimensions of the single-frame photometric and light source direction data matrix to extract the features of the single-frame data. The activation function uses the LeakyReLU function, and the dropout layer uses random dropout to prevent overfitting of the model.

[0040] 3.2.3) Design a 3D feature extraction module for the photometric stereo network model to obtain a feature map of the photometric data for each frame. Within this module, deploy inter-frame and spatial feature extraction blocks to simultaneously analyze the inter-frame and spatial photometric information variation patterns. The inter-frame feature extraction block consists of a 3D convolutional layer, a LeakyReLU activation layer, and a Dropout layer. The convolution kernel size of the 3D convolutional layer is M×1×1, and its function is to process the photometric variations of single pixels across M adjacent frames. The spatial feature extraction block consists of a 3D convolutional layer and a LeakyReLU activation layer. The convolution kernel size is 1×N×N, and the 3D convolutional layer is used to obtain the intensity variations between local pixels. This local information can improve performance when the surface reflectivity of an object changes, while also being robust under sparse illumination.

[0041] 3.2.4) Design a normal vector estimation module for the photometric stereo network model. Add a max-pooling layer before the normal vector estimation module. Convert the output of the feature extraction module with different dimensions into a fixed-dimensional feature map. The normal vector estimation module uses the feature map after the max-pooling operation to estimate the normal map. The normal vector estimation module consists of two fully connected layers.

[0042] 3.2.5) Design a loss function for the photometric stereo network model. The photometric stereo network model works by reducing the error between the predicted normal vector and the true normal vector at each point. The normal vector is essentially a unit space vector, and the difference between two unit space vectors can be reflected by the angle between the two vectors. The cosine similarity loss function is used, and the formula is as follows:

[0043]

[0044] Where H is the image length, W is the image width, i and j represent the number of rows and columns, i.e., the row and column numbers; N represents the obtained normal vector, and the N wavy line represents the true value normal vector.

[0045] Furthermore, the initial size of the sliding window in step 4 is set to 400x400 and the sliding step is set to 200.

[0046] A second aspect of the present invention provides a system for verifying a defect detection dataset based on photometric stereo surface reconstruction, comprising the following steps:

[0047] (1) Establish a verification model; build a faster-RCNN model, train the faster-RCNN model using a defect detection dataset, and obtain a verification model;

[0048] (2) performing defect detection on metal parts using the verification model obtained in step (1);

[0049] 2.1) Divide the surface of the metal part into several areas based on the topographical characteristics of the metal part and the field of view of the imaging device, and treat each area as a basic detection unit;

[0050] 2.2) Set the six-degree-of-freedom robot's position information and determine the position of the imaging device when capturing images of each area; set the parameters of the photometric stereo imaging device and determine the lighting scheme for imaging each area;

[0051] 2.3) Collect photometric images of each area and input the collected image data into the photometric stereo vision algorithm model to obtain a normal vector map;

[0052] 2.4) Input the normal vector map into the defect detection model to obtain the detection results;

[0053] 2.5) After the detection of a single area is completed, the mobile robot enters the next area and executes the loop until all detection areas are traversed. The detection is completed and the system is reset.

[0054] Preferably, the aster-RCNN model in step 1 is divided into four parts: feature extraction module, region generation network module, ROIPooling layer and target prediction module;

[0055] The feature extraction module uses the ResNet-50 residual network for feature extraction. The residual block in the ResNet-50 network consists of three convolutional layers with convolution kernel sizes of 1x1, 3x3, and 1x1 respectively.

[0056] The region generation network module screens out candidate regions where targets may exist. The input of this module is the feature map obtained by the feature extraction network module. The region is generated using the anchor box mechanism, that is, k rectangular boxes are generated in the original image at each point according to different aspect ratios. The rectangular boxes are used as the initial prediction of regions where defects may exist. When the feature size is m×n, the number of anchor boxes is m×n×k, and the value of k is 9. In target detection, the region where the target exists is called the foreground, and the region where the target does not exist is called the background. The region generation network selects the anchor boxes that may be the foreground from the generated anchor boxes and adjusts the positions of the anchor boxes as the candidate regions predicted by the network. The position information of the candidate regions is output and provided to the subsequent target prediction module for final target classification and position regression.

[0057] The input of ROI Pooling is the location information of the candidate region and the feature map extracted in the previous step. The location parameters of the candidate region are mapped to the feature map, corresponding to the rectangular region on the feature map. The features within the rectangular region are further extracted to generate a fixed-size 7x7 feature map as the output of this layer; the size of the output feature map is set to 7x7.

[0058] The target prediction module performs target classification and position regression based on the feature map of the candidate area obtained by the ROI Pooling layer to obtain the target information in the image; the target prediction module flattens the feature map of the candidate area obtained by the ROI Pooling layer, connects the two fully connected layers, and then inputs them into the fully connected layers for category prediction and position prediction respectively.

[0059] Preferably, an FPN (Feature Pyramid Networks) structure is added to the feature extraction network of ResNet-50 to output multiple features for subsequent region generation and target prediction. Specifically, the feature maps of each stage of ResNet-50 are upsampled with the deepest Feature 5 to obtain feature maps of the same size as Feature 4, 3, and 2, and then spliced ​​to generate feature outputs of different scales. The features of each stage play different roles in the defect detection task. For example, Feature 5 has the smallest size and can be used for the detection of large targets; Feature 2 has the largest size and can be used for the detection of small targets.

[0060] Preferably, the region generation network module first passes through a convolution layer with a convolution kernel size of 3×3, and then inputs the feature map into two convolution layers with a convolution kernel size of 1×1 for target classification and position coordinate regression; the target category classification determines the type of the pre-selected box, each anchor box uses two parameters to indicate whether it is foreground or background, and the number of convolution kernels of the classification feature extraction convolution layer is 2k; the position coordinate regression obtains the offset for anchor box adjustment, each anchor box uses 4 parameters to determine the position and size of the rectangular box, and the number of convolution kernels of the position coordinate feature extraction convolution layer is 4k; through target classification and position regression, a candidate region with position adjustment and category as foreground is obtained; in the model training stage, it is necessary to calculate the region generation network loss function to optimize the classifier and regression parameters. The loss function of RPN includes two parts: classification loss function and boundary regression box loss function. The formula is:

[0061]

[0062] Among them, p i Represents the probability that the i-th prediction box is predicted to be the true label; Represents the true label of the predicted box; t i Represents the bounding box regression parameters of the predicted i-th prediction box; N represents the bounding box regression parameter of the true box corresponding to the i-th predicted box; cls Indicates the number of samples sampled; N reg Represents the number of predicted box positions; λ is a balance parameter;

[0063] The classification loss function calculates the loss of background and foreground parameters predicted by the target classification convolution layer. The specific calculation uses the multi-category cross entropy loss function, and the formula is as follows:

[0064]

[0065] The boundary regression loss function is a loss calculation of the predicted box position and the real box position. The specific calculation uses the smooth-L1 loss function, and the formula is as follows:

[0066]

[0067]

[0068] The beneficial effects of the present invention are:

[0069] (1) A detection dataset preparation method based on photometric stereo vision is designed for reflective metal surface defect detection. By acquiring multiple images under different light source conditions, the information loss caused by surface reflection can be avoided. Photometric stereo vision can obtain surface normal vector maps. The photometric stereo reconstruction algorithm based on three-dimensional convolution has a certain improvement in reconstruction accuracy compared with other mainstream photometric stereo reconstruction algorithms.

[0070] (2) A verification system for defect detection datasets based on photometric stereo surface reconstruction was designed. The overall scheme was verified through systematic experiments, and effective identification of metal surface defects was achieved. BRIEF DESCRIPTION OF THE DRAWINGS

[0071] Figure 1 Images of a stationary metal object under different lighting conditions. (Each row represents an image of the object captured under different lighting directions. The boxes indicate the location of defects. Left: Visible defects. Right: Defects in overexposed or shadowed areas.)

[0072] Figure 2 Schematic diagram of four types of objects in the defect detection dataset production method.

[0073] Figure 3 It is an image processing flowchart of the defect detection dataset preparation method.

[0074] Figure 4 This is the intermediate result of image processing of the defect detection dataset preparation method.

[0075] Figure 5 It is the three-view diagram of the calibration sphere for the defect detection dataset preparation method.

[0076] Figure 6 This is a diagram showing the light source direction solution for the defect detection dataset creation method.

[0077] Figure 7 It is the overall framework diagram of the photometric stereo network model.

[0078] Figure 8 Schematic diagram of input data processing for the photometric stereo network model.

[0079] Figure 9 This is the structural diagram of the initial feature extraction module of the photometric stereo network model.

[0080] Figure 10 It is a block diagram of the 3D feature extraction of the photometric stereo network model.

[0081] Figure 11 It is a three-dimensional feature extraction module structure scheme of the photometric stereo network model.

[0082] Figure 12 This is the structure diagram of the normal vector estimation module of the photometric stereo network model.

[0083] Figure 13 It is a schematic diagram of the photometric stereo synthesis dataset of the photometric stereo network model.

[0084] Figure 14 It is a data set production process based on the sliding window idea.

[0085] Figure 15 This is the Faster RCNN network structure diagram.

[0086] Figure 16 This is the ResNe-50 FPN structure diagram.

[0087] Figure 17 This is the structural diagram of the region generation network module.

[0088] Figure 18 This is the ROI Pooling structure diagram.

[0089] Figure 19 It is the structure diagram of the target prediction module.

[0090] Figure 20 It is a hardware diagram of the verification system.

[0091] Explanation of the accompanying drawings: 1. Six-degree-of-freedom machine; 2. Photometric stereo imaging device; 3. Equipment frame. DETAILED DESCRIPTION

[0092] The following will clearly and completely describe the technical solution of the present invention in conjunction with the accompanying drawings. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0093] In the description of the present invention, it should be noted that the terms "center," "upper," "lower," "left," "right," "vertical," "horizontal," "inner," and "outer" and the like, indicating orientations or positional relationships, are based on the orientations or positional relationships shown in the accompanying drawings and are intended solely to facilitate and simplify the description of the present invention. They are not intended to indicate or imply that the devices or components referred to must have, be constructed, or operate in a specific orientation, and are therefore not to be construed as limitations on the present invention. Furthermore, the terms "first," "second," and "third" are used for descriptive purposes only and are not to be construed as indicating or implying relative importance.

[0094] In the description of the present invention, it should be noted that, unless otherwise expressly specified or limited, the terms "mounted," "connected," and "connected" should be understood in a broad sense. For example, they may refer to fixed connections, detachable connections, or integral connections; mechanical connections or electrical connections; direct connections or indirect connections through an intermediate medium; and internal communication between two components. Those skilled in the art will understand the specific meanings of the above terms in the present invention in specific contexts.

[0095] The present invention provides a method for preparing a defect detection data set based on photometric stereo surface reconstruction, comprising the following steps:

[0096] (1) Four types of objects are selected to generate a data set; the four types include a yellow stainless steel hemisphere, a silver stainless steel hemisphere, a yellow cone, and a silver cone. Each object is subjected to impact simulation during the production process to generate defects on its surface. The types of defects include pits, deformations, scratches, and abrasions.

[0097] (2) acquiring multiple images under different lighting conditions; sequentially stimulating light sources at different positions on each object to illuminate the object, and acquiring a set of photometric images of the object's surface area;

[0098] (3) Inputting the captured photometric image into the photometric stereo network model, using the trained photometric stereo network model to perform model inference based on the photometric image to estimate the surface normal vector at each pixel point, and obtaining the object surface normal vector map;

[0099] The photometric stereo network model includes the following steps:

[0100] 3.1) Establish an imaging environment model of the acquisition system and calibrate the light source information of the photometric stereo imaging device. The photometric stereo algorithm requires obtaining a light source system model of the photometric stereo image acquisition device. That is, establish a model of the industrial camera and LED light source, calibrate the light source direction in the camera coordinate system, and calibrate the light source intensity due to the differences in LED light sources. The acquired photometric image must be normalized to eliminate the changes in photometric values ​​caused by the differences in LED light sources.

[0101] 3.1.1) Light source direction calibration

[0102] The direction of the light source can be calibrated using the principle of mirror reflection. A chrome-plated metal sphere is used as the calibration sphere. The incident light from the light source strikes the surface of the metal sphere, reflects off the surface, and enters the camera, which captures the reflected light. In the image captured by the camera, a highlight appears on the surface of the metal sphere. This is the location where the incident light, parallel to the camera's viewing angle, is reflected. The direction of the line connecting this point and the center of the sphere is the direction of the normal to the sphere's surface. Due to the principle of mirror reflection, the direction of the incident light from the light source and the direction of the reflected light are symmetrical relative to the normal. Based on the relationship between this set of vectors, the direction of the incident light can be inferred.

[0103] The normal vector at the reflection point on the sphere, the sum of the direction vectors of the reflected light and the incident light are all unit vectors. The direction of the reflected light is parallel to the camera's viewing angle. The direction vector of the reflected light, that is, the viewing angle, is fixed to [0,0,1]. The normal vector of a point on the sphere is the difference between the coordinates of the point on the sphere and the coordinates of the center of the sphere. This paper uses image processing to solve the position of the highlight point on the surface of the steel ball, and then obtains the surface normal vector at that point. First, the center of the calibration ball is obtained. The image processing process is as follows: Figure 3 shown.

[0104] The input image is sequentially subjected to gradient calculation, morphological processing (erosion and expansion), connected domain analysis (deletion of small areas) and Canny edge detection, and the center and radius of the circle are found using Hough transform, and displayed on the original image. The processing results of each step are as follows Figure 4 shown.

[0105] Get the parameter center coordinates (x C ,y C ) and radius r, process the highlight point image, and obtain the coordinates of the highlight point P (x P ,y P ), calculate the spherical normal vector at this point. Figure 5 From the three-view image of the calibration sphere shown, we can see that the relative coordinates of the highlight point relative to the center of the circle are the x and y components of the normal vector. The z component can be calculated according to the Pythagorean theorem. N is expressed as:

[0106]

[0107] Now that the reflection direction vector and the normal direction vector are known, the incident direction vector to be obtained can be obtained by the following method. First, the reflection direction vector is calculated according to Figure 5 Translate and intersect with the normal direction vector extended vector S. According to the law of reflection of light, vectors L, R and S form an isosceles triangle. The incident direction vector L can be expressed as:

[0108] L=RS (2)

[0109] Vector S has the same direction as the normal vector and can be represented by vectors R and N:

[0110]

[0111] Combining the above formula, we can get:

[0112] L=R-2(R·N)N (4)

[0113] 3.1.2) Light source intensity calibration

[0114] Due to the difference in the actual brightness of the LED light source used in the actual image acquisition system and the difference in the installation position, the light intensity of each light source in the field of view is different, which will affect the calculation process. The light source intensity needs to be normalized to eliminate the intensity difference of each light source. The method used is to use a standard whiteboard and regard the standard whiteboard as an ideal Lambertian surface. The shape of the standard whiteboard is an ideal plane, and the normal vectors of each point are consistent. The plane normal vector n of the whiteboard is std =(0,0,1) T The photometric stereo formula for calibrating a whiteboard is as follows:

[0115]

[0116] Among them, the pixel response value of the whiteboard surface I std , light source direction l, whiteboard plane normal vector All are known parameters, kc is the camera internal parameter, Elight is the light intensity, T is the exposure time, ρ white is the reflectivity of the white board; rearranging the above formula yields:

[0117]

[0118] The left side of the above formula is a constant. By calculating the ratio of the constants between the light sources and then normalizing them, the relative intensity factors of the light sources are obtained.

[0119] 3.2) Build a 3D convolution-based photometric stereo network model and train it using a synthetic photometric stereo dataset until the model converges on the training dataset.

[0120] The 3D convolution-based network model consists of three parts: an initial feature extraction module, a 3D feature extraction module, and a normal map estimation module. The model input is a 3D matrix formed by concatenating the photometric images of each frame and the light source direction matrix. The initial feature extraction module abstracts the multi-channel data matrix of each frame into single-channel features, facilitating the subsequent extraction of inter-frame features. The 3D feature extraction module extracts features from both the inter-frame and spatial dimensions, then normalizes the number of channels in the feature map through a max-pooling operation, enabling the model to handle variations in the number of input images. The normal map estimation module converts the feature map into surface normals through a fully connected layer, outputting a normal map.

[0121] 3.2.1) Network model input design

[0122] Assume that q color images of the target object are obtained under different light source directions, where the image width is W and the image height is H. The number of image pixels p describing the object can be expressed as p = W × H. The surface normal vector N∈R 3×p , the light source direction L∈R of q images 3×q, luminosity matrix I∈R 3×p×q The relationship between can be expressed as:

[0123] I = Θ·repmat(max(N T L, 0), 3)(3-1)

[0124] Where Θ∈R 3×p×q Represents a complex function related to the surface normal, light source direction, and viewing angle. This chapter uses a designed photometric stereo network model to directly estimate the normal matrix N based on the photometric matrix I and the light source direction matrix L, without having to establish a mathematical model for the complex function Θ.

[0125] The input data is the luminosity matrix and the calibration matrix of the light source direction and light source intensity. In order to facilitate the model to process the data, the input data needs to be adjusted. Figure 8 As shown in the figure, the size of each photometric image frame is 3×W×H, and the corresponding light source direction and light source intensity calibration parameters are 3×1 vectors. The light source direction parameters are copied into the one-dimensional vector of the light source direction according to the spatial dimension to generate a 3×W×H light source direction map, which is then concatenated with the corresponding photometric image to generate a 6×W×H matrix. Then, the q photometric images are concatenated in the spatial dimension to generate a (6×q)×W×H three-dimensional matrix as the input of the model.

[0126] 3.2.2) Initial feature extraction module design

[0127] The initial feature extraction module performs a convolution operation on the input (6×q)×W×H data matrix to obtain a feature map for each frame, preparing for the subsequent extraction of spatiotemporal features using three-dimensional convolution. It consists of a three-dimensional convolution layer, an activation function layer, and a dropout layer. First, the original matrix is ​​convolved with a 6×1×1 three-dimensional convolution kernel. The convolution kernel's spatial dimensions match those of the single-frame luminosity and light source direction data matrix, extracting features from the single-frame data. The LeakyReLU function is used as the activation function, and the dropout layer uses random dropout to prevent overfitting.

[0128] 3.2.3) 3D feature extraction module design

[0129] Through the initial feature extraction module, the feature map of the photometric data of each frame is obtained. In the three-dimensional feature extraction module, the inter-frame feature extraction block and the spatial feature extraction block are respectively deployed to simultaneously analyze the change rules of the inter-frame and spatial photometric information.

[0130] The inter-frame feature extraction block consists of a three-dimensional convolution layer, a LeakyReLU activation layer, and a Dropout layer, as shown in Figure 10As shown in a. The convolution kernel size of the 3D convolution layer is M×1×1, and its function is to process the brightness changes of single pixels in M ​​adjacent frames. The inter-frame information plays an important role in solving the problem of abnormal points caused by shadows, highlights, etc. The spatial feature extraction block consists of a 3D convolution layer and a LeakyReLU activation layer, as shown in Figure 10 As shown in the figure, the convolution kernel size is 1×N×N. A 3D convolutional layer is used to capture local pixel intensity variations. This local information improves performance when surface reflectivity changes, while also ensuring robustness under sparse illumination. To strike a balance between model complexity and performance, three inter-frame feature extraction blocks and three spatial feature extraction blocks are deployed.

[0131] The deployment order of submodules in the 3D feature extraction module has a great impact on the model's ability to extract features. There are three deployment orders, such as Figure 11 As shown. The first solution is to first deploy the inter-frame feature extraction block and then deploy the spatial feature extraction block; the second solution is to first deploy the spatial feature extraction block and then deploy the inter-frame feature extraction block; the third solution is a mixed deployment of the spatial feature extraction block and the inter-frame feature extraction block. The choice of the three solutions is judged through experiments. Specifically, the model is implemented according to the network structure of the three solutions and trained in the same dataset. Table 3.1 is the test results in the DiLiGenT dataset. Table 3.1 Network structure comparison test

[0132] plan BALL CAT POT1 BEAR POT2 BUDD. GOBL. READ. COW HARV. Avg. Option 1 2.82 5.79 6.92 5.59 7.55 6.93 7.98 11.85 7.82 14.18 7.74 Option 2 2.58 6.02 6.99 6.62 8.68 7.34 9.46 13.68 8.79 15.95 8.61 Option 3 2.76 5.99 7.24 6.84 7.79 7.23 8.39 11.71 8.32 14.79 8.81

[0133] Based on experimental data, solution 1 performs best. Both inter-frame and spatial information play an important role in normal vector estimation, but inter-frame information is sensitive to outliers like highlights and shadows, and provides more useful information when estimating normal vectors on non-Lambertian surfaces.

[0134] 3.2.4) Normal vector estimation module design

[0135] Before the normal vector estimation module, a Max-pooling layer is designed to aggregate the most significant features extracted from the data of different light source directions, so that the model outputs a feature map of a fixed channel. The role of this link is to enable the model to be applicable to data input with different numbers of light sources. When the number of light sources changes, the feature map obtained by the feature extraction module does not match the input dimension of the normal vector estimation module. When the Max-pooling layer is added, the feature extraction modules of different dimensions can be output as feature maps of fixed dimensions. The normal vector estimation module uses the feature map to estimate the normal vector map after the max-pooling operation. The normal vector estimation module consists of two layers of fully connected layers. The specific structure is as follows Figure 12 shown.

[0136] 3.2.5) Loss Function Design

[0137] The photometric stereo network model is trained by minimizing the error between the predicted normal vector and the true normal vector at each point. A normal vector is essentially a unit space vector, and the difference between two unit space vectors can be reflected by the angle between them. The cosine value of two vectors is usually calculated to measure the difference in direction. Therefore, the cosine similarity loss function is used, as follows:

[0138]

[0139] Comparative test results and analysis

[0140] Model training and testing

[0141] The photometric stereo model is implemented based on the Pytorch framework, using a GeForce GTX1080Ti hardware platform and 96GB of RAM. The specific training parameter settings are shown in Table 3.2.

[0142] Table 3.2 Training parameter settings

[0143]

[0144] The photometric stereo model requires the use of the target's true normal vector map to supervise the model training. In order to obtain sufficient training data, this paper uses a synthetic dataset for model training and testing. In order to verify the model's ability to estimate the surface normal vectors of real objects, a publicly available photometric stereo dataset of real objects is used. The synthetic datasets are the Bloddy dataset and the Sculpture dataset, such as Figure 13 As shown in Figure 2, the Bloddy dataset uses a fluid-like model constructed from different imaging perspectives. This dataset features irregularly shaped objects with smooth surfaces. Compared to the Bloddy dataset, the Sculpture dataset contains complex normal vector distributions. The total number of samples in both datasets is 85,212. The real-world object dataset used for verification is the DiLiGenT dataset, which contains 10 objects with complex materials and various shapes.

[0145] 3.3) Evaluation indicators

[0146] The performance of the photometric stereo model is evaluated by the surface normal estimation accuracy, which is calculated as the mean angular error (MAE) between the predicted normal vector and the true normal vector. The calculation formula is:

[0147]

[0148] Among them, N k Represents the normal vector at the kth pixel in the predicted normal vector map, Represents the normal vector at the kth pixel in the true normal vector map, and K represents the total number of pixels in the normal vector map. A smaller MAE value indicates a higher accuracy in normal vector estimation.

[0149] (4) Based on the normal vector map obtained in step (3), a defect detection dataset based on photometric stereo surface reconstruction is constructed;

[0150] 4.1) Use the Python annotation script LabelImg to mark the defect target area with a rectangular box;

[0151] 4.2) Set the initial size of the sliding window to 400x400 and the step size to 200;

[0152] 4.3) Read the annotation file of each image, obtain the location and category information of each defect in the image, and traverse the image starting from the upper left corner;

[0153] 4.4) Determine whether there is a defective target in the area. If there is no defective target, discard the image cropped by the window; if there is a defect, save the information of the defective area that intersects with the window;

[0154] 4.5) Determine whether the window of the regional information saved in step (4.4) contains an incomplete defect. If not, convert the position information of the defect to the image coordinate system of the window, save it and generate a new annotation file; if there is an incomplete defect area, calculate the ratio of the area of ​​the defect in the window to the area of ​​the actual annotated defect area, and judge by a threshold. When the ratio is greater than 0.75, adjust the annotation information of the defect and use the window boundary as the new boundary. When the ratio is less than 0.75, discard the defect, save the window image and generate a new annotation file; after completion, obtain the defect sample image;

[0155] After statistics, a total of 711 defect samples were obtained. These samples were divided into training, validation, and test sets in a ratio of 3:1:1. The number of samples in each category after the division is complete is shown in Table 4.1.

[0156] Table 4.1 Defect sample statistics

[0157]

[0158] The standard images of the dataset were generated by the sliding window-based dataset production method. After the production was completed, a total of 2017 defect sample images were obtained. The training set and test set data are shown in Table 4.2.

[0159] Table 4.2 Dataset statistics

[0160]

[0161] A second aspect of the present invention provides a system for verifying a defect detection dataset based on photometric stereo surface reconstruction, comprising the following steps:

[0162] (1) Establish a verification model; build a faster-RCNN model, train the faster-RCNN model using a defect detection dataset, and obtain a verification model;

[0163] (2) performing defect detection on the metal part using the defect detection model obtained in step (1);

[0164] 2.1) Divide the surface of the metal part into several areas based on the topographical characteristics of the metal part and the field of view of the imaging device, and treat each area as a basic detection unit;

[0165] 2.2) Set the six-degree-of-freedom robot's position information and determine the position of the imaging device when capturing images of each area; set the parameters of the photometric stereo imaging device and determine the lighting scheme for imaging each area;

[0166] 2.3) Collect photometric images of each area and input the collected image data into the photometric stereo vision algorithm model to obtain a normal vector map;

[0167] 2.4) Input the normal vector map into the defect detection model to obtain the detection results;

[0168] 2.5) After the detection of a single area is completed, the mobile robot enters the next area and executes the loop until all detection areas are traversed. The detection is completed and the system is reset.

[0169] The Faster RCNN model consists of four parts: a feature extraction module, a region generation network module, a ROI Pooling layer, and a target prediction module. The model's input is a surface normal map, and its output is the predicted defect category and location information. The feature extraction module acts as a feature extractor to obtain the feature map of the input image. The region generation network module generates candidate regions and generates prior frames, called anchor frames, based on the pixels in the feature map. These frames are then divided into foreground and background based on the anchor frame's regional features. By combining category information with bounding box regression parameters, the anchor frames predicted as foreground are position-corrected and output as predicted candidate regions. The ROI Pooling layer obtains and standardizes the feature images corresponding to the candidate regions. Through the Pooling operation, the two-dimensional feature map of each candidate region is adjusted to a fixed-size feature vector output. The target prediction module then performs category classification and position regression based on the feature vectors.

[0170] The Faster RCNN benchmark model uses the first 13 convolutional layers of the VGG-16 network model as the feature extraction network. Since activation function layers and convolutional layers are often used together in convolutional neural networks, they will be collectively referred to as convolutional layers below. The sequence is two convolutional layers, a max pooling layer, two convolutional layers, a max pooling layer, three convolutional layers, a max pooling layer, three convolutional layers, a max pooling layer, and three convolutional layers. Experiments have shown that this detection model performs poorly. This paper uses the ResNet-50 residual network for feature extraction, which effectively improves the detection capabilities of the defect detection network.

[0171] Residual networks, by designing a residual structure that enables the network to learn the difference between input and output, effectively address the degradation problem of deep neural networks, reduce the loss of effective information, and improve the network's feature extraction capabilities. The residual block is the core of the residual network design. A residual block consists of multiple cascaded convolutional layers and a shortcut connection. Deep networks are constructed by connecting multiple residual blocks in series. The identity mapping and residual mapping within the residual blocks ensure that network performance does not degrade as the depth of the residual network increases.

[0172] The residual block in the ResNet-50 network used in the present invention consists of three convolutional layers with convolution kernel sizes of 1x1, 3x3, and 1x1, respectively;

[0173] Table 5.1 Specific structure of feature extraction network

[0174]

[0175]

[0176] The feature extraction module uses the ResNet-50 residual network for feature extraction. The residual block in the ResNet-50 network is composed of three convolutional layers with convolution kernel sizes of 1x1, 3x3, and 1x1, respectively. The FPN (Feature Pyramid Networks) structure is added to the feature extraction network of ResNet-50 to output multiple features for subsequent region generation and target prediction. Specifically, the feature maps of each stage of ResNet-50 are upsampled with the deepest layer Feature5 to obtain feature maps of the same size as Features4, 3, and 2, and then spliced ​​to generate feature outputs of different scales. The features of each stage play different roles in the defect detection task. For example, Feature 5 has the smallest size and can be used for detecting large targets; Feature 2 has the largest size and can be used for detecting small targets.

[0177] The region generation network module screens out candidate regions where targets may exist. The input of this module is the feature map obtained by the feature extraction network module. The region is generated using the anchor box mechanism, that is, k rectangular boxes are generated in the original image at each point according to different aspect ratios. The rectangular boxes are used as the initial prediction of regions where defects may exist. When the feature size is m×n, the number of anchor boxes is m×n×k, and the value of k is 9. In target detection, the region where the target exists is called the foreground, and the region where the target does not exist is called the background. The region generation network selects the anchor boxes that may be the foreground from the generated anchor boxes and adjusts the positions of the anchor boxes as the candidate regions predicted by the network. The position information of the candidate regions is output and provided to the subsequent target prediction module for final target classification and position regression.

[0178] The input of ROI Pooling is the location information of the candidate region and the feature map extracted in the previous step. The location parameters of the candidate region are mapped to the feature map, corresponding to the rectangular region on the feature map. The features within the rectangular region are further extracted to generate a fixed-size 7x7 feature map as the output of this layer; the size of the output feature map is set to 7x7.

[0179] The target prediction module performs target classification and position regression based on the feature map of the candidate area obtained by the ROI Pooling layer to obtain the target information in the image; the target prediction module flattens the feature map of the candidate area obtained by the ROI Pooling layer, connects the two fully connected layers, and then inputs them into the fully connected layers for category prediction and position prediction respectively.

[0180] The region generation network module first passes through a convolution layer with a convolution kernel size of 3×3, and then inputs the feature map into two convolution layers with a convolution kernel size of 1×1 for target classification and position coordinate regression; the target category classification determines the type of pre-selected box, each anchor box uses two parameters to indicate whether it is foreground or background, and the number of convolution kernels in the classification feature extraction convolution layer is 2k; the position coordinate regression obtains the offset for anchor box adjustment, each anchor box uses 4 parameters to determine the position and size of the rectangular box, and the number of convolution kernels in the position coordinate feature extraction convolution layer is 4k; through target classification and position regression, the candidate region with position adjustment and category as foreground is obtained; in the model training stage, the region generation network loss function needs to be calculated to optimize the classifier and regression parameters. The loss function of RPN includes two parts: classification loss function and boundary regression box loss function, and the formula is:

[0181]

[0182] Among them, p i Represents the probability that the i-th prediction box is predicted to be the true label; Represents the true label of the predicted box; t iRepresents the bounding box regression parameters of the predicted i-th prediction box; N represents the bounding box regression parameter of the true box corresponding to the i-th predicted box; cls Indicates the number of samples sampled; N reg Represents the number of predicted box positions; λ is a balance parameter;

[0183] The classification loss function calculates the loss of background and foreground parameters predicted by the target classification convolution layer. The specific calculation uses the multi-category cross entropy loss function, and the formula is as follows:

[0184]

[0185] The boundary regression loss function is a loss calculation of the predicted box position and the real box position. The specific calculation uses the smooth-L1 loss function, and the formula is as follows:

[0186]

[0187]

[0188] The hardware structure of the verification system of the present invention refers to Figure 20 , including a photometric stereo imaging device, a motion execution module and a device frame;

[0189] The device frame is a rectangular frame, the upper half of which is encapsulated with a stainless steel plate, and the interior of the rectangular frame is covered with a black light-shielding cloth to prevent the stainless steel plate from reflecting the light source and affecting the acquisition of the photometric information of the detection object;

[0190] The motion execution module includes a six-degree-of-freedom robot, a control box, and a host computer. The six-degree-of-freedom robot is installed at the top of the equipment frame, and a photometric stereo imaging device is installed at the end of the six-degree-of-freedom robot. The six-degree-of-freedom robot is electrically connected to the control box, and the control box is communicatively connected to the host computer. The host computer plans the robot's posture and movement trajectory and sends movement instructions to the control box. The control box receives the movement instructions sent by the host computer and controls the movement of the six-degree-of-freedom robot, so that the photometric stereo imaging device performs image acquisition at the designed point.

[0191] The photometric stereo imaging device includes an LED light source, an industrial camera, a mounting frame, and a Raspberry Pi microcomputer. The mounting frame is in the shape of a "mu" character frame, and multiple LED light sources are evenly distributed on the frame edges of the mounting frame; an industrial camera is installed at the center position of the mounting frame, and a Raspberry Pi microcomputer is installed on the frame edges of the mounting frame; the Raspberry Pi microcomputer is electrically connected to the LED light source and the industrial camera respectively, and the Raspberry Pi microcomputer is communicatively connected to the host computer; the host computer sends an image acquisition instruction to the Raspberry Pi microcomputer, the Raspberry Pi microcomputer receives the image acquisition instruction and controls the LED light source to light up and the industrial camera to take pictures, and after the image acquisition is completed, the Raspberry Pi microcomputer sends an acquisition success instruction to the host computer.

[0192] To verify the effectiveness of the above verification system, experimental verification was carried out from two aspects: the defect detection accuracy rate and the single detection running time of the system. a

[0193] Select 100 defective defective parts in the test set and form a test sample set with 100 non-defective parts. The system performs regional detection on each part. When a defect exists, the system is determined to be a defective product. If there is no defect after the operation is completed, it is determined to be a non-defective product. The experimental results are shown in Table 5.1. Through comprehensive calculation, the detection accuracy rate of the system is 89.5%.

[0194] Table 5.2

[0195]

[0196] During the surface detection process of the parts by the system, it is necessary to traverse each detection area. The time consumption in each detection area is divided into the robot movement time, the imaging device photographing time, and the detection algorithm running time. Among them, the detection algorithm takes the longest time, and the single detection time of the system is determined by the detection algorithm running time. The experimental results of the average time consumption and the overall time consumption of each item of the detection system are shown in Table 5.3;

[0197] Table 5.3

[0198]

[0199] The content described in the embodiments of this specification is only a list of the implementation forms of the inventive concept. The protection scope of the present invention should not be regarded as limited to the specific forms stated in the embodiments. The protection scope of the present invention also extends to equivalent technical means that can be conceived by those skilled in the art according to the inventive concept of the present invention.

Claims

1. A method for preparing a defect detection dataset based on photometric stereo surface reconstruction, characterized in that: The following steps are involved: (1) Construct a defect sample set, which consists of multiple samples. The sample attributes include color, material, shape, and defect type. Each sample has at least one attribute that is different from other samples. (2) stimulating light sources at different positions of each object in turn to illuminate the object, obtaining an image by stimulating the light source at each position, and forming a set of photometric images with at least three images; (3) Inputting the photometric image into the photometric stereo network model, using the trained photometric stereo network model to perform model inference based on the photometric image to estimate the surface normal vector at each pixel point, and obtain the object surface normal vector map; training the photometric stereo network model includes the following steps: 3.1) Establish an imaging environment model for the acquisition system and calibrate the light source information of the photometric stereo imaging device; 3.2) Build a 3D convolution-based photometric stereo network model and train it using a synthetic photometric stereo dataset until the model converges on the training dataset. The synthetic photometric stereo datasets are the Bloddy and Sculpture datasets, and the real-world object dataset used for verification is the DiLiGenT dataset. 3.3) Design evaluation metrics for the photometric stereo network model. The performance of the photometric stereo model is evaluated by the surface normal estimation accuracy. The surface normal estimation accuracy is calculated as the average angular error between the predicted normal vector and the true normal vector. The calculation formula is: Among them, N k Represents the normal vector at the kth pixel in the predicted normal vector map, Represents the normal vector at the kth pixel in the true normal vector map, and K represents the total number of pixels in the normal vector map. The smaller the MAE value, the higher the estimation accuracy of the normal vector. (4) Based on the normal vector map obtained in step (3), a defect detection dataset based on photometric stereo surface reconstruction is constructed; 4.1) Use the Python annotation script LabelImg to mark the defect target area with a rectangular box; 4.2) Read the annotation file for each image, obtain the location and category information of each defect in the current image, set a sliding window, and traverse the image with the sliding window starting from the upper left corner of the current image; 4.3) Determine whether there is a defective target within the current sliding window. If there is no defective target, discard the image cropped by the window; if there is a defect, save the information of the defective area that intersects with the sliding window; 4.4) Determine whether the window of the regional information saved in step (4.3) contains an incomplete defect. If not, convert the position information of the defect to the image coordinate system of the window, save it and generate a new annotation file; when an incomplete defect area exists, calculate the ratio of the area of ​​the defect in the window to the area of ​​the actual annotated defect area, and judge by a threshold. When the ratio is greater than 0.75, adjust the annotation information of the defect and use the window boundary as the new boundary. When the ratio is less than 0.75, discard the defect, save the window image and generate a new annotation file; after completion, obtain a defect sample image; 4.5) Divide the defect sample images obtained in step 4.6) into a training set, a validation set, and a test set.

2. The method for preparing a defect detection dataset based on photometric stereo surface reconstruction according to claim 1, wherein: The defect sample set includes at least four types of defect samples; the four types include yellow hemisphere, silver hemisphere, yellow cone and silver cone of stainless steel, and each sample has defects on its surface through the impact of simulating the production process, and the types of defects include at least point pits, deformation, scratches and abrasions.

3. The method for preparing a defect detection dataset based on photometric stereo surface reconstruction according to claim 1, wherein: The light source calibration includes the following steps:

1. Light source direction calibration; using a chrome-plated metal ball as the calibration ball, first obtain the calibration ball center, solve the position of the highlight point on the steel ball surface through image processing, and then obtain the surface normal vector at that point; Get the parameter center coordinates (x C ,y C ) and radius r, process the highlight point image, and obtain the coordinates of the highlight point P (x P ,y P ), calculate the spherical normal vector at that point; according to the calibration sphere three-view image, the relative coordinates of the highlight point compared to the center of the circle are the x and y components of the normal vector, and the z component is calculated according to the Pythagorean theorem. N is expressed as: Given the reflection direction vector and the normal direction vector, the incident direction vector can be obtained by the following method: First, translate the reflection direction vector and intersect it with the extended normal direction vector S. According to the law of reflection of light, vectors L, R, and S form an isosceles triangle; the incident direction vector L can be expressed as: L= RS (2) Vector S has the same direction as the normal vector and can be represented by vectors R and N: Combining the above formula, we can get: L=R-2(R·N)N (4) 3.1.2) Light source intensity calibration: Use a standard whiteboard and treat it as an ideal Lambertian surface. The shape of the standard whiteboard is an ideal plane, and the normal vectors of each point are consistent. The plane normal vector n of the whiteboard is std =(0,0,1) T ; The photometric stereo formula for calibrating a whiteboard is as follows: Among them, the pixel response value of the whiteboard surface I std , light source direction l, whiteboard plane normal vector All are known parameters, kc is the camera internal parameter, Elight is the light intensity, T is the exposure time, ρ white is the reflectivity of the white board; rearranging the above formula yields: The left side of the above formula is a constant. By calculating the ratio of the constants between the light sources and then normalizing them, the relative intensity factors of the light sources are obtained.

4. The method for preparing a defect detection dataset based on photometric stereo surface reconstruction according to claim 1, wherein: Building a photometric stereo network model based on 3D convolution specifically includes the following steps: 3.2.1) Design the network model input for the photometric stereo network model. The input data is the photometric matrix and the calibration matrix of the light source direction and light source intensity. The size of each photometric image frame is 3×W×H, and the corresponding light source direction and light source intensity calibration parameters are 3×1 vectors, where the image width is W and the image height is H. The number of image pixels p representing the object can be expressed as p=W×H. The light source direction parameters and the one-dimensional light source direction vector are replicated according to the spatial dimension to generate a 3×W×H light source direction map. This is concatenated with the corresponding photometric image to generate a 6×W×H matrix. Then, the q photometric images are concatenated in the spatial dimension to generate a (6×q)×W×H three-dimensional matrix as the model input. 3.2.2) Design the initial feature extraction module of the photometric stereo network model. This module performs a convolution operation on the input (6×q)×W×H data matrix to obtain the feature map of each frame, preparing for the subsequent extraction of spatiotemporal features using 3D convolution. The initial feature module consists of a 3D convolution layer, an activation function layer, and a dropout layer. First, a 6×1×1 3D convolution kernel is used to convolve the original matrix. The spatial dimensions of the convolution kernel are the same as the spatial dimensions of the single-frame photometric and light source direction data matrix to extract the features of the single-frame data. The activation function uses the LeakyReLU function, and the dropout layer uses random dropout to prevent overfitting of the model. 3.2.3) Design a 3D feature extraction module for the photometric stereo network model to obtain a feature map of the photometric data for each frame. Within this module, deploy inter-frame and spatial feature extraction blocks to simultaneously analyze the inter-frame and spatial photometric information variation patterns. The inter-frame feature extraction block consists of a 3D convolutional layer, a LeakyReLU activation layer, and a Dropout layer. The convolution kernel size of the 3D convolutional layer is M×1×1, and its function is to process the photometric variations of single pixels across M adjacent frames. The spatial feature extraction block consists of a 3D convolutional layer and a LeakyReLU activation layer. The convolution kernel size is 1×N×N, and the 3D convolutional layer is used to obtain the intensity variations between local pixels. This local information can improve performance when the surface reflectivity of an object changes, while also being robust under sparse illumination. 3.2.4) Design a normal vector estimation module for the photometric stereo network model. Add a max-pooling layer before the normal vector estimation module. Convert the output of the feature extraction module with different dimensions into a fixed-dimensional feature map. The normal vector estimation module uses the feature map after the max-pooling operation to estimate the normal map. The normal vector estimation module consists of two fully connected layers. 3.2.5) Design a loss function for the photometric stereo network model. The photometric stereo network model works by reducing the error between the predicted normal vector and the true normal vector at each point. The normal vector is essentially a unit space vector, and the difference between two unit space vectors can be reflected by the angle between the two vectors. The cosine similarity loss function is used, and the formula is as follows: Where H is the image length, W is the image width, i and j represent the number of rows and columns, i.e., the row and column numbers; N represents the obtained normal vector, and the N wavy line represents the true value normal vector.

5. The method for preparing a defect detection dataset based on photometric stereo surface reconstruction according to claim 1, wherein: The initial size of the sliding window in step 4 is set to 400x400 and the sliding step is set to 200.

6. The verification system for the method for producing a defect detection dataset based on photometric stereo surface reconstruction according to any one of claims 1 to 5, characterized in that: S1, establish a verification model; build a faster-RCNN model, train the faster-RCNN model using the defect detection dataset, and obtain a verification model; S2, performing defect detection on the metal part using the verification model obtained in step S1; S2.

1. Divide the surface of the metal part into several areas based on the topographical characteristics of the metal part and the field of view of the imaging device, and treat each area as a basic inspection unit; S2.2, set the posture information of the six-degree-of-freedom robot and determine the posture of the imaging device when collecting images of each area; set the parameters of the photometric stereo imaging device and determine the lighting scheme when imaging each area; S2.3, collecting photometric images of each area, and inputting the collected image data into the photometric stereo vision algorithm model to obtain a normal vector map; S2.4, inputting the normal vector map into the defect detection model to obtain the detection results; S2.5, the detection of a single area is completed, the mobile robot enters the next area position, and the cycle is executed until all detection areas are traversed. The detection is completed and the system is reset.

7. The verification system according to claim 6, wherein: In step S1, the Faster-RCNN model is divided into four parts: feature extraction module, region generation network module, ROI Pooling layer and target prediction module; The feature extraction module uses the ResNet-50 residual network for feature extraction. The residual block in the ResNet-50 network consists of three convolutional layers with convolution kernel sizes of 1x1, 3x3, and 1x1 respectively. The region generation network module selects candidate regions where objects may be present. The input of this module is the feature map obtained by the feature extraction network module. The region is generated using the anchor box mechanism, that is, at each point in the original image, k rectangular boxes are generated according to different aspect ratios. These rectangular boxes are used as the initial prediction of the region where defects may be present. When the feature size is m×n, the number of anchor boxes is m×n×k, with k being 9. In target detection, the area where the target exists is called the foreground, and the area where the target does not exist is called the background. The region generation network selects the anchor boxes that may be the foreground from the generated anchor boxes and adjusts the positions of the anchor boxes as the candidate regions predicted by the network. The position information of the candidate regions is output and provided to the subsequent target prediction module for final target classification and position regression. The input of the ROI Pooling is the location information of the candidate region and the feature map extracted in the early stage. The location parameters of the candidate region are mapped on the feature map, corresponding to the rectangular area on the feature map; the features within the rectangular area are further extracted to generate a feature map of a fixed size of 7x7 as the output of this layer; the size of the output feature map is set to 7x7; The target prediction module performs target classification and position regression based on the feature map of the candidate area obtained by the ROI Pooling layer to obtain the target information in the image; the target prediction module flattens the feature map of the candidate area obtained by the ROI Pooling layer, connects the two fully connected layers, and then inputs them into the fully connected layers for category prediction and position prediction respectively.

8. The verification system according to claim 7, wherein: An FPN structure is added to the feature extraction network of ResNet-50 to output multiple features for subsequent region generation and target prediction. Specifically, the feature maps of each stage of ResNet-50 are upsampled with the deepest layer, Feature 5, to obtain feature maps of the same size as Features 4, 3, and 2, which are then concatenated to generate feature outputs of different scales. The features at each stage play different roles in defect detection tasks.

9. The verification system according to claim 7, wherein: The region generation network module first passes through a convolution layer with a convolution kernel size of 3×3, and then inputs the feature map into two convolution layers with a convolution kernel size of 1×1 for target classification and position coordinate regression; the target category classification determines the type of pre-selected box, each anchor box uses two parameters to indicate whether it is foreground or background, and the number of convolution kernels in the classification feature extraction convolution layer is 2k; the position coordinate regression obtains the offset for anchor box adjustment, each anchor box uses 4 parameters to determine the position and size of the rectangular box, and the number of convolution kernels in the position coordinate feature extraction convolution layer is 4k; through target classification and position regression, the candidate region with position adjustment and category as foreground is obtained; in the model training stage, the region generation network loss function needs to be calculated to optimize the classifier and regression parameters. The loss function of RPN includes two parts: classification loss function and boundary regression box loss function, and the formula is: Among them, p i Represents the probability that the i-th prediction box is predicted to be the true label; Represents the true label of the predicted box; t i Represents the bounding box regression parameters of the predicted i-th prediction box; N represents the bounding box regression parameter of the true box corresponding to the i-th predicted box; cls Indicates the number of samples sampled; N reg Represents the number of predicted box positions; λ is a balance parameter; The classification loss function calculates the loss of background and foreground parameters predicted by the target classification convolution layer. The specific calculation uses the multi-category cross entropy loss function, and the formula is as follows: The boundary regression loss function is a loss calculation of the predicted box position and the real box position. The specific calculation uses the smooth-L1 loss function, and the formula is as follows:

Images