A high-precision three-dimensional shape and deformation measurement method based on projection imaging

Abstract

This invention provides a high-precision three-dimensional shape and deformation measurement method based on projection imaging. This method uses a camera-projector system to combine grid line projection technology with digital image correlation technology to measure the three-dimensional shape and deformation of an object. The camera-projector system is calibrated to determine the internal and external parameters of the camera and projector. A convolutional neural network is used to improve the quality of grid line images affected by speckle. A phase shift method is used to determine the phase of the grid lines, and the calibration results are combined to calculate the three-dimensional shape of the object before and after deformation. The positional relationship between corresponding points before and after deformation is matched with the speckle image to determine the object's three-dimensional displacement. Three-dimensional strain is then calculated from the three-dimensional displacement. This invention improves the accuracy of three-dimensional shape and deformation measurement.

Description

Technical Field

[0001] The present invention relates to the field of optical measurement experimental solid mechanics and image measurement technology, and in particular to a high-precision three-dimensional shape and deformation measurement method based on projection imaging. Background Art

[0002] Raster projection and digital image correlation techniques are developing rapidly and are widely used in the field of three-dimensional measurement. Raster projection utilizes a grating modulated by the object's surface shape to determine the phase value at each pixel, point by point. The object's height is then calculated based on this phase value. Digital image correlation is a non-contact optical measurement method that uses random speckle patterns applied to the object's surface and accurately matches corresponding points in the speckle pattern before and after deformation to measure the object's displacement, strain, and other data. In a camera and projector system, raster projection can reconstruct the object's three-dimensional shape before and after deformation, but it cannot determine the positions of corresponding points before and after deformation. Digital image correlation, on the other hand, can precisely match the positions of corresponding points on the object's surface using the speckle pattern. Combining the two techniques allows for the measurement of an object's three-dimensional shape and deformation. However, since digital image correlation requires applying speckle patterns to the object's surface, this significantly affects the quality of the grating patterns captured by the camera, and thus the accuracy and precision of the phase calculation. Currently, no accurate and fast method exists to address this problem, both domestically and internationally. Summary of the Invention

[0003] In order to solve the above problems, the present invention provides a high-precision three-dimensional shape and deformation measurement method based on projection imaging.

[0004] The technical solution to achieve the purpose of the present invention is: a high-precision three-dimensional shape and deformation measurement method based on projection imaging. The experimental device includes an industrial camera, a lens, an optical platform, an electronic computer, a projector, and an object to be measured. The measurement method includes the following steps:

[0005] Step 1. Fix the experimental device: Fix the industrial camera and projector on the optical platform, fix the object to be measured on the optical platform, adjust the direction of the camera lens to point to the object to be measured, and make the object to be measured in the center position in the camera's field of view, align the projector lens with the maximum plane direction of the object to be measured, and focus the camera and projector on the object to be measured;

[0006] Step 2: System calibration: perform camera-projector system calibration to obtain the internal and external parameters of the camera and projector;

[0007] Step 3: Acquisition of training data: Using a projector, a sinusoidal grating and a sinusoidal grating with speckle are projected onto the object to be measured. The object to be measured is rotated or moved to collect data from different scenes, which are then normalized to construct a training data set for the neural network.

[0008] Step 4: Build a convolutional neural network and train it: Build a convolutional neural network with N image inputs and N image outputs, use the sinusoidal speckle line image as the neural network input, and the sinusoidal line image as the neural network output, and train the convolutional neural network model;

[0009] Step 5. Acquisition of experimental data: Spray a speckle pattern onto the object to be measured and randomly place it in the field of view of the projector and camera. Focus the camera and projector on the object to be measured. Use the projector to project sinusoidal grating lines and white light, respectively. Then, apply deformation to the object to be measured, and use the camera to collect the grating line image and speckle image of the object to be measured before and after deformation.

[0010] Step 6: normalize the grating line images with sinusoidal speckles before and after deformation in step 5 and input them into the trained convolutional neural network to eliminate the speckles in the grating line images and improve the quality of the grating line images;

[0011] Step 7: Calculate the pixel displacement of the speckle using the speckle images before and after deformation in step 5, calculate the phase of the object to be measured using the grid line image predicted in step 6, and then calculate the three-dimensional morphology and deformation of the object to be measured in the world coordinate system by combining the internal and external parameters of the camera and projector in step 3.

[0012] Furthermore, in step 2, the camera-projector system is calibrated to obtain the intrinsic and extrinsic parameters of the camera and projector. The specific method is to treat the projector as another camera to "capture" the image, forming a dual-camera system calibration system. The specific steps are as follows:

[0013] (1) Prepare a red / blue checkerboard calibration plate and horizontal and vertical grid lines;

[0014] (2) Place the red / blue checkerboard in the camera's field of view so that it occupies half of the camera's field of view;

[0015] (3) The projector projects red light or blue light. In the captured image, the red / blue checkerboard appears as a black / white checkerboard. The camera aperture is adjusted so that the grayscale value in the checkerboard area is between 150 and 200, and the checkerboard image is captured.

[0016] (4) The projector projects horizontal and vertical grid line stripe patterns using white light, and the camera captures the grid line images;

[0017] (5) Rotate the chessboard position and repeat the operations in (3) and (4). In order to suppress the influence of noise, the chessboard position needs to be changed by at least 10 positions;

[0018] (6) Find the coordinates of corner points on the red or blue light checkerboard for camera calibration and subsequent projector calibration;

[0019] (7) Calculate the absolute phases of the horizontal and vertical grating lines prepared in (1) and the absolute phases of the horizontal and vertical grating lines collected in (4), and use them as the standard phase and actual phase in the two directions of the grating lines, respectively;

[0020] (8) According to the coordinates of the corner point and the actual phase of the grid line, the standard phase in the two directions of the corner point is determined, and the horizontal standard phase is used as the row coordinate and the vertical standard phase is used as the column coordinate to form the row and column coordinates of the corner point in the projector image;

[0021] (9) Based on the corner point coordinates in (6) and (8), the internal and external parameters of the camera and projector are calibrated using the single camera calibration principle;

[0022] sI=A[R,t]X W (1) Among them, I=[u,v,1] T is the pixel coordinate of the corner point, X W =[x w ,y w ,z w ,1] T is the world coordinate of the corner point, s is the scale factor, A is the intrinsic parameter of the camera or projector, R is the rotation matrix between the world coordinate and the camera or projector coordinate, t is the translation vector between the world coordinate and the camera or projector coordinate, [R, t] constitutes the extrinsic parameters of the camera or projector.

[0023] Furthermore, in step 3, a projector is used to project sinusoidal grating lines and sinusoidal grating lines with speckles onto the object to be measured, and the position of the object to be measured is rotated or moved to collect data of different scenes, and normalized to construct a training data set for the neural network. The specific method is as follows:

[0024] Step 3.1: The projector projects sinusoidal grating lines. The basic pattern is as follows:

[0025] In a grayscale image, the maximum grayscale value of the image is 255 and the minimum grayscale value is 0. The closer the grayscale value is to 255, the brighter the image is, and the closer the grayscale value is to 0, the darker the image is. Therefore, the sinusoidal grating lines are generated according to the following formula:

[0026]

[0027] Among them, I n , n=1,2,3,…,N represents the generated grating pattern, ω represents the frequency of the grating, x represents the width of the grating image, and N represents the number of phase shift steps;

[0028] In step 3.2, the projector projects sinusoidal speckle grating lines, the basic pattern of which is as follows:

[0029] The sinusoidal grating line speckle image only requires adding the digital speckle field to the sinusoidal grating line image in step 3.1. The digital speckle field is designed and produced by controlling the number of spots, the coordinates of the circle center, and the radius of the circle. The digital speckle field is generated by the following four formulas:

[0030]

[0031]

[0032]

[0033] n=ρA / (0.25·πd 2 ) (6)

[0034] Among them, (X1, Y1) is the coordinate of the center of the first scattered spot, (X i , Y i ) and (X i ', Y i ') are the center coordinates of the speckle spots in the regular and random distribution speckle fields, ɑ is the center distance between two speckle spots in the regular distribution speckle field, ρ is the duty cycle, d is the speckle diameter, f(r) represents a random function in the interval (-r, r), r is the randomness factor in the range (0, 1], and n is the number of speckles, which is related to the camera resolution A;

[0035] Step 3.3: Rotate or move the object to be measured to collect data from different scenes, perform normalization processing, and construct a training data set for the neural network. The basic model is as follows:

[0036] Since the grayscale value of the collected image is between 0 and 255, the normalization process is performed by dividing the grayscale value by 255 to obtain the training data set.

[0037] Furthermore, in step 4, a convolutional neural network with N image inputs and N image outputs is built. The sinusoidal speckle line image is used as the neural network input and the sinusoidal line image is used as the neural network output. The convolutional neural network model is trained. The convolutional neural network model includes two parts: feature extraction and feature fusion. The adaptive moment estimation ADAM is selected as the optimizer for training. The specific mode is as follows:

[0038] (a) Feature extraction

[0039] First, the image data input passes through the first convolution layer and the first batch normalization layer to obtain the shallow feature X1. The shallow feature X1 passes through the second convolution layer, the second batch normalization layer and the first dropout layer in sequence to obtain the further feature X2. The further feature X2 passes through the third convolution layer, the third batch normalization layer and the second dropout layer in sequence to obtain the further feature X3. Further, the feature X3 passes through the fourth convolution layer, the fourth batch normalization layer and the third dropout layer in sequence to obtain the deep feature X4.

[0040] (b) Feature fusion

[0041] Feature X4 passes through the first deconvolution, the fifth batch normalization layer, and the fourth dropout layer, and is fused with feature X3 through the first adder to obtain feature X5. Feature X5 passes through the second deconvolution, the sixth batch normalization layer, and the fifth dropout layer, and is fused with feature X2 through the second adder to obtain feature X6. Feature X6 passes through the third deconvolution, the seventh batch normalization layer, and the sixth dropout layer, and is fused with feature X1 through the third adder to obtain feature X7. Feature X7 passes through the fourth deconvolution to obtain feature X8. Feature X8 passes through a set of residual structures to obtain feature X9. Finally, feature X9 passes through the fifth convolution layer and is fused with feature X8 through the fourth adder to obtain the final fused feature X. 10 , which is the image data to be output.

[0042] Furthermore, in step 7, the pixel displacement of the speckle is calculated using the speckle images before and after deformation in step 5, and the phase of the object to be measured is calculated using the grid line image predicted in step 6. Then, the three-dimensional morphology and deformation of the object to be measured in the world coordinate system are calculated by combining the internal and external parameters of the camera and projector in step 3. The specific method is as follows:

[0043] Step 7.1: Calculate the pixel displacement of the speckle using the speckle images before and after deformation in step 5. The basic pattern is as follows:

[0044] The camera collects the speckle images before and after deformation, calculates the correlation coefficient between the search window and the reference sub-area, and selects the search window with the maximum correlation coefficient as the target sub-area. At this time, the center point of the target sub-area is regarded as the corresponding point of the reference sub-area after deformation. The pixel displacement is obtained by subtracting the coordinates of the center point of the reference sub-area from that of the target sub-area. The correlation coefficient C cc It is expressed as follows:

[0045]

[0046] Where, f(x i ,y i ) is the coordinate of the reference sub-area (x i ,y i ) point gray value, g(x i ′,y i ′) is the target sub-area with coordinates (x i ′,y i ′) (the coordinates are all local coordinates centered at the midpoint of the sub-region), is the average gray value of the reference sub-area, is the average grayscale value of the sub-region of the target image with the same size as the reference sub-region;

[0047] Step 7.2: Calculate the phase of the object to be measured using the grid line image predicted in step 6. The basic pattern is as follows:

[0048] Ideally, the image I captured by the camera n , expressed as:

[0049]

[0050] Where a(u,v) is the background light intensity, b(u,v) is the reflectivity of the object surface, is the phase to be determined, (u, v) is the position of the pixel, θ n , n=1,2,3,…,N is the phase shift, expressed as:

[0051]

[0052] The least squares phase solution formula is expressed as:

[0053]

[0054] Solving the above formula we get The value is in the range of [-π,π]. Since the period of the trigonometric function is 2π, the complete phase value Φ(u,v) is expressed as:

[0055]

[0056] Where, k(u,v) is the number of compensation cycles;

[0057] In step 7.3, the speckle image is used to match the pixel displacement of the object before and after deformation, and the positions of the corresponding points of the object before and after deformation are determined. The three-dimensional shape of the object before and after deformation is calculated by combining the internal and external parameters of the camera and projector calibration. The three-dimensional deformation (including three-dimensional displacement and full-field strain) needs to be obtained based on the positional relationship of the corresponding points before and after deformation of the object, as shown below:

[0058] Assume that the three-dimensional coordinates (X, Y, Z) of the corresponding points before and after deformation of the object to be measured and (X a , Y a , Z a ), the three-dimensional displacement is:

[0059]

[0060] Where U, V and W are the displacements in three directions respectively;

[0061] Then establish the local coordinate system O e , transform the three-dimensional coordinates and three-dimensional displacement of the grid points in the world coordinate system before deformation into the coordinate system O e In the equation (X e ,Y e ,Z e ) and (U e , V e , W e ), the displacement field function is obtained by using the quadratic surface fitting method, which is expressed as follows:

[0062]

[0063] in, and They are the displacement field function U e , V e , W e The coefficient of , the full-field strain is expressed as follows:

[0064]

[0065] Where, ε xx , ε yy , ε zz , ε yz , ε zy , ε xy , ε yx , ε zx , ε xz represents the strain tensor.

[0066] Compared with the existing technology, the present invention has the following significant advantages: 1) It innovatively applies deep learning models to high-precision object morphology measurement, solves the influence of speckle patterns caused by the combination of grid line projection and digital image correlation on the quality of grid line stripes, effectively eliminates speckle in the grid line stripes, and improves the quality of the grid line stripes and the accuracy of the solution. 2) The convolutional neural network constructed by the present invention improves the accuracy of specific application scenarios, can achieve the purpose of rapid measurement with very small data sets, and has achieved good results in experiments. 3) The present invention has high universality for different specific measurement processes. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] Figure 1 Schematic diagram of the experimental device of the present invention.

[0068] Figure 2 This is a neural network structure diagram of the method of the present invention.

[0069] Figure 3 It is the neural network prediction result of the method of the present invention.

[0070] Figure 4 The phase comparison between the grid line predicted by the method of the present invention and the original grid line solution is shown.

[0071] 1: Electronic computer;

[0072] 2: Projector

[0073] 3: Optical platform;

[0074] 4: White plane board and its fixing device;

[0075] 5: Industrial camera;

[0076] 6: High-resolution lens; DETAILED DESCRIPTION

[0077] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0078] The present invention, based on algorithm design and experimental testing, conveniently and quickly solves the impact of speckle on the quality of grid lines and fringes. The method has the advantages of high calculation accuracy, simple equipment, convenience and practicality, low algorithm complexity, and fast calculation speed.

[0079] like Figure 1As shown, a high-precision three-dimensional shape and deformation measurement method based on projection imaging is provided. The experimental device of the method includes the following equipment: an electronic computer (1), a projector (2), an optical platform (3), a plane plate to be measured and its fixing device (4), an industrial camera (5), and a high-resolution lens (6). The industrial camera used in the test experiment has a pixel of 4 million pixels and a lens focal length of 35 mm. The method includes the following steps:

[0080] Step 1. Fix the experimental device: Fix the industrial camera and projector on the optical platform, fix the flat plate on the optical platform, adjust the direction of the camera lens to point to the plane of the flat plate, and make the flat plate centered in the camera's field of view, align the projector lens with the maximum plane direction of the flat plate, and focus the camera and projector on the flat plate;

[0081] Step 2: System calibration: Perform camera-projector system calibration to obtain the intrinsic and extrinsic parameters of the camera and projector. The specific method is to treat the projector as another camera to "capture" the image, forming a dual-camera system calibration system. The specific steps are as follows:

[0082] (1) Prepare a red / blue checkerboard calibration plate and horizontal and vertical grid lines;

[0083] (2) Place the red / blue checkerboard in the camera's field of view so that it occupies half of the camera's field of view;

[0084] (3) The projector projects red light or blue light. In the captured image, the red / blue checkerboard appears as a black / white checkerboard. The camera aperture is adjusted so that the grayscale value in the checkerboard area is between 150 and 200, and the checkerboard image is captured.

[0085] (4) The projector projects horizontal and vertical grid line stripe patterns using white light, and the camera captures the grid line images;

[0086] (5) Rotate the chessboard position and repeat the operations in (3) and (4). In order to suppress the influence of noise, the chessboard position needs to be changed by at least 10 positions;

[0087] (6) Find the coordinates of corner points on the red or blue light checkerboard for camera calibration and subsequent projector calibration;

[0088] (7) Calculate the absolute phases of the horizontal and vertical grating lines prepared in (1) and the absolute phases of the horizontal and vertical grating lines collected in (4), and use them as the standard phase and actual phase in the two directions of the grating lines, respectively;

[0089] (8) According to the coordinates of the corner point and the actual phase of the grid line, the standard phase in the two directions of the corner point is determined, and the horizontal standard phase is used as the row coordinate and the vertical standard phase is used as the column coordinate to form the row and column coordinates of the corner point in the projector image;

[0090] (9) Based on the corner point coordinates in (6) and (8), the internal and external parameters of the camera and projector are calibrated using the single camera calibration principle;

[0091] sI=A[R,t]X W (1) Among them, I=[u,v,1] T is the pixel coordinate of the corner point, X W =[x w ,y w ,z w ,1] T is the world coordinate of the corner point, s is the scale factor, A is the intrinsic parameter of the camera or projector, R is the rotation matrix between the world coordinate and the camera or projector coordinate, t is the translation vector between the world coordinate and the camera or projector coordinate, [R, t] constitutes the extrinsic parameters of the camera or projector.

[0092] Step 3: Obtaining training data: Use a projector to project sinusoidal grating lines and sinusoidal grating lines with speckle onto a flat plate. Rotate or move the flat plate to collect data from different scenes, perform normalization, and construct a training data set for the neural network. The specific method is as follows:

[0093] Step 3.1: The projector projects sinusoidal grating lines. The basic pattern is as follows:

[0094] In a grayscale image, the maximum grayscale value of the image is 255 and the minimum grayscale value is 0. The closer the grayscale value is to 255, the brighter the image is, and the closer the grayscale value is to 0, the darker the image is. Therefore, the sinusoidal grating lines are generated according to the following formula:

[0095]

[0096] Among them, I n , n=1,2,3,…,N represents the generated grating pattern, ω represents the frequency of the grating, x represents the width of the grating image, and N represents the number of phase shift steps;

[0097] In step 3.2, the projector projects sinusoidal speckle grating lines, the basic pattern of which is as follows:

[0098] The sinusoidal grating line speckle image only requires adding the digital speckle field to the sinusoidal grating line image in step 3.1. The digital speckle field is designed and produced by controlling the number of spots, the coordinates of the circle center, and the radius of the circle. The digital speckle field is generated by the following four formulas:

[0099]

[0100]

[0101]

[0102] n=ρA / (0.25·πd 2 ) (6)

[0103] Among them, (X1, Y1) is the coordinate of the center of the first scattered spot, (X i , Y i ) and (X i ', Y i ') are the center coordinates of the speckle spots in the regular and random distribution speckle fields, ɑ is the center distance between two speckle spots in the regular distribution speckle field, ρ is the duty cycle, d is the speckle diameter, f(r) represents a random function in the interval (-r, r), r is the randomness factor in the range (0, 1], and n is the number of speckles, which is related to the camera resolution A;

[0104] Step 3.3: Rotate or move the plane to collect data from different scenes, perform normalization, and construct a training data set for the neural network. The basic model is as follows:

[0105] Since the grayscale value of the collected image is between 0 and 255, the normalization process is performed by dividing the grayscale value by 255 to obtain the training data set.

[0106] Step 4. Build a convolutional neural network and train it: Build a convolutional neural network with N image inputs and N image outputs. Use the sinusoidal speckle line image as the neural network input and the sinusoidal line image as the neural network output. Train the convolutional neural network model, which includes feature extraction and feature fusion. Select the adaptive moment estimation ADAM as the optimizer for training. The specific mode is as follows:

[0107] (a) Feature extraction

[0108] First, the image data input passes through the first convolution layer and the first batch normalization layer to obtain the shallow feature X1. The shallow feature X1 passes through the second convolution layer, the second batch normalization layer and the first dropout layer in sequence to obtain the further feature X2. The further feature X2 passes through the third convolution layer, the third batch normalization layer and the second dropout layer in sequence to obtain the further feature X3. Further, the feature X3 passes through the fourth convolution layer, the fourth batch normalization layer and the third dropout layer in sequence to obtain the deep feature X4.

[0109] (b) Feature fusion

[0110] Feature X4 passes through the first deconvolution, the fifth batch normalization layer, and the fourth dropout layer, and is fused with feature X3 through the first adder to obtain feature X5. Feature X5 passes through the second deconvolution, the sixth batch normalization layer, and the fifth dropout layer, and is fused with feature X2 through the second adder to obtain feature X6. Feature X6 passes through the third deconvolution, the seventh batch normalization layer, and the sixth dropout layer, and is fused with feature X1 through the third adder to obtain feature X7. Feature X7 passes through the fourth deconvolution to obtain feature X8. Feature X8 passes through a set of residual structures to obtain feature X9. Finally, feature X9 passes through the fifth convolution layer and is fused with feature X8 through the fourth adder to obtain the final fused feature X. 10 , which is the image data to be output.

[0111] Step 5: Acquisition of experimental data: Spray a speckle pattern onto a flat plate and randomly place it in the field of view of a projector and camera. Focus the camera and projector on the flat plate. Use the projector to project sinusoidal grating lines and white light, respectively. Then, deform the flat plate and use the camera to capture the grating line image and speckle image of the flat plate before and after deformation.

[0112] Step 6: normalize the grating line images with sinusoidal speckles before and after deformation in step 5 and input them into the trained convolutional neural network to eliminate the speckles in the grating line images and improve the quality of the grating line images;

[0113] Step 7: Calculate the pixel displacement of the speckle using the speckle images before and after deformation in step 5, calculate the phase of the plane plate using the grid line image predicted in step 6, and then calculate the three-dimensional shape and deformation of the plane plate in the world coordinate system by combining the internal and external parameters of the camera and projector in step 3;

[0114] Step 7.1: Calculate the pixel displacement of the speckle using the speckle images before and after deformation in step 5. The basic pattern is as follows:

[0115] The camera collects speckle images before and after deformation, and uses the correlation function for correlation matching. By calculating the correlation coefficient between the search window and the reference sub-area, the search window with the maximum correlation coefficient is selected as the target sub-area. At this time, the center point of the target sub-area can be regarded as the corresponding point after the deformation of the measured point. The pixel displacement can be obtained by subtracting the coordinates of the center point of the reference sub-area and the target sub-area, where the correlation coefficient C cc It is expressed as follows:

[0116]

[0117] Where, f(x i ,y i ) is the coordinate of the reference sub-area (x i ,y i ) point gray value, g(x i ′,y i ′) is the target sub-area with coordinates (x i ′,y i ′) (the coordinates are all local coordinates centered at the midpoint of the sub-region), is the average gray value of the reference sub-area, is the average grayscale value of the sub-region of the target image with the same size as the reference sub-region;

[0118] Step 7.2, use the grid line image predicted in step 6 to calculate the phase of the plane panel. The basic pattern is as follows:

[0119] Ideally, the image I captured by the camera n , expressed as:

[0120]

[0121] Where a(u,v) is the background light intensity, b(u,v) is the reflectivity of the object surface, is the phase to be determined, (u, v) is the position of the pixel, θ n , n=1,2,3,…,N is the phase shift, expressed as:

[0122]

[0123] The least squares phase solution formula is expressed as:

[0124]

[0125] Solving the above formula we get The value is in the range of [-π,π]. Since the period of the trigonometric function is 2π, the complete phase value Φ(u,v) is expressed as:

[0126]

[0127] Where, k(u,v) is the number of compensation cycles;

[0128] In step 7.3, the three-dimensional shape and deformation of the plane plate in the world coordinate system are calculated by combining the internal and external parameters of the camera and projector in step 3. The basic model is as follows:

[0129] First, the speckle image is used to match the pixel displacement of the object before and after deformation, and the corresponding point positions before and after the deformation are found. Then, the three-dimensional shape of the object before and after deformation can be obtained by combining the internal and external parameters of the camera and projector calibration. The three-dimensional deformation needs to be obtained based on the positional relationship of the corresponding points before and after the deformation of the object, as shown below:

[0130] Assume that the three-dimensional coordinates (X, Y, Z) of the corresponding points before and after the object is deformed and (X a , Y a , Z a ), the three-dimensional displacement is:

[0131]

[0132] Where U, V and W are the displacements in three directions respectively;

[0133] Then establish the local coordinate system O e , transform the three-dimensional coordinates and three-dimensional displacement of the grid points in the world coordinate system before deformation into the coordinate system O e In the equation (X e ,Y e ,Z e ) and (U e , V e , W e ), the displacement field function is obtained by using the quadratic surface fitting method, which is expressed as follows:

[0134]

[0135] in, and are the coefficients of the field functions of each displacement field, and the full field strain is expressed as follows:

[0136]

[0137] Where, ε xx , ε yy , ε zz , ε yz , ε zy , εxy , ε yx , ε zx , ε xz represents the strain tensor.

[0138] Example

[0139] In order to verify the effectiveness of the solution of the present invention, a plane plate is used as a measurement object to collect data and the following simulation experiment is carried out.

[0140] 1) Data collection and preprocessing

[0141] Take the three-step phase shift as an example: first, use a projector to project sinusoidal grating lines onto a flat plate, and use a camera to collect three images as the output data of the neural network; then project sinusoidal grating lines with speckles onto the flat plate, and use a camera to collect three images as the input data of the neural network; then, by moving and rotating the flat plate, a total of 60 groups of data from different scenes are collected, and the 60 groups of data are normalized to obtain training data.

[0142] 2) Build and train a convolutional neural network model

[0143] Build a convolutional neural network with 3 image data input and 3 image data output; select adaptive moment estimation ADAM as the optimizer training. The specific structure is as follows Figure 2 As shown:

[0144] 3) Prediction of actual data

[0145] In the actual experiment, the surface of the plane is sprayed with speckles, a sinusoidal grating is projected with a projector, and captured with a camera. The 1024X1024 area of ​​the captured image is input into the trained neural network to obtain a grating image with speckles removed, thereby improving the grating quality. The prediction results are as follows Figure 3 As shown, the neural network constructed by the present invention can effectively eliminate speckles in the grid line image and improve the quality of the grid line; Figure 4 As shown, the green solid line represents the original phase result containing speckles, and the red dotted line represents the phase solved by the neural network output grating line. It can be clearly seen that the patent of the present invention can effectively improve the phase accuracy of the grating line containing speckles.

Claims

1. A high-precision three-dimensional shape and deformation measurement method based on projection imaging, characterized in that: The experimental device includes an industrial camera, a lens, an optical platform, an electronic computer, a projector, and an object to be measured. The measurement method includes the following steps: Step 1. Fix the experimental device: Fix the industrial camera and projector on the optical platform, fix the object to be measured on the optical platform, adjust the direction of the camera lens to point to the object to be measured, and make the object to be measured in the center position in the camera's field of view, align the projector lens with the maximum plane direction of the object to be measured, and focus the camera and projector on the object to be measured; Step 2: System calibration: perform camera-projector system calibration to obtain the internal and external parameters of the camera and projector; Step 3: Acquisition of training data: Using a projector, a sinusoidal grating and a sinusoidal grating with speckle are projected onto the object to be measured. The object to be measured is rotated or moved to collect data from different scenes, which are then normalized to construct a training data set for the neural network. Step 4: Build a convolutional neural network and train it: Build a convolutional neural network with N image inputs and N image outputs, use the sinusoidal speckle line image as the neural network input, and the sinusoidal line image as the neural network output, and train the convolutional neural network model; Step 5. Acquisition of experimental data: Spray a speckle pattern onto the object to be measured and randomly place it in the field of view of the projector and camera. Focus the camera and projector on the object to be measured. Use the projector to project sinusoidal grating lines and white light, respectively. Then, apply deformation to the object to be measured, and use the camera to collect the grating line image and speckle image of the object to be measured before and after deformation. Step 6: normalize the grating line images with sinusoidal speckles before and after deformation in step 5 and input them into the trained convolutional neural network to eliminate the speckles in the grating line images and improve the quality of the grating line images; Step 7: Calculate the pixel displacement of the speckle using the speckle images before and after deformation in step 5, calculate the phase of the object to be measured using the grid line image predicted in step 6, and then calculate the three-dimensional morphology and deformation of the object to be measured in the world coordinate system by combining the internal and external parameters of the camera and projector in step 3.

2. The high-precision three-dimensional shape and deformation measurement method based on projection imaging according to claim 1 is characterized in that: In step 2, the camera-projector system is calibrated to obtain the intrinsic and extrinsic parameters of the camera and projector. The specific method is to treat the projector as another camera to "capture" the image, forming a dual-camera system calibration system. The specific steps are as follows: (1) Prepare a red / blue checkerboard calibration plate and horizontal and vertical grid lines; (2) Place the red / blue checkerboard in the camera's field of view so that it occupies half of the camera's field of view; (3) The projector projects red light or blue light. In the captured image, the red / blue checkerboard appears as a black / white checkerboard. The camera aperture is adjusted so that the grayscale value in the checkerboard area is between 150 and 200, and the checkerboard image is captured. (4) The projector projects horizontal and vertical grid line stripe patterns using white light, and the camera captures the grid line images; (5) Rotate the chessboard position and repeat the operations in (3) and (4). In order to suppress the influence of noise, the chessboard position needs to be changed by at least 10 positions; (6) Find the coordinates of corner points on the red or blue light checkerboard for camera calibration and subsequent projector calibration; (7) Calculate the absolute phases of the horizontal and vertical grating lines prepared in (1) and the absolute phases of the horizontal and vertical grating lines collected in (4), and use them as the standard phase and actual phase in the two directions of the grating lines, respectively; (8) According to the coordinates of the corner point and the actual phase of the grid line, the standard phase in the two directions of the corner point is determined, and the horizontal standard phase is used as the row coordinate and the vertical standard phase is used as the column coordinate to form the row and column coordinates of the corner point in the projector image; (9) Based on the corner point coordinates in (6) and (8), the internal and external parameters of the camera and projector are calibrated using the single camera calibration principle; sI=A[R,t]X W (1) Among them, I=[u,v,1] T is the pixel coordinate of the corner point, X W =[x w ,y w ,z w ,1] T is the world coordinate of the corner point, s is the scale factor, A is the intrinsic parameter of the camera or projector, R is the rotation matrix between the world coordinate and the camera or projector coordinate, t is the translation vector between the world coordinate and the camera or projector coordinate, [R, t] constitutes the extrinsic parameters of the camera or projector.

3. The high-precision three-dimensional shape and deformation measurement method based on projection imaging according to claim 1, characterized in that: Step 3: Use a projector to project sinusoidal grating lines and sinusoidal grating lines with speckles onto the object to be measured, rotate or move the object to be measured to collect data from different scenes, perform normalization processing, and construct a training data set for the neural network. The specific method is as follows: Step 3.1: The projector projects sinusoidal grating lines. The basic pattern is as follows: In a grayscale image, the maximum grayscale value of the image is 255 and the minimum grayscale value is 0. The closer the grayscale value is to 255, the brighter the image is, and the closer the grayscale value is to 0, the darker the image is. Therefore, the sinusoidal grating lines are generated according to the following formula: Among them, I n， n=1,2,3,…,N represents the generated grating pattern, ω represents the frequency of the grating, x represents the width of the grating image, and N represents the number of phase shift steps; In step 3.2, the projector projects sinusoidal speckle grating lines, the basic pattern of which is as follows: The sinusoidal grating line speckle image only requires adding the digital speckle field to the sinusoidal grating line image in step 3.

1. The digital speckle field is designed and produced by controlling the number of spots, the coordinates of the circle center, and the radius of the circle. The digital speckle field is generated by the following four formulas: n=ρA / (0.25·πd 2 ) (6) Among them, (x i ,y i ) is the coordinate point of the reference sub-area, (X1, Y1) is the coordinate of the center of the first scattered spot, (X i , Y i ) and (X i ', Y i ') are the center coordinates of the speckle spots in the regular and random distribution speckle fields, ɑ is the center distance between two speckle spots in the regular distribution speckle field, ρ is the duty cycle, d is the speckle diameter, f(r) represents a random function in the interval (-r, r), r is the randomness factor in the range (0, 1], and n is the number of speckles, which is related to the camera resolution A; Step 3.3: Rotate or move the object to be measured to collect data from different scenes, perform normalization processing, and construct a training data set for the neural network. The basic model is as follows: Since the grayscale value of the collected image is between 0 and 255, the normalization process is performed by dividing the grayscale value by 255 to obtain the training data set.

4. The high-precision three-dimensional shape and deformation measurement method based on projection imaging according to claim 1, characterized in that: In step 4, a convolutional neural network with N image inputs and N image outputs is built. The sinusoidal speckle line image is used as the neural network input and the sinusoidal line image is used as the neural network output. The convolutional neural network model is trained. The convolutional neural network model includes two parts: feature extraction and feature fusion. The adaptive moment estimation ADAM is selected as the optimizer for training. The specific mode is as follows: (a) Feature extraction First, the image data input passes through the first convolution layer and the first batch normalization layer to obtain the shallow feature X1. The shallow feature X1 passes through the second convolution layer, the second batch normalization layer and the first dropout layer in sequence to obtain the further feature X2. The further feature X2 passes through the third convolution layer, the third batch normalization layer and the second dropout layer in sequence to obtain the further feature X3. Further, the feature X3 passes through the fourth convolution layer, the fourth batch normalization layer and the third dropout layer in sequence to obtain the deep feature X4. (b) Feature fusion Feature X4 passes through the first deconvolution, the fifth batch normalization layer and the fourth dropout layer in sequence, and is fused with feature X3 through the first adder to obtain feature X5. Feature X5 passes through the second deconvolution, the sixth batch normalization layer and the fifth dropout layer in sequence, and is fused with feature X2 through the second adder to obtain feature X6. Feature X6 passes through the third deconvolution, the seventh batch normalization layer and the sixth dropout layer in sequence, and is fused with feature X1 through the third adder to obtain feature X7. Feature X7 passes through the fourth deconvolution to obtain feature X8. Further, feature X8 passes through a set of residual structures to obtain feature X9. Finally, feature X9 passes through the fifth convolution layer and is fused with feature X8 through the fourth adder to obtain the final fused feature X 10 , which is the image data to be output.

5. The high-precision three-dimensional shape and deformation measurement method based on projection imaging according to claim 1, characterized in that: In step 7, the pixel displacement of the speckle is calculated using the speckle images before and after deformation in step 5, and the phase of the object to be measured is calculated using the grid line image predicted in step 6. Then, the three-dimensional shape and deformation of the object to be measured in the world coordinate system are calculated by combining the internal and external parameters of the camera and projector in step 3. The specific method is as follows: Step 7.1: Calculate the pixel displacement of the speckle using the speckle images before and after deformation in step 5. The basic pattern is as follows: The camera collects the speckle images before and after deformation, calculates the correlation coefficient between the search window and the reference sub-area, and selects the search window with the maximum correlation coefficient as the target sub-area. At this time, the center point of the target sub-area is regarded as the corresponding point of the reference sub-area after deformation. The pixel displacement is obtained by subtracting the coordinates of the center point of the reference sub-area from that of the target sub-area. The correlation coefficient C cc It is expressed as follows: Where, f(x i ,y i ) is the coordinate of the reference sub-area (x i ,y i ) point gray value, g(x i ′,y i ′) is the target sub-area with coordinates (x i ′,y i ′) point gray value, is the average gray value of the reference sub-area, is the average grayscale value of the sub-region of the target image with the same size as the reference sub-region; Step 7.2: Calculate the phase of the object to be measured using the grid line image predicted in step 6. The basic pattern is as follows: Ideally, the image I captured by the camera n , expressed as: Where a(u,v) is the background light intensity, b(u,v) is the reflectivity of the object surface, is the phase to be determined, (u, v) is the position of the pixel, θ n， n=1,2,3,…,N is the phase shift, expressed as: The least squares phase solution formula is expressed as: Solving the above formula we get The value is in the range of [-π,π]. Since the period of the trigonometric function is 2π, the complete phase value Φ(u,v) is expressed as: Where, k(u,v) is the number of compensation cycles; In step 7.3, the speckle image is used to match the pixel displacement of the object before and after deformation, and the positions of the corresponding points of the object before and after deformation are determined. The three-dimensional shape of the object before and after deformation is calculated by combining the internal and external parameters of the camera and projector calibration. The three-dimensional deformation needs to be obtained based on the positional relationship of the corresponding points before and after deformation, as shown below: Assume that the three-dimensional coordinates (X, Y, Z) of the corresponding points before and after deformation of the object to be measured and (X a , Y a , Z a ), the three-dimensional displacement is: Where U, V and W are the displacements in three directions respectively; Then establish the local coordinate system O e , transform the three-dimensional coordinates and three-dimensional displacement of the grid points in the world coordinate system before deformation into the coordinate system O e In the equation (X e ,Y e ,Z e ) and (U e , V e , W e ), the displacement field function is obtained by using the quadratic surface fitting method, which is expressed as follows: in, and They are the displacement field function U e , V e , W e The coefficient of , the full-field strain is expressed as follows: In the formula, ε xx , ε yy , ε zz , ε yz , ε zy , ε[[ID=I1]] xy , ε yx , ε zx , ε xz represent the strain tensor.

Images