A Stereo Vision Deformation Measurement Method for Suppressing Heat Flow Disturbance

Abstract

The present invention provides a stereovision deformation measurement method for suppressing heat flow disturbance, which combines digital image correlation with machine learning to measure the deformation of an object by using a multi-camera system, calibrates the multi-camera system, and solves the internal and external parameters of each camera; uses a neural network to improve the quality of speckle images affected by heat flow disturbance; combines the positional relationship of corresponding points in the images before and after deformation to solve the three-dimensional displacement of the object, and calculates the three-dimensional strain from the three-dimensional displacement. The present invention can suppress the influence of heat flow disturbance on stereovision deformation measurement and has good applicability for measurements with different numbers of cameras.

Description

Technical Field

[0001] The present invention relates to the field of solid mechanics in optical measurement experiments and image measurement technologies, and particularly relates to a stereo vision deformation measurement method for suppressing heat flow disturbance. Background Art

[0002] Digital image correlation technology is a non-contact optical measurement method that uses random speckles sprayed on the surface of an object, collects speckle images before and after the object deforms, accurately matches corresponding points, and measures the deformation of the object. During the imaging process, temperature changes will cause the inhomogeneity of the propagation medium (air), thereby causing light deflection and imaging drift, affecting the test accuracy. And it is very difficult to quantitatively analyze and establish a mathematical description model for the influence of air disturbance caused by heat flow on imaging error. By using the neural network's ability to perform arbitrarily complex pattern classification and excellent multi-dimensional function mapping ability, adding heat flow disturbance to the neural network training set can enable the neural network to separate the imaging error caused by heat flow disturbance, thereby achieving the effect of suppressing heat flow disturbance. Summary of the Invention

[0003] In order to achieve the above technology, the object of the present invention is to provide a stereo vision deformation measurement method for suppressing heat flow disturbance.

[0004] The technical solution for achieving the object of the present invention is: a stereo vision deformation measurement method for suppressing heat flow disturbance, the experimental device includes industrial cameras, lenses, an optical platform, a camera fixing device, an electronic computer, and an object to be measured, and the measurement method includes the following steps:

[0005] Step 1, fixing of the experimental device: Four industrial cameras are orthogonally arranged and fixed on the optical platform, the object to be measured is fixed to the optical platform, the directions of the four camera lenses are adjusted to point to the object to be measured, and the object to be measured is centered in the camera view, and the camera aperture and focal length are adjusted to be appropriate;

[0006] Step 2, calibration of multi-camera system parameters: Calibrate the four cameras in pairs to determine the internal and external parameters of each camera;

[0007] Step 3, obtaining of training data: A speckle pattern is sprayed on the object to be measured and randomly placed in the field of view of the multi-camera system, and different operations are performed for different neural networks:

[0008] (1) For the convolutional neural network: When there is heat flow disturbance, move the object to be measured to different positions, and each camera collects N speckle images; When there is no heat flow disturbance, move the object to be measured to the same acquisition position as when there is heat flow disturbance, and each camera collects N speckle images;

[0009] (2) For the BP neural network: When there is heat flux perturbation, move the object to be measured to different positions, and each camera captures N speckle images. Solve to obtain 8N image coordinates (1 speckle image captured by 1 camera has 2-direction image coordinates, so 8N image coordinates are obtained from N speckle images captured by 4 cameras); When there is no heat flux perturbation, move the object to be measured to the same acquisition positions as when there is heat flux perturbation, and each camera captures N speckle images. Solve to obtain 3N world coordinate errors (3D coordinate errors of the specimen to be tested in 3 directions can be obtained from 1 speckle image captured by 4 cameras, so 3N world coordinate errors can be obtained from N speckle images captured by 4 cameras);

[0010] Step 4. Build the neural network and train it, and perform different operations for different neural networks:

[0011] (1) For the convolutional neural network: Build a convolutional neural network with N speckle images as input and N speckle images as output. Use the speckle images with heat flux perturbation as the input of the neural network, perform normalization processing on the input, and use the speckle images without heat flux perturbation as the output of the neural network to train the convolutional neural network model;

[0012] (2) For the BP neural network: Build a BP neural network with 8N data inputs and 3N data outputs. Use the image coordinates with heat flux perturbation as the input of the neural network, perform normalization processing on the input, and use the world coordinate errors without heat flux perturbation as the output of the neural network to train the BP neural network model;

[0013] Step 5. Obtain experimental data: When there is heat flux perturbation, move the object to be measured within the range where the object to be measured moves in Step 3, and capture the speckle images of the object to be measured;

[0014] Step 6. Input the experimental data, and perform different operations for different neural networks:

[0015] (1) For the convolutional neural network: Use the speckle images with heat flux perturbation before and after deformation in Step 5 as the input of the neural network, perform normalization processing on the input, and output the speckle images after suppressing the heat flux perturbation;

[0016] (2) For the BP neural network: Use the image coordinates with heat flux perturbation before and after deformation in Step 5 as the input of the neural network, perform normalization processing on the input, and output the world coordinate errors after suppressing the heat flux perturbation.

[0017] Step 7. Calculate the deformation of the object to be measured, and perform different operations for different neural networks:

[0018] (1) For the convolutional neural network: Calculate the world coordinates of the object to be measured in the world coordinate system using the speckle image predicted in step 6, and then the deformation of the object to be measured in step 5 can be solved.

[0019] (2) For the BP neural network: Calculate the world coordinates of the object to be measured in the world coordinate system using the world coordinate error predicted in step 6, and then the deformation of the object to be measured in step 5 can be solved.

[0020] Further, in step 2, calibrate the multi-camera system pairwise to determine the internal and external parameters of each camera. The process is as follows:

[0021] Step 2.1: Determine one camera in the multi-camera system as the central camera, and the remaining 3 cameras need to be calibrated with it respectively.

[0022] Step 2.2: Place a black and white checkerboard with appropriate size in the field of view of the camera so that it occupies half of the camera's field of view.

[0023] Step 2.3: Collect images of the checkerboard in different poses. The checkerboard needs to be transformed at least 10 times.

[0024] Step 2.4: Take the images of the checkerboard in different poses of the central camera and the first camera. By identifying the positions of the checkerboard corners, determine the internal parameters of the two cameras and the external parameters between the two cameras.

[0025] Step 2.5: Take the images of the checkerboard in different poses of the central camera and the second and third cameras, and repeat step 2.4 to obtain the internal parameters of all cameras and the external parameters between the cameras.

[0026] Further, in step 3, to obtain the training data, spray a speckle pattern on the object to be measured, randomly place it in the field of view of the multi-camera system, and perform different operations for different neural networks. The specific process is as follows:

[0027] (1) For the convolutional neural network:

[0028] Step 3A.1: Spray a speckle pattern on the object to be measured. The speckle pattern is generated according to the digital speckle field. The digital speckle field is designed and produced by controlling the number of spots, the center coordinates of the circles, and the radius of the circles. The digital speckle field is generated by the following 4 formulas:

[0029]

[0030]

[0031]

[0032] n = ρA / (0.25·πd2 ) (4)

[0033] Among them, (X1, Y1) is the center coordinate of the first speckle spot, (X i , Y i ) and (X i ', Y i ) are the center coordinates of the speckle spots in the regularly distributed speckle field and the randomly distributed speckle field respectively, a is the center distance between two speckle spots in the regularly distributed speckle field, ρ is the duty cycle, d is the speckle diameter, f(r) represents a random function in the interval (-r, r), r is a randomness factor with a range interval of (0, 1], and n is related to the number of speckles and the resolution A of the camera;

[0034] Step 3A.2: Collect speckle images, and the specific mode is as follows:

[0035] When there is heat flux disturbance, move the object to be measured to different positions, and each camera collects N speckle images; when there is no heat flux disturbance, move the object to be measured to the same acquisition position as when there is heat flux disturbance, and each camera collects N speckle images;

[0036] (2) For the BP neural network:

[0037] Step 3B.1: Spray a speckle pattern on the object to be measured, which is the same as step 3A.1;

[0038] Step 3B.2: Collect speckle images, which is the same as step 3A.2;

[0039] Step 3B.3: Solve the image coordinates from the speckle images, and the specific mode is as follows:

[0040] First, select a certain speckle image before deformation as the reference image, select a certain point in the reference image as the point to be measured, and determine the image coordinates (u0, v0) of the point to be measured (because this point is manually selected, so the image coordinates of this point can be determined). Set a reference sub-region with the point to be measured as the center, and find its corresponding target sub-region on the target image by satisfying the maximum value of the cross-correlation coefficient C cc . The center position of the target sub-region is the image coordinates (u1, v1) of the point to be measured in this target image, where the cross-correlation coefficient C cc is expressed as follows:

[0041]

[0042] In the formula, f(x i , y i ) is the gray value of the point with coordinates (x i , y i ) in the reference image sub-region, and g(x i ′, y i') is the grayscale value of the point with coordinates (x i ', y i ') in the target image sub-region (the coordinates are all local coordinates centered on the midpoint of the sub-region), is the average grayscale value of the reference sub-region, is the average grayscale value of the sub-region in the target image with the same size as the reference sub-region;

[0043] Step 3B.4. Solve the world coordinates from the image coordinates, and the specific mode is as follows:

[0044] Taking four cameras as an example, the coordinates of a point P in the world coordinate system are (X, Y, Z), and the image coordinates in the camera coordinate system are (u, v). According to the pinhole imaging model, the following relationship exists between the world coordinates and the image coordinates:

[0045]

[0046] where A i is the internal parameter matrix of the camera, R i , T i are called the rotation matrix and translation matrix of the camera, r and t are the elements in the matrix respectively, s is the projection of the distance from the object point to the optical center in the optical axis direction, and the image coordinates of the intersection of the optical axis and the image plane are c x and c y , the optical axis refers to the axis of symmetry of the optical system, f s is the tilt factor between the two coordinate axes of the image plane, also known as the distortion parameter, and the ratios of the focal length f to the horizontal and vertical physical sizes of a single pixel are the equivalent focal lengths f x , f y For a four-camera system, the superscripts 0, 1, 2, and 3 represent the left, right, upper, and lower cameras respectively. Then the projection of the spatial point on the camera plane is:

[0047]

[0048] Using the image coordinates of the corresponding points in the camera, obtain the world coordinates of the three-dimensional points in the world coordinate system:

[0049]

[0050] Step 3B.5. Solve the world coordinate error from the world coordinates, and the specific mode is as follows:

[0051] Under the state of no heat flow, take 100 groups of static speckle patterns of the object to be measured each time the movement is made, and calculate the average value of the world coordinates:

[0052]

[0053] Therefore, the world coordinate error is expressed as:

[0054]

[0055] In the formula, is the mean of the world coordinates of the point to be measured, X i , Y i , Z i are the world coordinates of the point to be measured.

[0056] Furthermore, in step 4, a neural network is built and trained, and different operations are performed for different neural networks. The specific process is as follows:

[0057] (1) For the convolutional neural network:

[0058] Taking the N speckle images with heat flux disturbance obtained in step 3 as the input and N speckle images without heat flux disturbance as the output, a convolutional neural network is constructed to train the speckle images;

[0059] Step 4A.1, Feature extraction: First, the input speckle image passes through the first convolutional layer and the first batch normalization layer to obtain the shallow feature X1. The shallow feature X1 passes through the second convolutional layer, the second batch normalization layer, and the first dropout layer in sequence to obtain the further feature X2. The further feature X2 passes through the third convolutional layer, the third batch normalization layer, and the second dropout layer in sequence to obtain the further feature X3. Further, the feature X3 passes through the fourth convolutional layer, the fourth batch normalization layer, and the third dropout layer in sequence to obtain the deep feature X4;

[0060] Step 4A.2, Feature fusion: The feature X4 passes through the first deconvolution layer, the fifth batch normalization layer, and the fourth dropout layer, and then is fused with the feature X3 through the first adder to obtain the feature X5. The feature X5 passes through the second deconvolution layer, the sixth batch normalization layer, and the fifth dropout layer, and then is fused with the feature X2 through the second adder to obtain the feature X6. The feature X6 passes through the third deconvolution layer, the seventh batch normalization layer, and the sixth dropout layer, and then is fused with the feature X1 through the third adder to obtain the feature X7. The feature X7 passes through the fourth deconvolution layer to obtain the feature X8. The further feature X8 passes through a group of residual structures to obtain the feature X9. Finally, the feature X9 passes through the fifth convolutional layer and is fused with the feature X8 through the fourth adder to obtain the final fused feature X 10 , which is the speckle image to be output;

[0061] (2) For the BP neural network:

[0062] Take the image coordinates of the 8N heat flux perturbations obtained in step 3 as the input and the world coordinate errors of the 3N heat flux-free perturbations as the output, and construct a BP neural network to train the data;

[0063] Step 4B.1, signal forward propagation: First, the image coordinates pass through each node of the input layer to obtain the output value O j ; Then O j propagates to each node of the hidden layer to obtain the output value P j ; Finally, P j propagates to each node of the output layer to obtain the output value Q k ;

[0064] Step 4B.2, error backpropagation: First, establish an error function E between the output value of the output layer and the true value, optimize the structure of the output layer by minimizing E, and then optimize the structure of the hidden layer by minimizing E;

[0065] Step 4B.3, repeat steps 4B.1 and 4B.2 until the error function E is satisfied. At this time, the output value Q k of the output layer is the world coordinate error to be output.

[0066] Furthermore, in step 7, calculate the deformation of the object to be measured and perform different operations for different neural networks. The specific process is as follows:

[0067] (1) For the convolutional neural network:

[0068] Step 7A.1, calculate the world coordinates of the object to be measured through the speckle image predicted by the convolutional neural network in step 6;

[0069] Step 7A.2, calculate the three-dimensional displacement of the object to be measured in step 5 from the world coordinates. The world coordinates of the test specimen before deformation are (X0, Y0, Z0), and the world coordinates after deformation are (X i , Y i , Z i ). Then the three-dimensional displacement is:

[0070]

[0071] In the formula, U, V, and W are the displacements in the three directions respectively, and r i is the total displacement;

[0072] Step 7A.3, calculate the three-dimensional strain of the object to be measured in step 5 from the three-dimensional displacement. Establish a local coordinate system O e , and transform the three-dimensional coordinates and three-dimensional displacements of the grid points in the world coordinate system before deformation into the coordinate system Oe In which, (X e , Y e , Z e ) and (U e , V e , W e ) are obtained, and the displacement field function is obtained by using the quadratic surface fitting method, which is expressed as follows:

[0073]

[0074] Wherein, and are the coefficients of the displacement field functions U e , V e , W e respectively, then the full-field strain is expressed as follows:

[0075]

[0076] In the formula, ε xx , ε yy , ε zz , ε yz , ε zy , ε xy , ε yx , ε zx , ε xz represent the strain tensor, (X e , Y e , Z e ) and (U e , V e , W e ) represent the coordinates and displacements in the local coordinate system;

[0077] (2) For the BP neural network:

[0078] Step 7B.1: Calculate the world coordinates of the object to be measured from the world coordinate error predicted by the BP neural network in Step 6. The specific process is as follows:

[0079]

[0080] In the formula, is the world coordinate mean value, ΔX i , ΔY i , ΔZ i are the world coordinate errors,

[0081] Step 7B.2: Calculate the three-dimensional displacement of the object to be measured in Step 5 from the world coordinates, which is the same as Step 7A.2;

[0082] Step 7B.3: Calculate the three-dimensional strain of the object to be measured in step 5 from the three-dimensional displacement, which is the same as step 7A.3.

[0083] Compared with the prior art, the present invention has the following significant advantages: 1) It creatively applies machine learning models to the application of digital image correlation technology, solves the impact of thermal flow disturbance on imaging quality, effectively suppresses light deflection and imaging drift caused by thermal flow disturbance in the image, and improves the calculation accuracy of digital image correlation technology. 2) The constructed neural network improves the accuracy of specific application scenarios, can achieve the purpose of rapid measurement with very small data sets, and obtains good results in experiments. 3) It has good applicability for measurements of different numbers of cameras and has high universality for different specific measurement processes. BRIEF DESCRIPTION OF THE DRAWINGS

[0084] Figure 1 Schematic diagram of the experimental device of the present invention.

[0085] Figure 2 The figure is a flow chart of the method of the present invention.

[0086] Figure 3 The figures are a comparison of the effects of the method of the present invention and the traditional method for different numbers of cameras, where (a) is a dual-camera system; (b) is a triple-camera system; and (c) is a quad-camera system.

[0087] 1: Electronic computer;

[0088] 2: Optical platform;

[0089] 3: Camera fixing device;

[0090] 4: Industrial cameras;

[0091] 5: High resolution lens;

[0092] 6: Object to be tested. DETAILED DESCRIPTION

[0093] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0094] The present invention solves the problem of the influence of heat flow disturbance on the quality of stereoscopic vision from the perspective of algorithm design and experimental detection. The method has the advantages of high calculation accuracy, simple equipment, convenience and practicality, low algorithm complexity and fast calculation speed.

[0095] like Figure 1As shown in the figure, a stereo vision deformation measurement method for suppressing heat flow disturbance, and the experimental device includes the following equipment: an electronic computer 1, an optical platform 2, a camera fixing device 3, an industrial camera 4, a high-resolution lens 5, and a to-be-measured object 6. The industrial camera used in the test experiment has 4 million pixels and a lens focal length of 35 mm. This method includes the following steps:

[0096] Step 1, Fixing of the experimental device: Orthogonally arrange four industrial cameras and fix them on the optical platform. Fix the object to be measured on the optical platform. Adjust the directions of the four camera lenses to point to the object to be measured, and make the object to be measured in the center of the camera view. Adjust the camera aperture and focal length to be appropriate;

[0097] Step 2, Calibration of multi-camera system parameters: Calibrate the four cameras pairwise to determine the internal and external parameters of each camera, specifically as follows:

[0098] Step 2.1, Determine one of the cameras in the multi-camera system as the central camera, and the remaining 3 cameras need to be calibrated with it respectively;

[0099] Step 2.2, Place a black and white checkerboard with appropriate size into the camera's field of view so that it occupies half of the camera's field of view;

[0100] Step 2.3, Collect images of the checkerboard in different poses. The checkerboard needs to be transformed at least 10 times;

[0101] Step 2.4, Take the images of the checkerboard in different poses of the central camera and the first camera. By identifying the positions of the checkerboard corners, determine the internal parameters of the two cameras and the external parameters between the two cameras;

[0102] Step 2.5, Take the images of the checkerboard in different poses of the central camera and the second and third cameras, and repeat Step 2.4 to obtain the internal parameters of all cameras and the external parameters between the cameras.

[0103] Step 3, Obtaining of training data: Spray a speckle pattern on the object to be measured and randomly place it in the field of view of the multi-camera system. Perform different operations for different neural networks:

[0104] (1) For the convolutional neural network: When there is heat flow disturbance, move the object to be measured to different positions, and each camera collects N speckle images; When there is no heat flow disturbance, move the object to be measured to the same acquisition position as when there is heat flow disturbance, and each camera collects N speckle images;

[0105] (2) For the BP neural network: When there is heat flux perturbation, move the object to be measured to the same acquisition position as when there is heat flux perturbation. Each camera acquires N speckle images, and 8N image coordinates are solved (one speckle image acquired by one camera has 2-direction image coordinates, so 8N image coordinates are obtained from N speckle images acquired by 4 cameras); when there is no heat flux perturbation, move the object to be measured to the same position. Each camera acquires N speckle images, and 3N world coordinate errors are solved (one speckle image acquired by 4 cameras can obtain the three-dimensional coordinate errors of the test piece in 3 directions, so 3N world coordinate errors can be obtained from N speckle images acquired by 4 cameras);

[0106] The specific process is as follows:

[0107] (1) For the convolutional neural network:

[0108] Step 3.1: Spray a speckle pattern on the object to be measured. The speckle pattern is generated according to the digital speckle field, and the digital speckle field is designed and produced by controlling the number of spots, the center coordinates of the circle, and the radius of the circle. The digital speckle field is generated by the following 4 formulas:

[0109]

[0110]

[0111]

[0112] n = ρA / (0.25·πd 2 ) (4)

[0113] Among them, (X1, Y1) are the center coordinates of the first speckle, (X i , Y i ) and (X i ', Y i ') are the center coordinates of the speckles in the regularly distributed speckle field and the randomly distributed speckle field respectively. a is the center distance between two speckles in the regularly distributed speckle field, ρ is the duty cycle, d is the speckle diameter, f(r) represents a random function in the interval (-r, r), r is a randomness factor in the range interval (0, 1], and n is the number of speckles related to the resolution A of the camera;

[0114] Step 3.2: Acquire speckle images, and the specific mode is as follows:

[0115] When there is heat flux perturbation, move the object to be measured to different positions. Each camera acquires N speckle images; when there is no heat flux perturbation, move the object to be measured to the same position. Each camera acquires N speckle images;

[0116] (2) For the BP neural network:

[0117] Step 3.3: Spray a speckle pattern on the object to be measured, which is the same as step 3.1.

[0118] Step 3.4: Collect the speckle image, which is the same as step 3.2.

[0119] Step 3.5: Solve the image coordinates from the speckle image. The specific method is as follows:

[0120] First, select a certain speckle image before deformation as the reference image, select a certain point in the reference image as the point to be measured, and determine the image coordinates (u0, v0) of the point to be measured (since this point is manually selected, its image coordinates can be determined). Set a reference sub-region centered on the point to be measured, and find the corresponding target sub-region on the target image by satisfying the maximum value of the cross-correlation coefficient C cc . The central position of the target sub-region is the image coordinates (u1, v1) of the point to be measured in this target image, where the cross-correlation coefficient C cc is expressed as follows:

[0121]

[0122] In the formula, f(x i , y i ) is the gray value of the point with coordinates (x i , y i ) in the reference image sub-region, g(x i ′, y i ′) is the gray value of the point with coordinates (x i ′, y i ′) in the target image sub-region (the coordinates are all local coordinates centered on the midpoint of the sub-region), is the average gray value of the reference sub-region, is the average gray value of the sub-region in the target image with the same size as the reference sub-region;

[0123] Step 3.6: Solve the world coordinates from the image coordinates. The specific method is as follows:

[0124] Taking four cameras as an example, the coordinates of a point P in the world coordinate system are (X, Y, Z), and the image coordinates in the camera coordinate system are (u, v). According to the pinhole imaging model, the following relationship exists between the world coordinates and the image coordinates:

[0125]

[0126] where A i is the internal parameter matrix of the camera, R i , T iIt is called the rotation matrix and translation matrix of the camera, where r and t are the elements in the matrix. s is the projection of the distance from the object point to the optical center in the optical axis direction, and the image coordinates of the intersection of the optical axis and the image plane are c x and c y , and the optical axis refers to the axis of symmetry of this optical system. f s is the tilt factor between the two coordinate axes of the image plane, also known as the distortion parameter, which is generally not considered. The ratios of the focal length f to the horizontal and vertical physical sizes of a single pixel are the equivalent focal lengths f x 、f y For the four-camera system, the superscripts 0, 1, 2, and 3 represent the left, right, upper, and lower cameras respectively, and the projection of the spatial point on the camera plane is:

[0127]

[0128] Conversely, once the image coordinates of the corresponding points in the camera are known, the world coordinates of the three-dimensional points in the world coordinate system can be obtained:

[0129]

[0130] The world coordinates of the point to be measured can be solved by solving this indeterminate equation

[0131] Step 3.6, Solve the world coordinate error from the world coordinates, and the specific mode is as follows:

[0132] In the state of no heat flux, 100 groups of static speckle patterns of the object to be measured are taken each time, and the mean value of the world coordinates is calculated:

[0133]

[0134] Therefore, the world coordinate error is expressed as:

[0135]

[0136] In the formula, is the mean value of the world coordinates of the point to be measured, and X i 、Y i 、Z i are the world coordinates of the point to be measured.

[0137] Step 4, Build and train the neural network, and perform different operations for different neural networks:

[0138] (1) For the convolutional neural network: Build a convolutional neural network with N speckle images as input and N speckle images as output. Use the speckle image with heat flux perturbation as the input of the neural network, perform normalization processing on the input, and use the speckle image without heat flux perturbation as the output of the neural network to train the convolutional neural network model;

[0139] (2) For the BP neural network: Build a BP neural network with 8N data inputs and 3N data outputs. Use the image coordinates with heat flux disturbance as the neural network input, and perform normalization on the input. Use the world coordinate error without heat flux disturbance as the neural network output to train the BP neural network model.

[0140] The specific process is as follows:

[0141] (1) For the convolutional neural network:

[0142] Through step 3, N speckle images with heat flux disturbance have been obtained as inputs and N speckle images without heat flux disturbance have been obtained as outputs. Next, build a convolutional neural network to train the data.

[0143] Step 4.1 Feature extraction: First, the input of the speckle image passes through the first convolutional layer and the first batch normalization layer to obtain the shallow feature X1. The shallow feature X1 passes through the second convolutional layer, the second batch normalization layer, and the first dropout layer in sequence to obtain the further feature X2. The further feature X2 passes through the third convolutional layer, the third batch normalization layer, and the second dropout layer in sequence to obtain the further feature X3. Further, the feature X3 passes through the fourth convolutional layer, the fourth batch normalization layer, and the third dropout layer in sequence to obtain the deep feature X4;

[0144] Step 4.2 Feature fusion: After the feature X4 passes through the first transposed convolution, the fifth batch normalization layer, and the fourth dropout layer in sequence, it is fused with the feature X3 through the first adder to obtain the feature X5. After the feature X5 passes through the second transposed convolution, the sixth batch normalization layer, and the fifth dropout layer in sequence, it is fused with the feature X2 through the second adder to obtain the feature X6. After the feature X6 passes through the third transposed convolution, the seventh batch normalization layer, and the sixth dropout layer in sequence, it is fused with the feature X1 through the third adder to obtain the feature X7. The feature X7 passes through the fourth transposed convolution to obtain the feature X8. The further feature X8 passes through a group of residual structures to obtain the feature X9. Finally, after the feature X9 passes through the fifth convolutional layer, it is fused with the feature X8 through the fourth adder to obtain the final fused feature X 10 , which is the required output speckle image.

[0145] (2) For the BP neural network:

[0146] Through step 3, 8N image coordinates have been obtained as input and 3N world coordinate errors have been obtained as output. Next, a BP neural network is constructed to train the data.

[0147] Step 4.3, Forward propagation of signals: First, the image coordinates pass through each node of the input layer to obtain the output value O of the input layer. j ; Then O j propagates to each node of the hidden layer to obtain the output value P of the hidden layer. j ; Finally, P j propagates to each node of the output layer to obtain the output value Q of the output layer. k .

[0148] Step 4.4, Backward propagation of errors: First, establish an error function E between the output value of the output layer and the true value. By minimizing E, the structure of the output layer is optimized, and then by minimizing E, the structure of the hidden layer is optimized.

[0149] Step 4.5, Repeat step 4.3 and step 4.4 until the error function E is satisfied. At this time, the output value Q of the output layer k is the world coordinate error to be output.

[0150] Step 5, Obtaining experimental data: When there is a heat flux disturbance, move the object to be measured within the range where the object to be measured moves in step 3, and collect the images of the object to be measured.

[0151] Step 6, Input of experimental data:

[0152] (1) For the convolutional neural network: Normalize the speckle images with heat flux disturbance before and after deformation in step 5, and input them into the trained convolutional neural network to output the speckle image after suppressing the heat flux disturbance.

[0153] (2) For the BP neural network: Normalize the image coordinates with heat flux disturbance before and after movement in step 5, and input them into the trained BP neural network to output the world coordinate error after suppressing the heat flux disturbance.

[0154] Step 7, Calculate the deformation of the object to be measured, and perform different operations for different neural networks:

[0155] (1) For the convolutional neural network: Calculate the world coordinates of the object to be measured in the world coordinate system through the speckle image predicted in step 6, and then the deformation of the object to be measured in step 5 can be solved.

[0156] (2) For the BP neural network: Calculate the world coordinates of the object to be measured in the world coordinate system through the world coordinate error predicted in step 6, and then the deformation of the object to be measured in step 5 can be solved.

[0157] The details are as follows:

[0158] (1) For the convolutional neural network:

[0159] Step 7.1: Calculate the world coordinates of the object to be measured from the speckle image predicted by the convolutional neural network in Step 6, which is the same as in Step 3.6;

[0160] Step 7.2: Calculate the deformation of the object to be measured that occurred in Step 5 from the world coordinates. The world coordinates of the test specimen before deformation are (X0, Y0, Z0), and the world coordinates after deformation are (X i , Y i , Z i ). Then the three-dimensional displacement is:

[0161]

[0162] where U, V, and W are the displacements in the three directions respectively, and r i is the total displacement;

[0163] Step 7.3: Calculate the three-dimensional strain of the object to be measured that occurred in Step 5 from the three-dimensional displacement. Establish a local coordinate system O e , and transform the three-dimensional coordinates and three-dimensional displacements of the grid points in the world coordinate system before deformation into the coordinate system O e to obtain (X e , Y e , Z e ) and (U e , V e , W e ). Use the quadratic surface fitting method to obtain the displacement field function, which is expressed as follows:

[0164]

[0165] where and are the coefficients of the displacement field functions U e , V e , W e respectively. Then the full-field strain is expressed as follows:

[0166]

[0167] where ε xx , ε yy , ε zz , ε yz , ε zy , ε xy , ε yx , ε zx , ε xz represent the strain tensor, (Xe , Y e , Z e ) and (U e , V e , W e ) represent the coordinates and displacements in the local coordinate system.

[0168] (2) For the BP neural network:

[0169] Step 7.4: Calculate the world coordinates of the object to be measured based on the world coordinate error predicted by the BP neural network in Step 6. The specific process is as follows:

[0170]

[0171] In the formula, is the mean of the world coordinates, ΔX i , ΔY i , ΔZ i are the world coordinate errors.

[0172] Step 7.5: Calculate the three-dimensional displacement of the object to be measured in Step 5 from the world coordinates, which is the same as in Step 7.2.

[0173] Step 7.6: Calculate the three-dimensional strain of the object to be measured in Step 5 from the three-dimensional displacement, which is the same as in Step 7.3.

[0174] Embodiment

[0175] To verify the effectiveness of the proposed solution of the present invention, the following experiment is carried out.

[0176] 1) Data acquisition and preprocessing

[0177] Spray a speckle pattern on the object to be measured and randomly place it in the field of view of the multi-camera system. When there is a heat flux disturbance, move the position of the object to be measured and collect 100 images at positions of 0 mm, 2.75 mm, 5.5 mm, 8.25 mm, and 11 mm. After normalizing the images, use them as the input of the training set; when there is no heat flux disturbance, move the position of the object to be measured and collect 100 images at the same positions as the output of the training set.

[0178] 2) Establish a neural network model and train

[0179] Build a BP neural network with 8 input layers, 3 output layers, and 10 hidden layers; select tansig as the activation function and mse as the error function for training.

[0180] Build a convolutional neural network with 3 input image data and 3 output image data; select the Adaptive Moment Estimation (ADAM) as the optimizer for training.

[0181] 3) Prediction of actual data

[0182] In the actual experiment, the surface of the flat plate was sprayed with speckles, and the flat plate was moved 1 mm, 2 mm, 3 mm, 4 mm, 5 mm, 6 mm, 7 mm, 8 mm, 9 mm, and 10 mm. After normalizing the collected images and inputting them into the trained neural network, the images after suppressing the heat flux disturbance can be obtained, and then the three-dimensional coordinates can be solved according to the images. The prediction results are as Figure 2 shown. The neural network built by the present invention can effectively eliminate the influence of heat flux disturbance on the speckle image and improve the measurement accuracy; as Figure 2 shown, the black solid line represents the displacement error calculated by the traditional three-dimensional reconstruction method, and the red solid line represents the displacement error calculated by the method of this patent. It can be clearly seen that for different numbers of multi-camera systems, the present invention patent can effectively improve the measurement accuracy of digital image correlation under heat flux disturbance (where a is a two-camera system, b is a three-camera system, and c is a four-camera system).

[0183] The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity of description, not all possible combinations of the technical features in the above embodiments are described. However, as long as these combinations of technical features do not conflict, they should be considered as the scope described in this specification.

[0184] The above-described embodiments only represent several implementation manners of the present application. The description is relatively specific and detailed, but it should not be construed as a limitation on the scope of the present application. It should be noted that for those of ordinary skill in the art, without departing from the concept of the present application, several modifications and improvements can still be made, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the appended claims.

Claims

1. A stereoscopic vision deformation measurement method for suppressing heat flow disturbance, characterized in that The experimental device includes an industrial camera, a lens, an optical platform, a camera fixing device, an electronic computer, and an object to be measured. The measurement method includes the following steps: Step 1, Fixing the experimental device: Orthogonally arrange four industrial cameras and fix them on the optical platform. Fix the object to be measured on the optical platform, adjust the directions of the four camera lenses to point to the object to be measured, and make the object to be measured centered in the camera's view. Adjust the camera aperture and focal length to be appropriate; Step 2, Calibrating the parameters of the multi-camera system: Calibrate the four cameras pairwise to determine the internal and external parameters of each camera; Step 3, Obtaining training data: Spray a speckle pattern on the object to be measured and randomly place it in the field of view of the multi-camera system. Perform different operations for different neural networks: (1) For the convolutional neural network: When there is a heat flux perturbation, move the object to be measured to different positions, and each camera collects N speckle images; When there is no heat flux perturbation, move the object to be measured to the same acquisition positions as when there is a heat flux perturbation, and each camera collects N speckle images; (2) For the BP neural network: When there is a heat flux perturbation, move the object to be measured to different positions, and each camera collects N speckle images, and solve to obtain 8N image coordinates; When there is no heat flux perturbation, move the object to be measured to the same acquisition positions as when there is a heat flux perturbation, and each camera collects N speckle images, and solve to obtain 3N world coordinate errors; Step 4, Building and training the neural network. Perform different operations for different neural networks: (1) For the convolutional neural network: Build a convolutional neural network with N speckle image inputs and N speckle image outputs. Use the speckle images with heat flux perturbation as the neural network input, perform normalization processing on the input, and use the speckle images without heat flux perturbation as the neural network output to train the convolutional neural network model; (2) For the BP neural network: Build a BP neural network with 8N data inputs and 3N data outputs. Use the image coordinates with heat flux perturbation as the neural network input, perform normalization processing on the input, and use the world coordinate errors without heat flux perturbation as the neural network output to train the BP neural network model; Step 5, Obtaining experimental data: When there is a heat flux perturbation, move the object to be measured within the range where the object to be measured moves in Step 3 and collect the speckle images of the object to be measured; Step 6, Inputting the experimental data. Perform different operations for different neural networks: (1) For the convolutional neural network: Use the speckle images with heat flux perturbation before and after deformation in Step 5 as the input of the neural network, perform normalization processing on the input, and output the speckle images after suppressing the heat flux perturbation; (2) For the BP neural network: Use the image coordinates with heat flux perturbation before and after deformation in Step 5 as the input of the neural network, perform normalization processing on the input, and output the world coordinate errors after suppressing the heat flux perturbation; Step 7, Calculating the deformation of the object to be measured. Perform different operations for different neural networks: (1) For the convolutional neural network: Calculate the world coordinates of the object to be measured in the world coordinate system using the speckle image predicted in step 6, and then the deformation of the object to be measured in step 5 can be solved. (2) For the BP neural network: Calculate the world coordinates of the object to be measured in the world coordinate system using the world coordinate error predicted in step 6, and then the deformation of the object to be measured in step 5 can be solved.

2. The three-dimensional vision deformation measurement method for suppressing heat flow disturbance according to claim 1, wherein In step 2, calibrate the multi-camera system pairwise to determine the internal and external parameters of each camera. The process is as follows: Step 2.1: Determine one camera in the multi-camera system as the central camera, and the remaining 3 cameras need to be calibrated with it respectively. Step 2.2: Place a black and white checkerboard with appropriate size in the field of view of the camera so that it occupies half of the camera's field of view. Step 2.3: Collect images of the checkerboard in different poses. The checkerboard needs to be transformed at least 10 times. Step 2.4: Take the images of the checkerboard in different poses of the central camera and the first camera. By identifying the positions of the checkerboard corners, determine the internal parameters of the two cameras and the external parameters between the two cameras. Step 2.5: Take the images of the checkerboard in different poses of the central camera and the second and third cameras, and repeat step 2.4 to obtain the internal parameters of all cameras and the external parameters between the cameras.

3. A stereoscopic vision deformation measurement method for suppressing heat flow disturbance according to claim 1, characterized in that In step 3, to obtain the training data, spray a speckle pattern on the object to be measured and randomly place it in the field of view of the multi-camera system. Different operations are performed for different neural networks. The specific process is as follows: (1) For the convolutional neural network: Step 3A.1: Spray a speckle pattern on the object to be measured. The speckle pattern is generated according to the digital speckle field, and the digital speckle field is designed and produced by controlling the number of spots, the center coordinates of the circles, and the radius of the circles. The digital speckle field is generated by the following 4 formulas: n = ρA / (0.25·πd 2 ) (4) Among them, (X1, Y1) is the center coordinate of the first speckle point, (X i , Y i ), and (X i ', Y i ') are the center coordinates of the speckle points in the regularly distributed speckle field and the randomly distributed speckle field respectively, a is the center distance between two speckle points in the regularly distributed speckle field, ρ is the duty cycle, d is the speckle diameter, f(r) represents a random function in the interval (-r, r), r is a randomness factor with a range interval of (0, 1], and n is related to the number of speckles and the resolution A of the camera; Step 3A.2: Collect the speckle images. The specific mode is as follows: When there is heat flux perturbation, move the object to be measured to different positions, and each camera collects N speckle images; when there is no heat flux perturbation, move the object to be measured to the same collection positions as when there is heat flux perturbation, and each camera collects N speckle images. (2) For the BP neural network: Step 3B.1: Spray a speckle pattern on the object to be measured, which is the same as step 3A.

1. Step 3B.2: Collect the speckle images, which is the same as step 3A.

2. Step 3B.3: Solve the image coordinates from the speckle images. The specific mode is as follows: First, select a certain speckle image before deformation as the reference image, select a certain point in the reference image as the point to be measured, determine the image coordinates (u0, v0) of the point to be measured, set a reference sub-region centered on the point to be measured, and find its corresponding target sub-region on the target image by satisfying the maximum value of the cross-correlation coefficient C cc The center position of the target sub-region is the image coordinates (u1, v1) of the point to be measured in the target image. The cross-correlation coefficient C cc is expressed as follows: where f(x i , y i ) is the gray value of the point with coordinates (x i , y i ) in the reference image sub-region, and g(x i ′, y i ′) is the gray value of the point with coordinates (x i ′, y i ′) in the target image sub-region, is the average gray value of the reference sub-region, is the average gray value of the sub-region in the target image with the same size as the reference sub-region; Step 3B.4: Solve the world coordinates from the image coordinates. The specific mode is as follows: Taking a four-camera as an example, the coordinates of a point P in the world coordinate system are (X, Y, Z), and the image coordinates in the camera coordinate system are (u, v). According to the pinhole imaging model, the following relationship exists between the world coordinates and the image coordinates: Among which A i is the internal parameter matrix of the camera, R i , T i are called the rotation matrix and translation matrix of the camera, r and t are the elements in the matrix respectively, s is the projection of the distance from the object point to the optical center in the optical axis direction, and the image coordinates of the intersection point of the optical axis and the image plane are c x and c y , the optical axis refers to the axis of symmetry of the optical system, f s is the tilt factor between the two coordinate axes of the image plane, also known as the distortion parameter, and the ratios of the focal length f to the horizontal and vertical physical sizes of a single pixel are the equivalent focal lengths f x , f y For a four-camera system, the superscripts 0, 1, 2, 3 represent the left, right, upper, and lower cameras respectively, and the projection of a spatial point on the camera plane is: Use the image coordinates of the corresponding points in the camera to obtain the world coordinates of the three-dimensional points in the world coordinate system: Step 3B.5: Solve the world coordinate error from the world coordinates. The specific mode is as follows: In the state of no heat flux, take 100 groups of static speckle images of the object to be measured each time the object is moved, and calculate the average value of the world coordinates: Therefore, the world coordinate error is expressed as: In the formula, is the mean of the world coordinates of the point to be measured, and X i , Y i , Z i are the world coordinates of the point to be measured.

4. A stereo vision deformation measurement method for suppressing heat flow disturbance according to claim 1, characterized in that, In step 4, build the neural network and train it. Different operations are performed for different neural networks. The specific process is as follows: (1) For the convolutional neural network: Using the N speckle images with heat flux perturbations obtained in step 3 as input and the N speckle images without heat flux perturbations as output, construct a convolutional neural network to train the speckle images; Step 4A.1, Feature extraction: First, the input speckle image passes through the first convolutional layer and the first batch normalization layer to obtain the shallow feature X1. The shallow feature X1 then passes through the second convolutional layer, the second batch normalization layer, and the first dropout layer in sequence to obtain the further feature X2. The further feature X2 then passes through the third convolutional layer, the third batch normalization layer, and the second dropout layer in sequence to obtain the further feature X3. Further still, the feature X3 passes through the fourth convolutional layer, the fourth batch normalization layer, and the third dropout layer in sequence to obtain the deep feature X4; Step 4A.2, Feature Fusion: Feature X4 passes through the first transposed convolution, the fifth batch normalization layer, and the fourth dropout layer in sequence, and then is fused with Feature X3 through the first adder to obtain Feature X5. Feature X5 passes through the second transposed convolution, the sixth batch normalization layer, and the fifth dropout layer in sequence, and then is fused with Feature X2 through the second adder to obtain Feature X6. Feature X6 passes through the third transposed convolution, the seventh batch normalization layer, and the sixth dropout layer in sequence, and then is fused with Feature X1 through the third adder to obtain Feature X7. Feature X7 passes through the fourth transposed convolution to obtain Feature X8. Further, Feature X8 passes through a group of residual structures to obtain Feature X9. Finally, Feature X9 passes through the fifth convolution layer and is fused with Feature X8 through the fourth adder to obtain the final fused feature X 10 , which is the speckle image to be output; (2) For the BP neural network: Using the 8N image coordinates with heat flux perturbations obtained in step 3 as input and the 3N world coordinate errors without heat flux perturbations as output, construct a BP neural network to train the data; Step 4B.1, Forward propagation of signals: First, the image coordinates pass through each node of the input layer to obtain the output value O of the input layer j ; Then O j propagates to each node of the hidden layer to obtain the output value P of the hidden layer j ; Finally, P j propagates to each node of the output layer to obtain the output value Q of the output layer k ; Step 4B.2, Error backpropagation: First, establish the error function E between the output value and the true value of the output layer. Optimize the structure of the output layer by minimizing E, and then optimize the structure of the hidden layer by minimizing E; Step 4B.

3. Repeat Step 4B.1 and Step 4B.2 until the error function E is satisfied. At this time, the output value Q of the output layer k is the required output world coordinate error.

5. A stereoscopic vision deformation measurement method for suppressing heat flow disturbance according to claim 1, characterized in that In step 7, calculate the deformation of the object to be measured and perform different operations for different neural networks. The specific process is as follows: (1) For the convolutional neural network: Step 7A.1, Calculate the world coordinates of the object to be measured from the speckle image predicted by the convolutional neural network in step 6; Step 7A.

2. Calculate the three-dimensional displacement of the object to be measured in Step 5 from the world coordinates. The world coordinates of the specimen to be tested before deformation are (X0, Y0, Z0), and the world coordinates after deformation are (X i , Y i , Z i ). Then the three-dimensional displacement is as follows: where U, V, and W are the displacements in three directions respectively, and r i is the total displacement; Step 7A.

3. Calculate the three-dimensional strain that the object under test undergoes in Step 5 from the three-dimensional displacements, and establish a local coordinate system O e . Transform the three-dimensional coordinates and three-dimensional displacements of the grid points in the world coordinate system before deformation into the coordinate system O e , and obtain (X e , Y e , Z e ) and (U e , V e , W e ). Use the quadratic surface fitting method to obtain the displacement field function, which is expressed as follows: Among them, and are the coefficients of the displacement field functions U e , V e , W e respectively, and the full-field strain is expressed as follows: where ε xx , ε yy , ε zz , ε yz , ε zy , ε xy , ε yx , ε zx , ε xz represent the strain tensor, (X e , Y e , Z e ) and (U e , V e , W e ) represent the coordinates and displacements in the local coordinate system; (2) For the BP neural network: Step 7B.1, Calculate the world coordinates of the object to be measured from the world coordinate error predicted by the BP neural network in step 6. The specific process is as follows: In the formula, is the mean of world coordinates, and ΔX i , ΔY i , and ΔZ i are the world coordinate errors. Step 7B.2, Calculate the three-dimensional displacement of the object to be measured in step 5 from the world coordinates, which is the same as step 7A.2; Step 7B.3, Calculate the three-dimensional strain of the object to be measured in step 5 from the three-dimensional displacement, which is the same as step 7A.3.

Images