A flexible fingertip deformation measurement method based on DIC

Abstract

The present invention discloses a flexible fingertip deformation measurement method based on DIC, establishes a flexible fingertip deformation measurement system, first collects camera calibration image data, and calibrates the camera using a combination method based on multi-step phase shift and LBP encoding; then collects images before and after glass refraction, and uses a BP neural network to correct the glass refraction; finally, collects image data to be measured, uses the DIC method and combines the proposed camera calibration method and glass refraction correction algorithm to complete the measurement and analysis of the surface deformation of the soft object under test when pressing the glass. The present invention uses a flexible fingertip deformation measurement system based on DIC, an improved camera calibration method based on multi-step phase shift method and LBP encoding, and a BP neural network glass refraction correction method to detect the three-dimensional deformation information of the target, and the detection is fast, and the detection result is consistent with the actual deformation situation.

Description

Technical Field

[0001] The invention relates to the fields of optical three-dimensional measurement and computer vision, and in particular to a flexible fingertip deformation measurement technology based on DIC. Background Art

[0002] Digital image correlation uses the texture information of the material surface (natural or artificially sprayed) as a carrier, and obtains the matching relationship of the texture image before and after the material is deformed through correlation calculation, so as to analyze the deformation of the measured material. From the measurement dimension, the digital image correlation DIC method is divided into 2D-DIC and 3D-DIC, which can measure the in-plane displacement of the material surface and can also be used to evaluate the three-dimensional deformation of non-planar objects.

[0003] Soft materials are complex substances between solids and ideal fluids. They include liquid crystals, polymers, colloids, biological tissues and other soft materials. Compared with hard materials such as metals, stones, and hard plastics, soft materials have the characteristics of low modulus and stretchability. Soft materials can respond to tiny external physical and chemical stimuli. In recent years, they have played an increasingly important role in flexible smart devices, medical devices, wearable devices and other fields. However, compared with traditional solid materials, soft materials are more likely to lose their original surface morphology under the influence of external factors, which brings certain difficulties to the analysis of the mechanical behavior of the system. Affected by different external loads and the mechanical properties of the material itself, the stress area often produces different degrees of deformation. Therefore, studying the deformation law of materials under stress is an urgent problem to further understand the mechanical behavior of soft materials in their applications. Summary of the invention

[0004] The technical problem to be solved by the present invention is to provide a flexible fingertip deformation measurement technology based on DIC, use an improved camera calibration method based on multi-step phase shift and LBP encoding to calibrate the camera, and use a BP neural network to establish a glass refraction correction model, ultimately achieving three-dimensional deformation measurement of the target.

[0005] The technical solution of the present invention is:

[0006] A method for measuring deformation of a flexible fingertip based on DIC comprises the following steps:

[0007] (1) Establishing a flexible fingertip deformation measurement system: The flexible fingertip deformation measurement system includes a base, two camera modules arranged on the base, a cover shell arranged outside the two camera modules, and a processor. The top plate and two parallel side plates of the cover shell are both shading plates, and the other two parallel side plates of the cover shell are transparent plates. The two camera modules face the two transparent plates respectively. Each camera module includes a left camera and a right camera. The two camera modules are connected to the processor.

[0008] (2) Camera model establishment: First, the upper left corner of the LCD screen is used as the origin O, the X-axis is established along the axis of the short side of the LCD screen, the Y-axis is established along the axis of the long side of the LCD screen, and the horizontal axis perpendicular to the LCD screen is used as the Z-axis, thereby establishing the world coordinate system O-XYZ. Then, a method combining multi-step phase shift and LBP encoding is used to collect the PWG image displayed on the LCD screen and perform camera calibration. Finally, a binocular imaging model is established based on the calibration results of each group of camera modules.

[0009] (3) Glass refraction correction based on neural network: First, two groups of camera modules in the housing respectively collect images of periodic cross-stripe targets outside the housing and arranged parallel to the transparent plate. Then, Fourier transform is used to solve the wrapped phase of the periodic cross-stripe target in the periodic cross-stripe target image. Then, the phase is unwrapped according to the mark points on the periodic cross-stripe target to obtain the absolute phase. Finally, the training data set of the BP neural network is obtained by pixel matching based on the corresponding relationship between the absolute phases before and after refraction. The training data set is substituted into the BP neural network for training. The training result is the glass refraction correction model.

[0010] (4) Image data acquisition: The soft object to be measured is pressed against the transparent plate of the cover with fingertips. During the pressing process, the camera module is used to acquire images, and the acquired images are sent to the processor. The processor brings the acquired images into the binocular imaging model and the glass refraction correction model for image processing to obtain the processed images;

[0011] (5) Deformation measurement: The digital image correlation method (DIC) is used to evaluate the three-dimensional deformation of the processed image. After three-dimensional reconstruction of each feature point on the image of the soft object to be measured, the continuous three-dimensional deformation of the surface of the soft object to be measured in space is obtained based on the inter-frame connection of the same sub-area.

[0012] The two groups of camera modules are arranged opposite to each other, and the lenses of the left camera and the right camera of each group of camera modules are both placed upward, with an elevation angle of 70-80 degrees, and each group of camera modules faces the opposite transparent plate.

[0013] An FPGA development board is arranged on the base, and both camera modules are connected to the FPGA development board, and the FPGA development board is connected to the external processor for communication.

[0014] In the step (2), the specific steps of using a method combining multi-step phase shift and LBP coding to collect the PWG image displayed on the LCD screen and perform camera calibration are as follows: first, a PWG array pattern is established, and the period number set and the maximum radius of the PWG array pattern are set to T = [3, 4, 5] and r respectively. max=80 pixels, the spacing between adjacent feature points is set to 180 pixels, the PWG array pattern is a group of 3×3, there are 12 groups in total, the PWG pattern period in the center of each group is 5, indicating the center position of a group, the number of periods of the eight surrounding PWG patterns is 3 or 4, representing 1 and 0 in the LBP code respectively, the code formed by the eight PWG patterns in counterclockwise order is the rotation-invariant LBP code, and then the PWG pattern generated by combining the PWG code and the rotation-invariant LBP code is displayed on the LCD screen as a target, in which X w and Y w Axis is coplanar with target, Z w The axis is perpendicular to the target surface, so the Z of all feature points on the target w The coordinates are all 0; then the camera is calibrated according to the target. During the calibration process, the PWG pattern on the target is first photographed by a single camera. When the single camera captures the complete PWG image that constitutes the LBP code, the sorting of all feature points is completed without obtaining the entire target image. Finally, the multi-step phase shift method is used to calibrate the camera on the complete PWG image to obtain the phase information of the PWG image.

[0015] In the step (2), the steps of establishing the binocular imaging model are as follows: firstly, a linear mathematical model of imaging of a point in space on a single camera is established, that is, a single camera imaging model, see formula (1):

[0016]

[0017] In formula (1), s = Z c represents the scale factor; p = [uv] represents the coordinates of a point in space in the pixel coordinate system of the camera, where u represents the positive direction of the x-axis in the pixel coordinate system, and v represents the negative direction of the y-axis in the pixel coordinate system; dx and dy represent the physical size of the pixel in the x and y directions, respectively, χ is the tilt factor, (u0, v0) is the intersection of the optical axis and the image plane; f is the focal length of the camera; is the rotation matrix, t=[t X t Y t Z ] T is the translation vector; P = [X W ,Y W ,Z W ] T is the coordinate of a point in space in the world coordinate system; A is called the intrinsic parameter matrix of the camera, and its parameters depend only on the inherent characteristics of the camera lens group, which is composed of (u0, v0), χ, and the focal length in the u and v directions (f u ,f v ); E is called the external parameter matrix, which represents the rigid motion of the camera coordinate system relative to the world coordinate system; M = AE is the projection matrix;

[0018] Then, according to the rule that the world coordinate system is established on the left camera coordinate system in the stereo imaging system, the binocular imaging model is shown in formula (2):

[0019]

[0020] In formula (2), R l , R r Respectively represent the rotation matrix of the left camera and the right camera, t l , t r Respectively represent the offset vectors of the left camera and the right camera, both obtained through single camera calibration; R rl and t rl They represent the rotation matrix and offset vector between the left camera and the right camera respectively, which together construct the spatial triangular relationship of the stereo system.

[0021] In the step (3), the wrapped phase of the periodic cross-stripe target in the periodic cross-stripe target image is solved by Fourier transform, and then the phase is unwrapped according to the mark points on the periodic cross-stripe target to obtain the absolute phase, and finally the training data set of the BP neural network is obtained by pixel matching according to the corresponding relationship between the absolute phase before and after refraction. The specific process is as follows:

[0022] a. First, generate a periodic cross-fringe target, and its intensity is expressed as formula (6):

[0023] I(u,v)=b+k u cos(2πu / p u )+k v cos(2πv / p v ) (6);

[0024] In formula (6), b, k u and k v are three constants defining the target grayscale distribution; p u and p v The fringe periods along the u and v axes are respectively, each fringe period is encoded, and the encoded target is displayed on an LCD screen;

[0025] b. Make the LCD and the transparent plate of the cover parallel and close to each other, and use the single camera to collect the target image on the LCD screen through the transparent plate. The collected target image is defined as image 1. Then remove the cover, and use the single camera to directly collect the target image on the LCD screen. The collected target image is defined as image 2.

[0026] c. Select a region of interest (ROI) in image 1, perform Fourier transform on the image in ROI, use a filter to extract the base frequency in the image spectrum, and then perform inverse Fourier transform to obtain the wrapped phase of image 1 in the u and v directions.

[0027] d. Use the median filter to separate the white code points in the ROI and combine them with the wrapped phase obtained in step (3) to obtain the absolute phase of image 1. and

[0028] e. Repeat steps c and d to process image 2 and obtain the absolute phase Φ of image 2. u With Φ v Therefore, for any point (u g ,v g ), and obtain the corresponding absolute phase value By searching in the solution space, the coordinates of the minimum value of the following equation (7) are obtained, and the matching point (u p ,v p ):

[0029]

[0030] In formula (7), [Φ u (u p ,v p ),Φ v (u p ,v p )] is the matching point (u p ,v p ) absolute phase value; matching point (u p ,v p )The minimum value of the absolute phase value is less than the convergence threshold τ. When the match is successful, linear interpolation is used to further optimize the matching coordinates to the sub-pixel level. The offset corresponding to each pixel point is given by (d u ,d v )=(u g ,v g )-(u p ,v p )get;

[0031] f. Evenly select points in the ROI area and calculate the image coordinates of these points before refraction (u p ,v p ) and offset (d u ,d v ) and use them as the training data set of BP neural network.

[0032] In the step (5), the three-dimensional deformation of the processed image is evaluated by using the digital image correlation method, that is, a clear area in the image is selected as a sub-area, and the correlation degree of the sub-area texture is used as the judgment basis to establish the corresponding relationship between the sub-areas before and after the deformation:

[0033] When all points in a subregion undergo the same displacement, the coordinates of a point in the subregion See formula (8):

[0034]

[0035] In formula (8), Δu and Δv represent the offset of a point in the sub-region relative to the center of the sub-region respectively; the images before and after deformation are called reference image and deformed image respectively, and their corresponding sub-regions are reference sub-region and deformed sub-region respectively. is the center coordinate of the deformation sub-region; ζ u With v They represent the displacement of the sub-area center in the u and v directions during the deformation process respectively; is the center coordinate of the reference sub-area, and the coordinate of a point in the sub-area after displacement is determined by Δu, Δv and P s Function Plus Joint decision, where With P s are called shape function and parameter vector of shape function respectively;

[0036] When the image is rotated, uniformly stretched or sheared, the zero-order shape function of formula (8) is no longer applicable. The first-order shape function or the second-order shape function is used to describe the displacement relationship of the sub-area pixels before and after the deformation. The first-order shape function is shown in formula (9):

[0037]

[0038] The second-order shape function is shown in formula (10):

[0039]

[0040] In formula (9) and formula (10), and represents the first-order displacement gradient of the sub-region, and Then it represents the second-order displacement gradient of the sub-area;

[0041] In addition, in order to evaluate the correlation between the grayscale of the images before and after deformation, the grayscale of the reference image and the deformed image are set to be and The correlation between the reference sub-region and the deformation sub-region is expressed as formula (11):

[0042]

[0043] In formula (11), C(P s ) is called the correlation function, which is a function of the unknown parameters in the shape function and describes the correlation between the signals of the two sub-areas under the current parameters;

[0044] The normalized covariance correlation coefficient is selected to measure the correlation between the reference sub-region and the deformed sub-region, as shown in formula (12):

[0045]

[0046] In formula (12), and are the mean grayscale values ​​of pixels in the reference sub-area and the deformed sub-area respectively; the area size of the reference sub-area before deformation is (2M+1)×(2M+1);

[0047] Compared with 2D-DIC, 3D-DIC not only considers the sub-area matching problem between consecutive images taken by a single camera, but also involves the matching of the same-name points in the same-frame images of the left and right cameras: a point (u l ,v l ) in the right camera image (u r ,v r ) is defined as formula (13):

[0048]

[0049] In formula (13), (u l ,v l ) and (u r ,v r ) represent the coordinates of a certain point in the left camera and the right camera respectively; α1 to α8 are 8 unknown quantities; in the initialization process, the AKAZE feature matching algorithm is first used to find the matching relationship between several points in the left image collected by the left camera and the right image collected by the right camera. These corresponding relationships constitute a least squares problem. The SVD decomposition is used to obtain the initial guess of these 8 parameters, and then the multidimensional simplex method is used to optimize these 8 parameters. The objective function is set to the sum of the differences in the grayscale of the feature points in the left image and the corresponding feature points in the right image under the current projection relationship. The optimal solution of the parameters is obtained when the objective function obtains the minimum value, thereby obtaining the projection mapping relationship of the left and right sub-areas; in order to reduce the amount of calculation, the left camera and the right camera are only matched in the first frame of the image, and then the 2D-DIC algorithm is used to match them in the left camera and the right camera respectively, so as to evaluate the deformation of the soft object under three-dimensional deformation.

[0050] In the step (5), the specific steps of performing three-dimensional reconstruction on each feature point on the image of the soft object to be measured are:

[0051] First, based on the single-camera imaging model, the projection relationship from the left camera coordinate system to the two-dimensional image plane is derived, as shown in equation (14);

[0052]

[0053] In formula (14), s l is the left camera scale factor; (u l ,v l ) is the pixel coordinate of the left camera is the intrinsic focal length of the left camera Tilt factor χ l , principal point coordinates The left camera intrinsic parameter matrix is ​​constructed; is the coordinate in the left camera coordinate system, express The matrix after increasing the dimension;

[0054] Since there is the following conversion relationship between the coordinate systems of the left camera and the right camera, see formula (15):

[0055]

[0056] In formula (15), is the coordinate in the left camera coordinate system; is the coordinate in the right camera coordinate system; R rl and t rl They represent the rotation matrix and offset vector between the left camera and the right camera respectively;

[0057] According to the relationship in formula (15), the projection of the right camera coordinate system to the right image plane is obtained, and its form is shown in formula (16):

[0058]

[0059] Combine equation (14) with equation (16) and eliminate s l With s r We can get equation (17):

[0060]

[0061] Formula (17) constitutes the unknown quantity In an ideal situation, the two rays will intersect at the real space object point, but due to the influence of image noise, stereo matching error and calibration error, it is difficult for the two rays to intersect. However, since equation (17) is an overdetermined equation system with four independent equations, the optimal solution that meets the equation is obtained by the least squares method.

[0062] In formula (17), (u l ,v l ) and (u r ,v r ) represent the sub-area centers in the left and right images respectively. They are obtained as known quantities by the 3D-DIC algorithm, and other parameters are obtained by camera calibration. l ,v l ) and (u r ,v r ), and the matrix obtained by camera calibration and All the parameters in the image are used to reconstruct each feature point in the image of the soft object to be measured in three dimensions, and then the deformation of the surface of the soft object to be measured is analyzed according to the connection between the left camera sub-area and the right camera sub-area and their respective deformation sub-areas.

[0063] Advantages of the present invention:

[0064] (1) The present invention uses a method combining multi-step phase shift and LBP coding to collect the PWG image displayed on the LCD screen and perform camera calibration. The multi-step phase shift method is used to solve the problem of feature point extraction accuracy when the camera is defocused or the imaging quality is poor, and LBP coding is used to solve the problem of feature point sorting when the camera cannot capture the complete target;

[0065] (2) The present invention uses a BP neural network to establish a nonlinear mapping between the pixel position and pixel offset of the transparent plate of the cover before and after refraction, and corrects the refracted image by bilinear interpolation;

[0066] (3) The present invention can evaluate the deformation of all feature points in a certain area on the surface of a soft object under non-contact conditions. The flexible fingertip deformation measurement system has a simple structure and has the advantage of being quickly transplanted to other systems for use as a deformation measurement sensor.

[0067] (4) The present invention uses digital image correlation (DIC) technology to analyze the deformation of the surface of the soft object to be measured. The measurement results intuitively and sensitively reveal the mechanical behavior of the material of the soft object to be measured and effectively reflect the three-dimensional deformation information of the soft object to be measured. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] Figure 1 It is a front view of the flexible fingertip deformation measurement system of the present invention.

[0069] Figure 2 It is a cross-sectional view of the flexible fingertip deformation measurement system of the present invention.

[0070] Figure 3 It is a structural schematic diagram of the camera calibration of the present invention.

[0071] Figure 4 These are PWG images of different periods and their corresponding wrapped phase diagrams of the present invention.

[0072] Figure 5 It is a flow chart of solving the absolute phase of the present invention.

[0073] Figure 6 It is a deformation image and deformation sub-area map of the DIC technology of the present invention.

[0074] Figure numerals: 1-base, 2-cover, 21-shading plate, 22-transparent plate, 3-camera module, 4-FPGA development board, 5-LCD screen, 6-PWG pattern, 7-single camera, 8-complete PWG image. DETAILED DESCRIPTION

[0075] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0076] A method for measuring deformation of a flexible fingertip based on DIC comprises the following steps:

[0077] (1) Establish a flexible fingertip deformation measurement system: see Figure 1 and Figure 2 The flexible fingertip deformation measurement system includes a base 1, two camera modules 3 arranged on the base 1, a cover 2 covered outside the two camera modules, an FPGA development board 4 arranged on the base 1 and an external processor, the top plate of the cover 2 and two parallel side plates are both shading plates 21, and the other two parallel side plates of the cover 2 are transparent plates 22, the two camera modules face the two transparent plates 22 respectively, each camera module 3 includes a left camera and a right camera, the two camera modules 3 are arranged oppositely, the lenses of the left camera and the right camera of each camera module 3 are both upwardly placed, with an elevation angle of 75 degrees, each camera module 3 faces the opposite transparent plate, an FPGA development board is arranged on the base 1, the two camera modules 3 are both connected to the FPGA development board 4, and the FPGA development board 4 is communicatively connected to the external processor;

[0078] (2) Camera model establishment: First, the upper left corner of the LCD screen is used as the origin O, the X-axis is established along the axis of the short side of the LCD screen, the Y-axis is established along the axis of the long side of the LCD screen, and the horizontal axis perpendicular to the LCD screen is used as the Z-axis, thereby establishing the world coordinate system O-XYZ. Then, a method combining multi-step phase shift and LBP encoding is used to collect the PWG image displayed on the LCD screen and perform camera calibration. Finally, a binocular imaging model is established based on the calibration results of each group of camera modules.

[0079] See Figure 3 The specific steps of using the method of combining multi-step phase shift and LBP coding to collect the PWG image displayed on the LCD screen and perform camera calibration are as follows: First, a PWG (planar waveguide grating) array pattern is established, and the period number set and the maximum radius of the PWG array pattern are set to T = [3, 4, 5] and r respectively. max =80 pixels, the spacing between adjacent feature points is set to 180 pixels, the PWG array pattern is a group of 3×3, there are 12 groups in total, the PWG pattern period in the center of each group is 5, indicating the center position of a group, the number of periods of the eight surrounding PWG patterns is 3 or 4, representing 1 and 0 in the LBP code respectively, the code formed by combining the eight PWG patterns in counterclockwise order is the rotation-invariant LBP code, and then the PWG pattern 6 generated by combining the PWG code and the rotation-invariant LBP code is displayed on the LCD screen 5 and used as a target, in which X w and Y w Axis is coplanar with target, Z w The axis is perpendicular to the target surface, so the Z of all feature points on the target w The coordinates are all 0; then the camera is calibrated according to the target. During the calibration process, the single camera 7 first captures the PWG pattern on the target. When the single camera captures the complete PWG image 8 constituting the LBP code, the sorting of all feature points is completed without obtaining the entire target image. Finally, the multi-step phase shift method is used to calibrate the camera on the complete PWG image 8 to obtain the phase information of the PWG image.

[0080] Taking the three-step phase shift in the multi-step phase shift method as an example, the intensity of a PWG with a label k is expressed by the following equations (3)-(5):

[0081]

[0082] φ(x k ,y k )=θ(x k ,y k )×2π / T (4);

[0083]

[0084] In formulas (3)-(5), They represent the three phase intensities of the PWG image in the three-step phase shift method; a1 and a2 are constants for adjusting the pattern intensity; φ(x k ,y k ) is called the wrapping phase; T represents the number of cycles of PWG, which can be 3, 4, or 5 depending on its own encoding; θ(x k ,y k ) is the point (x k ,y k ) with the positive x-axis; PWGs of different periods and their corresponding wrapping phases are shown in Figure 4 As shown; Figure 4 middle, Represents a point to its center The Euclidean distance of max Represents r k The maximum value of is used to limit the maximum radius of the PWG;

[0085] The steps of establishing the binocular imaging model are as follows: first, a linear mathematical model of the imaging of a point in space on a single camera is established, that is, a single camera imaging model, see formula (1):

[0086]

[0087] In formula (1), s = Z c represents the scale factor; p = [uv] represents the coordinates of a point in space in the pixel coordinate system of the camera, where u represents the positive direction of the x-axis in the pixel coordinate system, and v represents the negative direction of the y-axis in the pixel coordinate system; dx and dy represent the physical size of the pixel in the x and y directions, respectively, χ is the tilt factor, (u0, v0) is the intersection of the optical axis and the image plane; f is the focal length of the camera; is the rotation matrix, t=[t X t Y t Z ] T is the translation vector; P = [X W ,Y W ,Z W ] T is the coordinate of a point in space in the world coordinate system; A is called the intrinsic parameter matrix of the camera, and its parameters depend only on the inherent characteristics of the camera lens group, which is composed of (u0, v0), χ, and the focal length in the u and v directions (f u ,f v ); E is called the external parameter matrix, which represents the rigid motion of the camera coordinate system relative to the world coordinate system; M = AE is the projection matrix;

[0088] Then, according to the rule that the world coordinate system is established on the left camera coordinate system in the stereo imaging system, the binocular imaging model is shown in formula (2):

[0089]

[0090] In formula (2), R l , R r Respectively represent the rotation matrix of the left camera and the right camera, t l , t r Respectively represent the offset vectors of the left camera and the right camera, both obtained through single camera calibration; R rl and t rl They represent the rotation matrix and offset vector between the left camera and the right camera respectively, which together construct the spatial triangular relationship of the stereo system;

[0091] (3) Glass refraction correction based on neural network: First, two groups of camera modules in the housing respectively collect images of periodic cross-stripe targets outside the housing and arranged parallel to the transparent plate. Then, Fourier transform is used to solve the wrapped phase of the periodic cross-stripe target in the periodic cross-stripe target image. Then, the phase is unwrapped according to the mark points on the periodic cross-stripe target to obtain the absolute phase. Finally, the training data set of the BP neural network is obtained by pixel matching based on the corresponding relationship between the absolute phases before and after refraction. The training data set is substituted into the BP neural network for training. The training result is the glass refraction correction model.

[0092] See Figure 5 , the Fourier transform is used to solve the wrapped phase of the periodic cross-fringe target in the periodic cross-fringe target image, and then the phase is unwrapped according to the mark points on the periodic cross-fringe target to obtain the absolute phase. Finally, the training data set of the BP neural network is obtained by pixel matching based on the correspondence between the absolute phases before and after refraction. The specific process is as follows:

[0093] a. First, generate a periodic cross-fringe target, and its intensity is expressed as formula (6):

[0094] I(u,v)=b+k u cos(2πu / p u )+k v cos(2πv / p v ) (6);

[0095] In formula (6), b, k u and k v are three constants defining the target grayscale distribution; p u and p v The fringe periods along the u and v axes are respectively, each fringe period is encoded, and the encoded target is displayed on an LCD screen;

[0096] b. Make the LCD and the transparent plate of the cover parallel and close to each other, and use the single camera to collect the target image on the LCD screen through the transparent plate. The collected target image is defined as image 1. Then remove the cover, and use the single camera to directly collect the target image on the LCD screen. The collected target image is defined as image 2.

[0097] c. Select a region of interest (ROI) in image 1, perform Fourier transform on the image in ROI, use a suitable filter such as Hanning window to extract the base frequency in the image spectrum, and then perform inverse Fourier transform to obtain the wrapped phase of image 1 in the u and v directions.

[0098] d. Use the median filter to separate the white code points in the ROI and combine them with the wrapped phase obtained in step (3) to obtain the absolute phase of image 1. and

[0099] e. Repeat steps c and d to process image 2 and obtain the absolute phase Φ of image 2. u With Φ v Therefore, for any point (u g ,v g ), and obtain the corresponding absolute phase value By searching in the solution space, the coordinates of the minimum value of the following equation (7) are obtained, and the matching point (u p ,v p ):

[0100]

[0101] In formula (7), [Φ u (u p ,v p ),Φ v (u p ,v p )] is the matching point (u p ,v p ). The absolute phase value of the matching point (u p ,v p )The minimum value of the absolute phase value is less than the convergence threshold τ. When the match is successful, linear interpolation is used to further optimize the matching coordinates to the sub-pixel level. The offset corresponding to each pixel point is given by (d u ,d v )=(u g ,v g )-(u p ,v p )get;

[0102] f. Evenly select points in the ROI area and calculate the image coordinates of these points before refraction (u p ,v p ) and offset (d u ,d v ) and use them as the training data set of BP neural network.

[0103] (4) Image data acquisition: The soft object to be measured is pressed against the transparent plate of the cover with fingertips. During the pressing process, the camera module is used to acquire images, and the acquired images are sent to the processor. The processor brings the acquired images into the binocular imaging model and the glass refraction correction model for image processing to obtain the processed images;

[0104] (5) Deformation measurement: The digital image correlation method (DIC) is used to evaluate the three-dimensional deformation of the processed image. After three-dimensional reconstruction of each feature point on the image of the soft object to be measured, the continuous three-dimensional deformation of the surface of the soft object to be measured in space is obtained based on the inter-frame connection of the same sub-area.

[0105] The digital image correlation method is used to evaluate the three-dimensional deformation of the processed image, that is, a clear area in the image is selected as a sub-area, and the correlation degree of the sub-area texture is used as the basis for judgment to establish the corresponding relationship between the sub-areas before and after deformation:

[0106] When all points in a subregion undergo the same displacement, the coordinates of a point in the subregion See formula (8):

[0107]

[0108] In formula (8), Δu and Δv represent the offset of a point in the sub-region relative to the center of the sub-region; the images before and after deformation are called reference image and deformed image, respectively, and their corresponding sub-regions are reference sub-region and deformed sub-region (see Figure 6 ), is the center coordinate of the deformation sub-region; ζ u With v They represent the displacement of the sub-area center in the u and v directions during the deformation process respectively; is the center coordinate of the reference sub-area, and the coordinate of a point in the sub-area after displacement is determined by Δu, Δv and P s Function Plus Joint decision, where With P s are called shape function and parameter vector of shape function respectively;

[0109] When the image is rotated, uniformly stretched or sheared, the zero-order shape function of formula (8) is no longer applicable. The first-order shape function or the second-order shape function is used to describe the displacement relationship of the sub-area pixels before and after the deformation. The first-order shape function is shown in formula (9):

[0110]

[0111] The second-order shape function is shown in formula (10):

[0112]

[0113] In formula (9) and formula (10), and represents the first-order displacement gradient of the sub-region, and Then it represents the second-order displacement gradient of the sub-area;

[0114] In addition, in order to evaluate the correlation between the grayscale of the images before and after deformation, the grayscale of the reference image and the deformed image are set to be and The correlation between the reference sub-region and the deformation sub-region is expressed as formula (11):

[0115]

[0116] In formula (11), C(P s ) is called the correlation function, which is a function of the unknown parameters in the shape function and describes the correlation between the signals of the two sub-areas under the current parameters;

[0117] The normalized covariance correlation coefficient is selected to measure the correlation between the reference sub-region and the deformed sub-region, as shown in formula (12):

[0118]

[0119] In formula (12), and are the mean grayscale values ​​of pixels in the reference sub-area and the deformed sub-area respectively; the area size of the reference sub-area before deformation is (2M+1)×(2M+1) (see Figure 6 );

[0120] Compared with 2D-DIC, 3D-DIC not only considers the sub-area matching problem between consecutive images taken by a single camera, but also involves the matching of the same-name points in the same-frame images of the left and right cameras: a point (u l ,v l ) in the right camera image (u r ,v r ) is defined as formula (13):

[0121]

[0122] In formula (13), (u l ,v l ) and (u r ,v r ) respectively represent the coordinates of a certain point in the left camera and the right camera; α1 to α8 are 8 unknown quantities; in the initialization process, the AKAZE feature matching algorithm is first used to find the matching relationship between several points in the left image collected by the left camera and the right image collected by the right camera. These corresponding relationships constitute a least squares problem. The SVD decomposition is used to obtain the initial guess of these 8 parameters, and then the multidimensional simplex method is used to optimize these 8 parameters. The objective function is set to the sum of the differences in the grayscale of the feature points of the left image and the corresponding feature points of the right image under the current projection relationship. The optimal solution of the parameters is obtained when the objective function obtains the minimum value, thereby obtaining the projection mapping relationship of the left and right sub-areas; in order to reduce the amount of calculation, the left camera and the right camera are only matched in the first frame of the image, and then the 2D-DIC algorithm is used to match them in the left camera and the right camera respectively, so as to evaluate the deformation of the soft object under three-dimensional deformation;

[0123] The specific steps for 3D reconstruction of each feature point on the image of the soft object are as follows:

[0124] First, based on the single-camera imaging model, the projection relationship from the left camera coordinate system to the two-dimensional image plane is derived, as shown in equation (14);

[0125]

[0126] In formula (14), s l is the left camera scale factor; (u l ,v l ) is the pixel coordinate of the left camera is the intrinsic focal length of the left camera Tilt factor χ l , principal point coordinates The left camera intrinsic parameter matrix is ​​constructed; is the coordinate in the left camera coordinate system, express The matrix after increasing the dimension (the first three rows correspond to the parameters, and the last row is all 0, the same for the right camera);

[0127] Since there is the following conversion relationship between the coordinate systems of the left camera and the right camera, see formula (15):

[0128]

[0129] In formula (15), is the coordinate in the left camera coordinate system; is the coordinate in the right camera coordinate system; R rl and t rl They represent the rotation matrix and offset vector between the left camera and the right camera respectively;

[0130] According to the relationship in formula (15), the projection of the right camera coordinate system to the right image plane is obtained, and its form is shown in formula (16):

[0131]

[0132] Combine equation (14) with equation (16) and eliminate s l With s r We can get equation (17):

[0133]

[0134] Formula (17) constitutes the unknown quantity In an ideal situation, the two rays will intersect at the real space object point, but due to the influence of image noise, stereo matching error and calibration error, it is difficult for the two rays to intersect. However, since equation (17) is an overdetermined equation system with four independent equations, the optimal solution that meets the equation is obtained by the least squares method.

[0135] In formula (17), (u l ,v l ) and (u r ,v r ) represent the sub-area centers in the left and right images respectively. They are obtained as known quantities by the 3D-DIC algorithm, and other parameters are obtained by camera calibration. l ,v l ) and (u r ,v r ), and the matrix obtained by camera calibration and All the parameters in the image are used to reconstruct each feature point in the image of the soft object to be measured in three dimensions, and then the deformation of the surface of the soft object to be measured is analyzed according to the connection between the left camera sub-area and the right camera sub-area and their respective deformation sub-areas.

[0136] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A flexible fingertip deformation measurement method based on DIC, characterized in that: The following steps are included: (1) Establishing a flexible fingertip deformation measurement system: The flexible fingertip deformation measurement system includes a base, two camera modules arranged on the base, a cover shell arranged outside the two camera modules, and a processor. The top plate and two parallel side plates of the cover shell are both shading plates, and the other two parallel side plates of the cover shell are transparent plates. The two camera modules face the two transparent plates respectively. Each camera module includes a left camera and a right camera. The two camera modules are connected to the processor. (2) Camera model establishment: First, the upper left corner of the LCD screen is used as the origin O, the X-axis is established along the axis of the short side of the LCD screen, the Y-axis is established along the axis of the long side of the LCD screen, and the horizontal axis perpendicular to the LCD screen is used as the Z-axis, thereby establishing the world coordinate system O-XYZ. Then, a method combining multi-step phase shift and LBP encoding is used to collect the PWG image displayed on the LCD screen and perform camera calibration. Finally, a binocular imaging model is established based on the calibration results of each group of camera modules. (3) Glass refraction correction based on neural network: First, two groups of camera modules in the housing respectively collect images of periodic cross-stripe targets outside the housing and arranged parallel to the transparent plate. Then, Fourier transform is used to solve the wrapped phase of the periodic cross-stripe target in the periodic cross-stripe target image. Then, the phase is unwrapped according to the mark points on the periodic cross-stripe target to obtain the absolute phase. Finally, the training data set of the BP neural network is obtained by pixel matching based on the corresponding relationship between the absolute phases before and after refraction. The training data set is substituted into the BP neural network for training. The training result is the glass refraction correction model. (4) Image data acquisition: The soft object to be measured is pressed against the transparent plate of the cover with fingertips. During the pressing process, the camera module is used to acquire images, and the acquired images are sent to the processor. The processor brings the acquired images into the binocular imaging model and the glass refraction correction model for image processing to obtain the processed images; (5) Deformation measurement: The digital image correlation method (DIC) is used to evaluate the three-dimensional deformation of the processed image. After three-dimensional reconstruction of each feature point on the image of the soft object to be measured, the continuous three-dimensional deformation of the surface of the soft object to be measured in space is obtained based on the inter-frame connection of the same sub-area.

2. The method for measuring deformation of a flexible fingertip based on DIC according to claim 1, characterized in that: The two groups of camera modules are arranged opposite to each other, and the lenses of the left camera and the right camera of each group of camera modules are both placed upward, with an elevation angle of 70-80 degrees, and each group of camera modules faces the opposite transparent plate.

3. The method for measuring deformation of a flexible fingertip based on DIC according to claim 1, characterized in that: An FPGA development board is arranged on the base, and both camera modules are connected to the FPGA development board, and the FPGA development board is connected to the external processor for communication.

4. The method for measuring deformation of a flexible fingertip based on DIC according to claim 1, characterized in that: In the step (2), the specific steps of using a method combining multi-step phase shift and LBP coding to collect the PWG image displayed on the LCD screen and perform camera calibration are as follows: first, a PWG array pattern is established, and the period number set and the maximum radius of the PWG array pattern are set to T = [3, 4, 5] and r respectively. max =80 pixels, the spacing between adjacent feature points is set to 180 pixels, the PWG array pattern is a group of 3×3, there are 12 groups in total, the PWG pattern period in the center of each group is 5, indicating the center position of a group, the number of periods of the eight surrounding PWG patterns is 3 or 4, representing 1 and 0 in the LBP code respectively, the code formed by the eight PWG patterns in counterclockwise order is the rotation-invariant LBP code, and then the PWG pattern generated by combining the PWG code and the rotation-invariant LBP code is displayed on the LCD screen as a target, in which X w and Y w Axis is coplanar with target, Z w The axis is perpendicular to the target surface, so the Z of all feature points on the target w The coordinates are all 0; then the camera is calibrated according to the target. During the calibration process, the PWG pattern on the target is first photographed by a single camera. When the single camera captures the complete PWG image that constitutes the LBP code, the sorting of all feature points is completed without obtaining the entire target image. Finally, the multi-step phase shift method is used to calibrate the camera on the complete PWG image to obtain the phase information of the PWG image.

5. The method for measuring deformation of a flexible fingertip based on DIC according to claim 1, characterized in that: In the step (2), the steps of establishing the binocular imaging model are as follows: firstly, a linear mathematical model of imaging of a point in space on a single camera is established, that is, a single camera imaging model, see formula (1): In formula (1), s = Z c represents the scale factor; p = [uv] represents the coordinates of a point in space in the pixel coordinate system of the camera, where u represents the positive direction of the x-axis in the pixel coordinate system, and v represents the negative direction of the y-axis in the pixel coordinate system; dx and dy represent the physical size of the pixel in the x and y directions, respectively, χ is the tilt factor, (u0, v0) is the intersection of the optical axis and the image plane; f is the focal length of the camera; is the rotation matrix, t=[t X t Y t Z ] T is the translation vector; P = [X W ,Y W ,Z W ] T is the coordinate of a point in space in the world coordinate system; A is called the intrinsic parameter matrix of the camera, and its parameters depend only on the inherent characteristics of the camera lens group, which is composed of (u0, v0), χ, and the focal length in the u and v directions (f u ,f v ); E is called the external parameter matrix, which represents the rigid motion of the camera coordinate system relative to the world coordinate system; M = AE is the projection matrix; Then, according to the rule that the world coordinate system is established on the left camera coordinate system in the stereo imaging system, the binocular imaging model is shown in formula (2): In formula (2), R l , R r Respectively represent the rotation matrix of the left camera and the right camera, t l , t r Respectively represent the offset vectors of the left camera and the right camera, both obtained through single camera calibration; R rl and t rl They represent the rotation matrix and offset vector between the left camera and the right camera respectively, which together construct the spatial triangular relationship of the stereo system.

6. The method for measuring deformation of a flexible fingertip based on DIC according to claim 1, characterized in that: In the step (3), the wrapped phase of the periodic cross-stripe target in the periodic cross-stripe target image is solved by Fourier transform, and then the phase is unwrapped according to the mark points on the periodic cross-stripe target to obtain the absolute phase, and finally the training data set of the BP neural network is obtained by pixel matching according to the corresponding relationship between the absolute phase before and after refraction. The specific process is as follows: a. First, generate a periodic cross-fringe target, and its intensity is expressed as formula (6): I(u,v)=b+k u cos(2πu / p u )+k v cos(2πv / p v ) (6); In formula (6), b, k u and k v are three constants defining the target grayscale distribution; p u and p v The fringe periods along the u and v axes are respectively, each fringe period is encoded, and the encoded target is displayed on an LCD screen; b. Make the LCD and the transparent plate of the cover parallel and close to each other, and use the single camera to collect the target image on the LCD screen through the transparent plate. The collected target image is defined as image 1. Then remove the cover, and use the single camera to directly collect the target image on the LCD screen. The collected target image is defined as image 2. c. Select a region of interest (ROI) in image 1, perform Fourier transform on the image in ROI, use a filter to extract the base frequency in the image spectrum, and then perform inverse Fourier transform to obtain the wrapped phase of image 1 in the u and v directions. d. Use the median filter to separate the white code points in the ROI and combine them with the wrapped phase obtained in step (3) to obtain the absolute phase of image 1. and e. Repeat steps c and d to process image 2 and obtain the absolute phase Φ of image 2. u With Φ v Therefore, for any point (u g ,v g ), and obtain the corresponding absolute phase value By searching in the solution space, the coordinates of the minimum value of the following equation (7) are obtained, and the matching point (u p ,v p ): In formula (7), [Φ u (u p ,v p ),Φ v (u p ,v p )] is the matching point (u p ,v p ) absolute phase value; matching point (u p ,v p )The minimum value of the absolute phase value is less than the convergence threshold τ. When the match is successful, linear interpolation is used to further optimize the matching coordinates to the sub-pixel level. The offset corresponding to each pixel point is given by (d u ,d v )=(u g ,v g )-(u p ,v p )get; f. Evenly select points in the ROI area and calculate the image coordinates of these points before refraction (u p ,v p ) and offset (d u ,d v ) and use them as the training data set of BP neural network.

7. The method for measuring deformation of a flexible fingertip based on DIC according to claim 1, characterized in that: In the step (5), the three-dimensional deformation of the processed image is evaluated by using the digital image correlation method, that is, a clear area in the image is selected as a sub-area, and the correlation degree of the sub-area texture is used as the judgment basis to establish the corresponding relationship between the sub-areas before and after the deformation: When all points in a subregion undergo the same displacement, the coordinates of a point in the subregion See formula (8): In formula (8), Δu and Δv represent the offset of a point in the sub-region relative to the center of the sub-region respectively; the images before and after deformation are called reference image and deformed image respectively, and their corresponding sub-regions are reference sub-region and deformed sub-region respectively. is the center coordinate of the deformation sub-region; ζ u With v They represent the displacement of the sub-area center in the u and v directions during the deformation process respectively; is the center coordinate of the reference sub-area, and the coordinate of a point in the sub-area after displacement is determined by Δu, Δv and P s Function Plus Joint decision, where With P s are called shape function and parameter vector of shape function respectively; When the image is rotated, uniformly stretched or sheared, the zero-order shape function of formula (8) is no longer applicable. The first-order shape function or the second-order shape function is used to describe the displacement relationship of the sub-area pixels before and after the deformation. The first-order shape function is shown in formula (9): The second-order shape function is shown in formula (10): In formula (9) and formula (10), and represents the first-order displacement gradient of the sub-region, and Then it represents the second-order displacement gradient of the sub-area; In addition, in order to evaluate the correlation between the grayscale of the images before and after deformation, the grayscale of the reference image and the deformed image are set to be and The correlation between the reference sub-region and the deformation sub-region is expressed as formula (11): In formula (11), C(P s ) is called the correlation function, which is a function of the unknown parameters in the shape function and describes the correlation between the signals of the two sub-areas under the current parameters; The normalized covariance correlation coefficient is selected to measure the correlation between the reference sub-region and the deformed sub-region, as shown in formula (12): In formula (12), and are the mean grayscale values ​​of pixels in the reference sub-area and the deformed sub-area respectively; The area size of the reference sub-area before deformation is (2M+1)×(2M+1); Compared with 2D-DIC, 3D-DIC not only considers the sub-area matching problem between consecutive images taken by a single camera, but also involves the matching of the same-name points in the same-frame images of the left and right cameras: a point (u l ,v l ) in the right camera image (u r ,v r ) is defined as formula (13): In formula (13), (u l ,v l ) and (u r ,v r ) represent the coordinates of a certain point in the left camera and the right camera respectively; α1 to α8 are 8 unknown quantities; in the initialization process, the AKAZE feature matching algorithm is first used to find the matching relationship between several points in the left image collected by the left camera and the right image collected by the right camera. These corresponding relationships constitute a least squares problem. The SVD decomposition is used to obtain the initial guess of these 8 parameters, and then the multidimensional simplex method is used to optimize these 8 parameters. The objective function is set to the sum of the differences in the grayscale of the feature points in the left image and the corresponding feature points in the right image under the current projection relationship. The optimal solution of the parameters is obtained when the objective function obtains the minimum value, thereby obtaining the projection mapping relationship of the left and right sub-areas; in order to reduce the amount of calculation, the left camera and the right camera are only matched in the first frame of the image, and then the 2D-DIC algorithm is used to match them in the left camera and the right camera respectively, so as to evaluate the deformation of the soft object under three-dimensional deformation.

8. The method for measuring deformation of a flexible fingertip based on DIC according to claim 7, characterized in that: In the step (5), the specific steps of performing three-dimensional reconstruction on each feature point on the image of the soft object to be measured are: First, based on the single-camera imaging model, the projection relationship from the left camera coordinate system to the two-dimensional image plane is derived, as shown in equation (14); In formula (14), s l is the left camera scale factor; (u l ,v l ) is the pixel coordinate of the left camera is the intrinsic focal length of the left camera Tilt factor χ l , principal point coordinates The left camera intrinsic parameter matrix is ​​constructed; is the coordinate in the left camera coordinate system, express The matrix after increasing the dimension; Since there is the following conversion relationship between the coordinate systems of the left camera and the right camera, see formula (15): In formula (15), is the coordinate in the left camera coordinate system; is the coordinate in the right camera coordinate system; R rl and t rl They represent the rotation matrix and offset vector between the left camera and the right camera respectively; According to the relationship in formula (15), the projection of the right camera coordinate system to the right image plane is obtained, and its form is shown in formula (16): Combine equation (14) with equation (16) and eliminate s l With s r We can get equation (17): Formula (17) constitutes the unknown quantity In an ideal situation, the two rays will intersect at the real space object point, but due to the influence of image noise, stereo matching error and calibration error, it is difficult for the two rays to intersect. However, since equation (17) is an overdetermined equation system with four independent equations, the optimal solution that meets the equation is obtained by the least squares method. In formula (17), (u l ,v l ) and (u r ,v r ) represent the sub-area centers in the left and right images respectively. They are obtained as known quantities by the 3D-DIC algorithm, and other parameters are obtained by camera calibration. l ,v l ) and (u r ,v r ), and the matrix obtained by camera calibration and All the parameters in the image are used to reconstruct each feature point in the image of the soft object to be measured in three dimensions, and then the deformation of the surface of the soft object to be measured is analyzed according to the connection between the left camera sub-area and the right camera sub-area and their respective deformation sub-areas.

Images