A 3D video extensometer with a single CCD camera

Abstract

The present invention discloses a 3D video extensometer with a single CCD camera, comprising a camera, a lens, a blue light source, an electrical control terminal and a computer, wherein the electrical control terminal is connected to the camera and the computer; the computer is configured with a parameter calibration unit, a feature marking unit and a three-dimensional reconstruction unit; the parameter calibration unit is configured to perform camera self-calibration to determine the internal and external parameters of the camera; the feature marking unit is configured to prepare characteristic speckles on the surface of a specimen, and match and track the motion trajectory of the speckles of the specimen through the self-calibrated camera; the three-dimensional reconstruction unit is configured to: based on the principle of the pyramid method, solve the conversion relationship between the camera coordinate system and the world coordinate system, project an image point through the optical center onto the plane of a calibration plate based on the conversion relationship, solve the world coordinate of the projection point by using the geometric relationship between the projection point and the image point, add the calibration plate thickness in the Z direction to the world coordinate of the projection point, obtain the position of the object point on the specimen surface, and complete the conversion from two-dimensional image point strain to three-dimensional object point strain.

Description

Technical Field

[0001] The present invention relates to the technical field of measuring instruments, and in particular to a 3D video extensometer with a CCD single camera. Background Art

[0002] The field of new materials continues to grow rapidly each year. As the key to material testing, extensometers will also usher in higher and more demand. Currently, there are many types of extensometers. According to whether they are in contact with the test piece, they can be divided into two categories: contact extensometers and non-contact extensometers.

[0003] Contact extensometers are an earlier type of extensometer, primarily encompassing mechanical and electronic types. Mechanical extensometers utilize mechanical devices such as gears and levers to amplify deformation data for display. These devices require the specimen to be clamped and fixed, placing certain demands on surface roughness and dimensions. Large specimens or those involving significant deformation are often prioritized. Electronic extensometers, on the other hand, utilize sensors to convert deformation signals into electrical signals, which are then processed by a processor to produce a strain reading. For example, a resistance strain gauge electronic extensometer measures deformation by attaching the extensometer to the specimen. The blades contact the specimen, sensing the elongation within the distance between the two blades. The deformation of the rod induces strain in the elastic element, which is then converted by strain gauges into resistance changes. This is then converted to a voltage signal using an appropriate measurement amplifier circuit. Compared to mechanical extensometers, these strain-sensing systems offer greater flexibility, are much less sensitive to specimen size, and require less complex operation, making them suitable for strain measurement on small and medium-sized specimens. In summary, both of the above two contact extensometers collect and convert strain data by fixing a certain mechanical device on the test piece.

[0004] Contact extensometers have long been the mainstay of material mechanical property testing due to their simple structure, mature technology, and low price. In recent years, with the development of digital image processing technology, the accuracy and computational efficiency of non-contact strain measurement methods have been significantly improved. Among them, video extensometers have become a new means of strain measurement in material mechanics due to their wide range of measurement objects, small size, and diverse application scenarios. The principle is to use a camera to capture an image of the specimen throughout its strain process, then use DIC digital image rapid matching and tracking technology to determine the deformation of characteristic areas in the image, and then use three-dimensional reconstruction technology to confirm the specimen's true physical strain. The video extensometer hardware system primarily consists of a CCD camera, an optical fixed-focus lens, an illumination light source, a computer for image processing, and an electrical controller. The entire test process does not require the connection of mechanical sensors or the fixing of high-precision specimens, and the fracture of the specimen can be measured without damage.

[0005] During use, contact extensometers have exposed the following limitations: 1) It is difficult to securely mount test pieces with smooth surfaces, which can easily cause slippage; 2) The gauge section of the equipment is fixed, which limits the specifications of the test piece; 3) It cannot withstand the rigid impact caused by the fracture of the test piece. Before the test piece breaks, the test piece needs to be removed from the extensometer, so it is impossible to measure the strain data at the moment of fracture, nor to determine the actual elongation of the test piece after fracture; 4) For specific harsh environments (such as high temperature environments), specific high temperature environment extensometer tests need to be customized, and the environment dependence is high; 5) Clamping the test piece surface is equivalent to applying additional weight to the test piece, that is, adding load to the test piece, which is not suitable for test pieces with lower stiffness.

[0006] In comparison, the video extensometer has the following advantages: 1) It does not directly contact the specimen and does not need to be clamped. Specimens with smooth surfaces can also be measured without worrying about measurement errors caused by slipping; 2) There are no strict requirements for the gauge length, and it supports strain measurement from a few millimeters to a few meters. Replacing the lens / adjusting the field of view can meet the strain measurement needs of specimens of different sizes; 3) The strain of complex structures or the elongation between two points can be measured throughout the entire process, and the fracture of the specimen can be measured without destruction, and the measurement of elongation at fracture and after fracture can be handled; 4) Specimen feature marking is fast and convenient, and can be applied to tiny specimens; 5) The overall system is simple and efficient to operate, and can capture images of the test process in real time.

[0007] However, there are still areas that need further improvement in video extensometers: 1) Due to the influence of lens processing technology, lens distortion is inevitable, and the closer to the edge of the lens, the more obvious the impact of distortion on the image; but most video extensometers on the market are calibrated mainly based on two-point calibration of the scale, and no full-field distortion correction is performed; 2) During the entire measurement process, the camera is required to be aligned with the specimen, which is difficult to operate; 3) The real-time performance of DIC digital image fast matching and tracking technology is limited. For specimens, especially some materials with large strain (such as rubber), the state of the specimen mark in the late strain period will undergo large strain, requiring accurate and efficient feature tracking algorithms; 4) The degree of light intensity contrast between the specimen and the marked features directly affects the tracking accuracy of the video extensometer. Due to the diversity of specimen materials and colors, the production of feature marks also requires certain skills and experience; 5) Large camera field of view measurement scenes are basically achieved by replacing the lens of a single camera, and the accuracy is limited.

[0008] Based on the current development of extensometers, it is not difficult to see that although video extensometers have obvious advantages over contact extensometers, they also have many defects that need to be improved. One way to improve some of these defects is to improve the performance of hardware equipment, such as using zoom magnification lenses and more accurate CCDs, but this will inevitably lead to a significant increase in the cost of video extensometers. Another option is to use more accurate matching and detection algorithms, and the quality of the algorithms will undoubtedly become the main factor affecting accuracy.

[0009] When measuring strain using a video extensometer, the core technical issues are:

[0010] 1) Lens distortion causes camera image distortion:

[0011] The imaging principle of a monocular vision system is based on the pinhole imaging principle. Due to process variations such as lens manufacturing errors and installation position, the image will experience radial and tangential distortion. Radial distortion results in different magnifications in different parts of the lens, while tangential distortion causes near objects to appear larger than far objects, and circles to appear elliptical. Due to imperfect optical systems, the actual image plane often does not completely overlap with the photosensitive chip. This resulting distortion is called image plane distortion. Lens distortion causes the actual image to become increasingly distorted towards the edges, while also causing the camera's principal point c to shift and the focal length f to change. This can significantly impact subsequent landmark detection, 3D reconstruction, and feature matching and tracking. Experiments have confirmed that even lenses with minimal lens distortion can have a significant impact on the overall image.

[0012] 2) The traditional DIC matching algorithm has a complex calculation process and lengthy data:

[0013] Video extensometers use a matching algorithm based on feature markers, using points, lines, and regions as matching units for feature extraction, matching, and tracking. Feature matching requires enormous amounts of real-time image data; therefore, while feature marker matching is effective, reaching pixel-level or even sub-pixel resolution, this algorithm relies on correlating two images of the object's surface before and after deformation to determine a unique strain. The nonlinear mapping function makes the solution complex and data intensive, posing a significant challenge to computer processing capabilities.

[0014] The key to feature marking lies in feature preparation. The quality of the feature will directly affect the final results of feature matching and tracking. If there is a gap between the feature marking and the specimen, the final result will not match the actual strain of the specimen. If there is no large contrast between the feature marking and the specimen, the final result cannot accurately reflect the actual strain of the specimen.

[0015] 3) The measurement accuracy of a large field of view based on a single camera is low:

[0016] Traditional vision sensors often provide stable imaging, but their field of view is fixed and cannot be expanded. Multiple cameras are used to capture specimen strain in segments within a wide field of view. However, how to quickly, efficiently, and accurately record all strain states presents significant algorithmic challenges. Common short-focus industrial lenses are approximately 6mm, making strain measurement within a wide field of view while maintaining image quality a technical challenge for current extensometers. Summary of the Invention

[0017] The main purpose of the present invention is to address the above core technical problems and propose a 3D video extensometer with a single CCD camera. By improving the optimization algorithm and customizing the software, the present invention focuses on solving the problems of camera self-calibration, the impact of its own distortion on accuracy, the depth of field problem in the two-dimensional strain solution process, the problem of decreased accuracy of the field of view visual method, the image calculation speed problem and the feature tracking problem.

[0018] To achieve the above object, the present invention proposes the following technical solutions:

[0019] A 3D video extensometer using a single CCD camera comprises: a camera, a lens, a blue light source, an electrical control terminal, and a computer, wherein the electrical control terminal is connected to the camera and the computer; the computer is configured with a parameter calibration unit, a feature marking unit, and a three-dimensional reconstruction unit; the parameter calibration unit is configured to perform camera self-calibration to determine the camera's intrinsic and extrinsic parameters; the feature marking unit is configured to produce characteristic speckles on a specimen surface and match and track the speckle motion trajectory of the specimen using the self-calibrated camera; the three-dimensional reconstruction unit is configured to: solve the transformation relationship between the camera coordinate system and the world coordinate system based on the pyramid method principle, project an image point through the optical center onto a calibration plate plane based on the transformation relationship, solve the world coordinates of the projection point using the geometric positional relationship between the projection point and the image point, add the calibration plate thickness in the Z direction to the world coordinates of the projection point to obtain the position of the object point on the specimen surface, and complete the transformation from two-dimensional image point strain to three-dimensional object point strain.

[0020] Compared with the prior art, the beneficial effect of the technical solution of the present invention is that: the traditional video extensometer lacks calibration of the internal and external parameters of the camera, while the present invention reduces the influence of lens distortion on image accuracy by calibrating the internal and external parameters of the camera, so that the three-dimensional reconstruction process is not constrained by depth of field, Z-direction constraint and image plane displacement; on this basis, the three-dimensional reconstruction of the present invention completes the conversion of two-dimensional image point strain to three-dimensional object point strain, realizes the real strain measurement process of the specimen, and obtains more accurate real strain measurement results. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 This is a user operation flow chart of a 3D video extensometer with a single CCD camera according to an embodiment of the present invention;

[0022] Figure 2 This is a schematic diagram of the operation of a 3D video extensometer with a single CCD camera according to an embodiment of the present invention;

[0023] Figure 3 is a schematic diagram of a camera calibration plate according to an embodiment of the present invention;

[0024] Figure 4 1 is a schematic diagram of a single-camera calibration shooting process according to an embodiment of the present invention;

[0025] Figure 5 2. This is a schematic diagram of camera imaging deviation according to an embodiment of the present invention;

[0026] Figure 6 This is a schematic diagram of the imaging effect of the calibration plate according to an embodiment of the present invention;

[0027] Figure 7 is a schematic diagram of ellipse correction according to an embodiment of the present invention;

[0028] Figure 8 is a flow chart of camera internal parameter calibration according to an embodiment of the present invention;

[0029] Figure 9 1 is a schematic diagram of the principle of the pyramid method according to an embodiment of the present invention;

[0030] Figure 10 This is a basic principle diagram of the digital image correlation method according to an embodiment of the present invention;

[0031] Figure 11 This is a schematic diagram of earthworm patches of various sizes according to an embodiment of the present invention;

[0032] Figure 12 This is a schematic diagram of the effect of earthworm spot treatment according to an embodiment of the present invention;

[0033] Figure 13 This is a schematic diagram of making spots with a paint pen according to an embodiment of the present invention;

[0034] Figure 14 Schematic diagram of non-coding point spot making according to an embodiment of the present invention;

[0035] Figure 15 2. Schematic diagram of paint speckle pattern according to an embodiment of the present invention;

[0036] Figure 16 3D reconstruction using a single camera according to an embodiment of the present invention;

[0037] Figure 17 Schematic diagram of lossless traversal of a moving point trajectory according to an embodiment of the present invention;

[0038] Figure 18 2. It is a schematic diagram of the object point reprojection theory according to an embodiment of the present invention;

[0039] Figure 19 is the calibration result of the camera internal parameters according to the embodiment of the present invention;

[0040] Figure 20 2 is a schematic diagram of a test environment for extrinsic parameter calibration of a camera according to an embodiment of the present invention;

[0041] Figure 21 is the calibration result of the camera extrinsic parameters according to the embodiment of the present invention;

[0042] Figure 22Schematic diagram of the resolution of a level 1 extensometer according to an embodiment of the present invention;

[0043] Figure 23 Schematic diagram of the limit of the system error of the first-level extensometer in the embodiment of the present invention. DETAILED DESCRIPTION

[0044] The present invention will be further described below with reference to the accompanying drawings and specific embodiments.

[0045] An embodiment of the present invention provides a single-CCD camera 3D video extensometer, comprising a camera, a lens, a blue light source, an electrical control terminal, and a computer, the electrical control terminal being connected to the camera and the computer. The camera is connected to the lens to capture images, and the blue light source provides brightness for the test environment. The blue light source can eliminate the influence of other colors of light in the environment, facilitating subsequent image processing. The computer is equipped with a parameter calibration unit, a feature marking unit, and a 3D reconstruction unit. The parameter calibration unit is configured to perform camera self-calibration to determine the camera's intrinsic and extrinsic parameters. The feature marking unit is configured to generate a characteristic speckle pattern on the specimen surface and match and track the speckle pattern's motion trajectory on the specimen using the self-calibrated camera. The three-dimensional reconstruction unit is configured to calculate the transformation relationship between the camera coordinate system and the world coordinate system based on the pyramid method principle, project the image point onto the calibration plate plane through the optical center based on the transformation relationship, calculate the world coordinates of the projected point using the geometric positional relationship between the projected point and the image point, add the calibration plate thickness in the Z direction to the world coordinates of the projected point to obtain the position of the object point on the specimen surface, and complete the conversion of two-dimensional image point strain to three-dimensional object point strain. In some embodiments, to measure strain over a large field of view, the 3D video extensometer can be expanded to multiple cameras. Accordingly, the computer can also be configured with a multi-camera point crossing algorithm unit to achieve three-dimensional reconstruction of strain measured over a large field of view using a multi-camera array.

[0046] In the embodiment of the present invention, the camera adopts a monocular CCD camera (or CCD single camera), equipped with a fixed focus lens and a blue light source. In view of the entire strain measurement process, the actual user operation process of the 3D video extensometer in the embodiment of the present invention is as follows: Figure 1 As shown, it mainly includes two steps: a) system adjustment and b) strain measurement:

[0047] a) System Adjustment: First, start the testing machine, computer, and video extensometer. Use the electrical control terminal to control the testing machine, video extensometer camera, and light source. Adjust the distance between the camera and the testing machine by moving the extensometer system to determine the optimal working distance. Continue using a custom calibration plate with a ten-parameter model to complete camera self-calibration, confirming the camera's internal parameters, lens distortion, and the rotation and translation matrix relationships between the camera's coordinate system and the calibration plate's coordinate system.

[0048] b) Strain Measurement: First, remove the test specimen and prepare the marking features. The testing machine clamps the specimen with the marking features and assembles it close to and parallel to the calibration plate. The strain tensile test is then performed, and a camera records the specimen strain image throughout the test. The true strain displacement of the specimen is determined using DIC (digital image correlation) fast matching with an improved least squares algorithm and single-camera 3D reconstruction technology.

[0049] like Figure 2 The diagram shows the operation of a 3D video extensometer according to an embodiment of the present invention. First, upon receiving a start signal from a tensile machine (i.e., a testing machine) holding a specimen with a characteristic mark, the control terminal (i.e., the electrical control terminal) sends camera trigger and light source start signals to a CCD camera and a blue light source. After the tensile machine is started, the specimen is stretched in both stress directions 1 and 2. Simultaneously, the CCD camera is activated and the light source is turned on, synchronously capturing images of the characteristic mark points on the specimen from the moment of start-up until the specimen finally breaks. The real-time captured specimen images are transmitted to a PC (i.e., a computer) via a USB data port. The strain data of the specimen is analyzed and stored using a software optimization algorithm configured on the computer according to the present invention.

[0050] In order to clearly present the details of the technical solution of the present invention, the configuration and working principle of the parameter calibration unit, the feature marking unit, the three-dimensional reconstruction unit and the multi-camera point crossing algorithm unit are described in detail below.

[0051] 1. Parameter calibration unit

[0052] Embodiments of the present invention propose a ten-parameter calibration model for calibrating camera intrinsic and extrinsic parameters. The calibration plate in the embodiments of the present invention can also be a ruler. The ten-parameter calibration model in the embodiments of the present invention uses established calibration steps to capture a calibration plate image, extract image grayscale regions, perform ellipse fitting, and perform corrections to determine the intrinsic and extrinsic parameters of the current camera system. These intrinsic parameters include lens distortion, the deviation of the principal point c caused by lens distortion, and the focal length f. This parameter self-calibration method significantly reduces the impact of lens distortion on image position accuracy, laying a solid foundation for subsequent feature tracking.

[0053] 1.1 Camera internal parameter calibration

[0054] refer to Figure 8 The process of performing intrinsic parameter calibration by the parameter calibration unit of an embodiment of the present invention includes: landmark point detection, estimation of initial camera intrinsic parameter values, first bundling adjustment, landmark point ellipse correction, second bundling adjustment, photogrammetry acquisition of global points on the calibration plate and convergence, and determination of camera intrinsic parameters. Specifically, the parameter calibration unit is configured to perform camera intrinsic parameter calibration according to the following processes 1) to 6):

[0055] 1) Use a camera to capture an image of a calibration plate with landmarks, and perform landmark detection on the image of the calibration plate;

[0056] 2) Determine the spatial position coordinates of the center of each marker point, and estimate the initial values of the camera's intrinsic parameters based on the image point coordinates and world coordinates of each marker point; the image point coordinates of each marker point can be directly obtained after the camera is finished shooting; here, the world coordinates refer to the point coordinates in the world coordinate system of the calibration plate, that is, "the world coordinate system of the calibration plate is in its own plane"; and the spatial position coordinates generally default to the position of the lower left corner of the first photo taken by the camera as the world coordinate system point. In this embodiment of the present invention, a ten-parameter calibration model is used to estimate the initial values of the camera's intrinsic parameters;

[0057] 3) Use the bundle adjustment method to iterate the camera intrinsic parameters and the calibration plate image extrinsic parameters to make the first bundle adjustment to the estimated initial values of the intrinsic parameters;

[0058] 4) Perform ellipse correction on each mark point on the calibration plate;

[0059] 5) After completing the ellipse calibration of all the marking points on the calibration plate, perform the second bundling adjustment using the same method as the first bundling adjustment;

[0060] 6) After completing the second bundle adjustment, combine photogrammetry to obtain the global points of the calibration plate, converge, and confirm the camera internal parameters.

[0061] The camera internal parameter calibration is mainly aimed at the image distortion problem caused by the lens processing technology, which will affect the 3D reconstruction accuracy of the visual system, especially at the edge of the image. The embodiment of the present invention uses a calibration plate with markers printed on it (such as Figure 3 As shown in the figure, the object is taken as the shooting object and calibrated. The specific shooting process is as follows Figure 4 As shown, Figure 4 (a) to (j) represent ten different specific placement positions of calibration plates and actual working distance requirements (the measurement distance is 280mm), where (a) represents the calibration plate facing the camera vertically, (b) represents the forward tilted camera system (calibration plate tilted forward), (c) represents the backward tilted camera system (calibration plate tilted backward), (d) represents the left forward tilted camera system, (e) represents the right forward tilted camera system, (f) represents the calibration plate flipped 180 degrees to face the camera, (g) represents the forward tilted camera system after the calibration plate is flipped 180 degrees, (h) represents the backward tilted camera system after the calibration plate is flipped 180 degrees, (i) represents the left forward tilted camera system after the calibration plate is flipped 180 degrees, and (j) represents the right forward tilted camera system after the calibration plate is flipped 180 degrees. Figure 3 As shown in the figure, to simplify the calculation model, during camera calibration, the world coordinate system of the calibration plate is aligned to its own plane, with the surface normal direction being the Z axis.

[0062] 1.1.1 Estimating the initial values of camera parameters

[0063] like Figure 5 As shown in the figure, in the camera imaging model, P is the object point, S is the optical center of the lens, P' is the actual image point of the object point P, P' is the ideal image point, O is the principal point, and O' is the ideal principal point. After the camera captures the image of the calibration plate with the marker points, the calibration plate is detected using the Canny edge detection algorithm, sub-pixel edge extraction or ellipse fitting process to determine the spatial position coordinates of each marker point on the calibration plate; combined with the encoding rules of the encoding points, the number of the encoded marker points is confirmed. Since the world coordinate system of the calibration plate is in its own plane, the spatial coordinates of all marker points on the calibration plate are uniformly expressed as (X w ,Y w ,0).

[0064] Continue to refer Figure 5 In the actual projection imaging process, due to the influence of camera internal and external parameters and lens distortion, the following collinear equations are obtained:

[0065]

[0066] The above formula (1) is the mathematical model of the ten-parameter calibration model of the embodiment of the present invention. w ,Y w ,Z w ) represents the coordinates of the object point in the world coordinate system / mm; (X s ,Y s ,Z s ) represents the coordinates of the projection center point in the world coordinate system / mm; (x, y) represents the coordinates of the actual image point / pixel; (c x ,c y ) represents the principal point deviation / pixel; (△x,△y) represents the image point deviation / pixel caused by lens distortion; f represents the focal length / mm. a1, a2, and a3 represent the coefficient factors of the world coordinate system's X-axis coordinates; b1, b2, and b3 represent the coefficient factors of the world coordinate system's Y-axis coordinates; and c1, c2, and c3 represent the coefficient factors of the world coordinate system's Z-axis coordinates. The following three matrix relationships exist:

[0067]

[0068] That is, the camera's external parameter conversion relationship is as follows:

[0069] C=R -1 (WT) (2)

[0070] Among them, matrices C and R are expressed as follows:

[0071]

[0072]

[0073] (X c ,Y c ,Z c ) is the coordinate of the object point in the camera coordinate system / mm. represents the X-axis angle between the object point and the projection point in the camera coordinate system, k represents the Y-axis angle between the object point and the projection point in the camera coordinate system, and ω represents the Z-axis angle between the object point and the projection point in the camera coordinate system.

[0074] In general, lens distortion includes radial distortion, decentering distortion, and image plane distortion. The distortion model used in the embodiment of the present invention is:

[0075]

[0076] Among them, the expressions of radial distortion, eccentric distortion, and image plane distortion are:

[0077]

[0078] Among them, △x r , △y r They represent the components of radial distortion in the X-axis and Y-axis in the image coordinate system, △x l , △y l They represent the components of the eccentric distortion in the X-axis and Y-axis in the image coordinate system, △x p , △y p They represent the components of the image plane on the X-axis and Y-axis in the image coordinate system respectively; K1, K2, K3 are radial distortion parameters, B1, B2 are eccentric distortion parameters, and E1, E2 are image plane distortion parameters;

[0079] Due to the existence of collinearity equation (1), when estimating the initial value of the intrinsic parameters, the image distortion problem is temporarily not considered. The initial value of the distortion parameter can be set to zero, so the simplified collinearity equation is as follows:

[0080]

[0081] In formula (4), the subscripts R and T represent the elements in the aforementioned matrices R and T. The subscripts indicate the position in the matrix. For example, (0,0) represents the first row and first column of the matrix, (0,1) represents the first row and second column of the matrix, (1,0) represents the second row and first column of the matrix, and so on. Since equation (4) cannot be solved by linear methods, a vector parameter η is defined to linearize it:

[0082] η=[η1 η2 η3 η4 η5 η6 η7 η8] T

[0083] After linearization transformation, the following recursive linear equation is obtained:

[0084]

[0085] in:

[0086]

[0087]

[0088] Solve the linear equation containing the vector parameter η for all calibration plate images; then, through {η i}(i=1,2,…,8) to solve the intrinsic parameters of the camera and the extrinsic parameters of each calibration plate image. For Equation (5), each calibration plate image has 8 unknowns, so there must be at least 4 object points on the calibration plate and 4 corresponding image points in the camera image. Solve the above 8 unknowns using the least squares method. Simplifying Equation (5) yields:

[0089] Aη=b (6)

[0090] Solving the matrix yields:

[0091] η=(A T A) -1 A T b (7)

[0092] in:

[0093]

[0094] b=[x1 y1 x2 y2 x3 y3 ... x n y n ] T

[0095] Among them, Xw1, Yw1~Xw n ,Yw n Represents the world coordinates of n marker points. x1,y1~x n ,y n Represents the ideal image point coordinates of n marker points. It should be noted that the T in the upper right corner of the matrix represents the transpose of the matrix. This algorithm is a common method in visual measurement, and the vector parameter η can be successfully solved. However, the vector parameter η exists in each calibration plate image, and further calculation and processing are required to solve the intrinsic parameters of the camera. From formula (7), it can be seen that for each calibration plate image, there are three unknown intrinsic parameters (c x ,c y ,f), and six unknown external parameters

[0096] Assume {η i} is known in each calibration plate image, and a parameter separation algorithm is derived by factorization. Since the rotation matrix R is a standard orthogonal matrix, the following linear equation can be obtained:

[0097]

[0098] in:

[0099]

[0100] Similar to solving equation (5), equation (8) is also solved by the least squares method. Since it contains four unknowns, at least two images are required to solve it. When ξ=[ξ1 ξ2 ξ3 ξ4] T After obtaining, the camera internal parameters can be deconstructed:

[0101]

[0102] Finally, the extrinsic parameters of each image are calculated as shown below. The translation matrix calculation formula is:

[0103]

[0104] The rotation matrix calculation formula is:

[0105]

[0106] Since the above algorithm does not guarantee the orthogonality of the rotation matrix R, the final result needs to be orthogonalized. The new rotation matrix obtained by performing SVD decomposition on the R matrix obtained above is:

[0107] R=UV T

[0108] This completes the estimation of the initial values of the camera's internal parameters.

[0109] 1.1.2 First Iteration (Bundling) Adjustment

[0110] Taylor expansion is performed on the collinear equations (1), and the linearized expression is obtained as follows:

[0111]

[0112] in, is the actual image point coordinate; is the coordinate of the image point obtained by calculating the initial value of the intrinsic parameter; is the partial derivative of the parameter; △X is the correction number of the parameter; V is the residual of the image point coordinate.

[0113] Further subdividing the parameters, we can obtain the error equation (10):

[0114] V=AX1+BX2+CX3-L (10)

[0115] In formula (10), X1, X2, and X3 are the correction values of the intrinsic parameters (including distortion parameters), extrinsic parameters, and the three-dimensional coordinates of the object point, respectively; A, B, and C are the partial derivative matrices of X1, X2, and X3, respectively; and L is the deviation between the true value and the initial value of the observation, which can be expressed as follows:

[0116]

[0117]

[0118]

[0119]

[0120] X1=[Δf Δc x Δc y Δk1 Δk2 Δk3 ΔB1 ΔB2 ΔE1 ΔE2] T ;

[0121]

[0122] X3=[ΔX W ΔY W ΔZ W ];

[0123] Among them, v x 、v y Indicates that the image point coordinate residual is in the X-axis component, and the image point coordinate residual is in the Y-axis component. △f, △c x ,△c y , △k1, △k2, △k3, △B1, △B2, △E1, △E2, △X S ,△Y S , △Z S 、 △ω、△k、△X W ,△Y W , △Z W These are all related parameters generated by Taylor expansion.

[0124] As can be seen from the previous step, after estimating the initial values of the camera's intrinsic parameters and the extrinsic parameters of each image, since the world coordinates of each marker point on the calibration plate (the world coordinate system of the calibration plate is aligned to the calibration plate plane, and the surface normal direction is the Z axis) are known in advance through DP photogrammetry, the error equation (10) can be used to use the light adjustment method to iterate the camera's intrinsic parameters and the calibration image's extrinsic parameters. To ensure stability, a global adjustment is performed in three steps. First, the camera's intrinsic parameters and global point coordinates are fixed, and only the extrinsic parameters of each image are adjusted. Second, the extrinsic parameters and global point coordinates of each image are fixed, and only the camera's intrinsic parameters are adjusted. Finally, the global point coordinates are fixed, and the camera's intrinsic parameters and the extrinsic parameters of each image are adjusted at the same time. After completing the bundled adjustment, a set of intrinsic parameters of the current camera can be output.

[0125] 1.1.3 Marker point ellipse correction

[0126] Generally, after the initial value of the camera's intrinsic and extrinsic parameters is solved, the elliptical image needs to be calibrated twice in order to better iteratively adjust and determine a set of camera intrinsic and extrinsic parameters. In the actual calibration process, the calibration plate will obtain images at different positions of the camera. The same elliptical image taken at different positions of the calibration plate will have obvious differences. Figure 6 As shown in the figure, if the angle between the camera and the calibration plate is large, the image of the marker center will be severely deformed into an ellipse. Using the ellipse to extract the marker center will lead to large solution errors. (This phenomenon is particularly obvious when using a combination of coded and non-coded points, as non-coded points are generally randomly numbered based on their position relative to the coded points in the image.) Therefore, after completing the initial iteration and adjustment of the intra-phase parameters, a secondary ellipse correction is required for each point on the calibration plate.

[0127] like Figure 6 As shown in the figure, the left side shows the imaging effect of the calibration plate under ideal conditions, where the ideal image point C' at the center of the marker coincides with the actual center image point E', that is, C'=E', and the distance between the two is e=0; while the right side shows the actual shooting process, where the offset distance between the ideal image point C' at the center of the marker and the actual center image point E' is e≠0.

[0128] refer to Figure 7 In the ideal state, the images of each point on the marker ellipse are reference points, and in the actual shooting, each point on the marker is the actual image point. Assume that the actual image points form the image point set {p i}(i=1,2,…,n), the corresponding ideal reference point set is {P i}(i=1,2,…,n), where n is the number of all the marking points on the calibration plate. Assume that any pixel p i (x i ,y i ), corresponding to the reference point P i (Xi ,Y i ), through the matrix conversion relationship, we can see that there is the following mapping relationship between the two:

[0129]

[0130] Therefore {P i} and {p i} also have the above mapping relationship, which can be obtained through matrix processing:

[0131]

[0132] in:

[0133]

[0134]

[0135] The simplified expression is:

[0136] P=S(a0,...,c2,p) (12)

[0137] First, extract the point P on the edge of the landmark point i Perform the above matrix transformation to obtain the corresponding points P on the reference point boundary i ; Then, for the point set {P i} Perform ellipse fitting to find the center E of the circle; since the above matrix relationship is not a linear reversible relationship, a series of solution methods are required to solve the inverse function relationship S - (a0, ..., c2, p); and use the inverse function relationship to perform matrix transformation on the circle center E to find the corrected circle center E'. So far, the ellipse center correction work has been completed, and the corrected marker point (circle center) coordinate set {p i '}. The coordinate set of the center of the circle of the landmark point {p i '}The error equation (10) is taken back as the known object point coordinates to participate in the iteration of camera intrinsic parameters and image extrinsic parameters.

[0138] 1.1.4 Second Bundling Adjustment

[0139] After completing the ellipse calibration for all markers on the calibration plate, the second bundling adjustment can be performed. The actual operation is essentially the same as the first bundling adjustment. The goal is to slightly adjust the center coordinates of all points after the ellipse calibration. After the second bundling adjustment, the coordinates of the points will be closer to the actual coordinates. After the adjustment is completed, the converged intrinsic parameters are used as the camera intrinsic parameters in subsequent image processing.

[0140] 1.2 Camera extrinsic parameter calibration

[0141] Camera extrinsic calibration confirms the camera's position relative to the subject. Because the actual plane location of strain during specimen strain cannot be accurately estimated, extrinsic calibration requires the calibration plate to be parallel to the specimen plane and tightly adhere to it. As previously mentioned, the center of the calibration plate's plane serves as the origin of the world coordinate system. An image of the calibration plate is acquired, and image processing is performed to determine the rotation and translation matrices between the camera coordinate system and the world coordinate system.

[0142] Since the thickness of the calibration plate is known, the position of the image point in the world coordinate system is determined based on the positional relationship between the camera and the calibration plate plane. On this basis, the calibration plate thickness value is added in the Z direction to obtain the true position coordinates of the image point.

[0143] 1.2.1 Initial estimation of camera extrinsic parameters

[0144] The camera intrinsic parameters have been solved previously. Combined with the known coordinates of the landmark points, the error equation (10) can be simplified to:

[0145] V=BX2-L (13)

[0146] As can be seen from Equation (13), there are six extrinsic parameters. The process of solving the camera extrinsic parameters only requires the coordinates of three or more object points. Since the single-pixel rear intersection algorithm is an iterative calculation after linearizing the nonlinear equation, it is first necessary to know the initial values of the unknowns. The commonly used method is the cone method. The present invention will be explained later using the cone method as an example.

[0147] The cone method uses the principle that the vertex angles between the light rays in the image space and the object space in the photographic beam cone are equal to determine the external parameters of the image. Figure 9 As shown, the coordinates of any non-collinear known points P1, P2, and P3 on the calibration plate are expressed as three known coordinate object points (control points) in the world coordinate system. The lowercase p1, p2, and p3 are the corresponding image points of the above three points on the camera station S. Since the pyramids S-p1p2p3 and S-P1P2P3 are similar, we can get:

[0148]

[0149] Since the object point P i The coordinates of (i=1,2,3) in the world coordinate system O-XYZ and the corresponding image point p i The coordinates of (i=1,2,3) in the image space coordinate system S-xyz are all known, such as Figure 9 As shown. Therefore, the lengths of line segments Sp1, Sp2, Sp3, p1p2, p2p3, p1p3, P1P2, P2P3 and P1P3 are all known.

[0150] In the image space coordinate system, for the pyramid S-p1p2p3, we can calculate:

[0151]

[0152]

[0153]

[0154] In the world coordinate system, we can see that for the pyramid S-P1P2P3, the same set of relationships exists:

[0155]

[0156]

[0157]

[0158] In the above formula, only the lengths of SP1, SP2, and SP3 are unknown. Since it is a nonlinear equation, an iterative solution is required. The method for finding the initial value is as follows:

[0159]

[0160]

[0161]

[0162] After obtaining the lengths of the three sides SP1, SP2, and SP3, we can get the coordinates of the three object points in the image space coordinate system:

[0163]

[0164] At this point, the coordinates of the three object points in the world coordinate system and the image space coordinate system have been obtained. To solve the rotation matrix between the two coordinate systems, first calculate the centers of the three object points in the world coordinate system and the image space coordinate system:

[0165]

[0166]

[0167] Among them, Pw is the center coordinate of the three object points P1, P2, and P3 in the world coordinate system; Pc is the center coordinate of the three object points P1, P2, and P3 in the image space coordinate system;

[0168] Therefore, the formula for calculating the rotation matrix is:

[0169] R″=[P1-P w , P2-P w , P3-Pw ]*[P1′-P c , P2′-P c , P3′-P c ] -1

[0170] After solving the rotation matrix, we can solve the translation matrix T":

[0171] T″=mR″*M

[0172] In the above formula, m is the coordinate of the object point's center of gravity in image space, and M is the coordinate of the object point's center of gravity in the world coordinate system. This solves the initial rotation matrix R' and translation matrix T' from the image space coordinate system to the world coordinate system, completing the initial estimation of the camera's extrinsic parameters.

[0173] 1.2.2 Iterative adjustment of camera external parameters

[0174] The initial values of the rotation and translation matrices obtained by the above solution are brought back to the error equation (13). The bundle adjustment method is used to optimize and iterate the external parameters, and finally the rotation matrix relationship R* and translation matrix relationship T* between the world coordinate system and the camera coordinate system are solved. That is, the following relationship exists for each mark point on the calibration plate:

[0175] W=R * C+T * (15)

[0176] In formula (15), W represents the coordinates of a landmark point in the world coordinate system, and C represents the coordinates of the landmark point in the image space coordinate system (i.e., the camera coordinate system). After determining the relationship between the world coordinate system and the camera coordinate system in this step, this set of extrinsic parameters is crucial for the subsequent 3D reconstruction of image points.

[0177] 2. Feature marking unit (DIC matching and spot making)

[0178] 2.1 Improved Digital Image Correlation (DIC) Algorithm

[0179] The most widely used traditional feature matching algorithm is the least squares method of digital image correlation. This algorithm correlates two images of an object's surface before and after deformation to determine a unique strain. However, the mapping function is nonlinear, resulting in a complex solution and tedious data processing. For tracking the trajectory of a specimen's strain image, the present invention proposes an optimization based on existing matching algorithms. By reversing the roles of the variables at both ends of the equation in the mapping function, the entire solution process is cleverly transformed into a linear relationship solution. This significantly reduces the amount of computational data, significantly improves feature point tracking efficiency, shortens tracking time, and significantly reduces the computing power required, lowering overall hardware investment costs. Specifically, the present invention proposes a fast feature tracking algorithm based on DIC image correlation matching. First, a specific speckle pattern is formed on the specimen surface using a specific speckle generation process. Then, a self-calibrated camera is used to match and track the motion trajectory of the speckle pattern during the strain test. This algorithm is not only applicable to conventional material mechanics strain testing scenarios, but also provides stable image tracking without loss under complex and harsh testing conditions, even in extreme high and low temperature testing environments. The specific principles are as follows:

[0180] Digital image correlation method is a method to calculate the uniqueness and strain by performing correlation calculation on two images before and after deformation of the object surface. The basic principle of this method is as follows Figure 10 As shown, in the reference image (i.e., the image of the specimen without strain), a rectangular image area of size (2n+1)*(2m+1) centered at the point to be matched (x0, y0) is taken as the reference image sub-area; in the image to be matched (the image of the specimen after strain displacement), a certain image search method is used to find the image sub-area with the greatest similarity to the selected reference image sub-area, as the target image sub-area, and its center (x', y') is obtained. In the image search process, the predefined correlation coefficient is a function that measures the similarity between the reference image sub-area and the target image sub-area. The digital image correlation method completes the image matching by finding the extreme value of the correlation coefficient, and then calculates the displacement and strain field. The embodiment of the present invention uses a least squares correlation coefficient to evaluate the similarity between two image sub-areas:

[0181]

[0182] In formula (16), C SSD (p) represents the similarity of the image sub-region, f(x, y) is the grayscale value of the sub-region point (x, y) in the reference image F; g(x′, y′) is the grayscale value of the sub-region point (x', y') in the target image G; r0 and r1 represent the grayscale difference between the two images due to different brightness.

[0183] In order to achieve better convergence, the embodiment of the present invention uses the first-order mapping function shown in formula (17) to express the matching relationship between two image sub-regions:

[0184]

[0185] In formula (17), u and v represent the unit pixel length of the image on the x-axis and the y-axis, respectively, and a x ,a y Represents the coefficient components of the two image sub-areas in the x-axis direction; b x 、b y Identify the coefficient components of the two image sub-regions in the y-axis direction.

[0186] Therefore, for the two image sub-regions, their center coordinates satisfy the following:

[0187]

[0188] At this time, the matching relationship of formula (10) can be rewritten as follows:

[0189]

[0190] in:

[0191]

[0192] In the above formula, d is the translation part and m is the deformation part.

[0193] Get the minimum C SSD (p) is a nonlinear optimization problem, which is solved by the least squares adjustment method (ILS) in the embodiment of the present invention. Equation (16) adds a bias component so that all pixels in the reference image subregion can satisfy:

[0194] f(x,y)-e(x,y)=r0+r1g(x′,y′) (19)

[0195] Among them, e(x, y) represents the deviation component. g(x′, y′) is not linear because of the existence of the mapping function [m, d]. According to the least squares iteration theory, the entire calculation process requires the calculation of the gradient value at g(x′, y′), and it needs to be continuously updated in each iteration, which requires a lot of calculation. Here, an improved least squares adjustment method is introduced, which does not repeatedly calculate the gradient value of the image sub-area, greatly reducing the overall calculation amount. The mapping function used in this method is a first-order reversible transformation, which can be obtained by reversibly transforming Equation (18):

[0196]

[0197] in:

[0198]

[0199] Therefore, the roles of the reference image F and the target image G are transformed. The initial method is to continuously adjust and update the image subregion of G to satisfy F. The linear equation after the above transformation is:

[0200] r0+r1g(x′,y′)-e(x,y)=f 0 (x, y) + f x du′+f x Δx′du x ′+f x Δy′du y ′+f y dv′+f y Δx′dv x ′+f y Δy′dv y '

[0201] Among them, f 0 (x,y),f x 、f y 、u'、△x'、u x ', △y' represents, u y '、v'、v x '、v y 'Represents the parameters of f(x,y) after Taylor expansion.

[0202] According to the derivation process from equation (16) to this point, the iteration step size [Δm′, Δd′] can be determined. At this time, the step size is not superimposed on the current transformation matrix [m′, d′], but is reversely superimposed on [m, d]. According to the reversible transformation formula of equation (20), the new iteration relationship is derived as follows:

[0203]

[0204] Through this transformation relationship, each gradient calculation process can be performed under image F, while the calculation subregion of F (reference image subregion) does not actually change during the band adjustment. Therefore, the gradient remains unchanged in each iteration, and there is no need to re-determine a large number of calculations, thereby improving the speed and stability of the matching operation. According to multiple tests, the improved least squares adjustment method proposed in the embodiment of the present invention, based on the same number of matching points and the same patch size, significantly shortens the matching time and improves the matching efficiency through algorithm optimization. See the subsequent data comparison table for details.

[0205] 2.2 Speckle preparation method and process requirements

[0206] In addition to the improvement of the algorithm, the embodiment of the present invention also proposes an upgrade method for the feature marking process. Through comparison, it was found that earthworm spots are a marking feature that is more suitable for feature tracking, and spot making stickers and spot making stamps were proposed. Under the premise of ensuring obvious feature comparison effects, the overall spot making operation difficulty and operation time are reduced. As mentioned above, the specimen needs to be speckled before speckle tracking. A simple and efficient speckle making process and procedure can not only reduce the time for overall image acquisition and matching, but also good speckle features can contribute to the accurate matching of speckles. Due to the differences in the physical properties of various new materials, in order to more effectively extract image feature points in engineering, the following five feature point preparation methods have been determined through continuous experimental comparisons:

[0207] A. Earthworm spot seal

[0208] like Figure 11 The following diagrams show two commonly used line width specifications for earthworm patches (left image (a) shows a line width of 0.25mm, and right image (b) shows a line width of 0.5mm). The tracking speed and stability of various features (dots, earthworm patches, BMW logos, and other patterns) in actual engineering projects show that earthworm patches can track any point pair, and the strain / stress curve feedback is close to linear. (The linear transformation represents the entire tensile process of the test piece, with continuous tracking and no loss of the target area or large offset.) Generally speaking, the line width of the earthworm patch can be adjusted to meet the feature tracking needs of multiple fields of view simultaneously.

[0209] Specifically, the earthworm spots can be copied onto the test piece by pressing the stamp to release the ink. However, this step requires the test piece to have a smooth surface without obvious bumps and preferably a metallic color. Since most inks generally have a reflective effect, after the earthworm spots are applied to the test piece, the operator will be required to apply a layer of matte paint to eliminate the reflective effect. Figure 12 As shown, if the metallic luster of the test piece's surface is not very obvious, the earthworm stain can be directly applied or covered. However, if the surface has a strong metallic luster, it is necessary to spray white primer first, and then apply or cover the earthworm stain after the paint finish is formed. This method is quick and efficient, and has a high tracking speed, but it also places certain requirements on the smoothness of the test piece surface and the operator.

[0210] B. Earthworm stickers

[0211] Similar to the A-worm spot stamp, the tracking effect of the earthworm spot sticker is very similar to that of the stamp. The biggest difference between the two lies in the production process. Generally, the sticker is a 3M sticker with a self-adhesive backing, and the earthworm spot feature is formed through surface printing / imprinting. When affixed to the test piece, it does not require the application of matte paint, simplifying the overall operation process. However, it also has limitations. Due to the inherent properties of the sticker, some rigid test pieces may not deform completely synchronously with the test piece, and the tracking data often shows small gaps in the final stage of the test piece deformation process.

[0212] C. Paint pen spot making

[0213] For scenarios where neither of the above two types of spots can meet the requirements, you can use Figure 13 For test pieces with fewer surface features but requiring feature tracking, a ruler with holes and distance markings can be used as a template. Place it on the test piece and use a durable paint pen to draw one or more pairs of dots at predetermined distances, allowing for feature tracking.

[0214] D、non-coding point stickers (such as Figure 14 As shown, its effect is similar to the paint pen spot making method C above, and it is more suitable for test piece surfaces that cannot be painted with paint pens.

[0215] E. Spray paint to make spots

[0216] like Figure 15 As shown, spray painting is a relatively conservative method for creating speckles (speckles). It's also the most widely used, applicable to a wide range of materials. First, the entire surface of the test piece is sprayed with a white or black primer. Black or white paint is then sprayed at a slightly distance to create speckles. Finally, matte paint is applied to create a matte effect. Its speckle tracking performance is roughly comparable to that of earthworm spotting.

[0217] 3. 3D reconstruction unit (single-camera 3D reconstruction technology)

[0218] Three-dimensional reconstruction is a key step in restoring the position of object points from image points. It is a mathematical model for three-dimensional objects that is suitable for computer representation and processing. It is the basis for processing, operating and analyzing its properties in a computer environment. It is also a key technology for establishing a virtual reality that expresses the objective world in a computer. Three-dimensional reconstruction is the process of restoring the real physical position of an object. Although the calibration method of the aforementioned ten-parameter model can reduce the influence of distortion on the image to a certain extent, it is difficult to ensure that the camera is facing the plane of the test piece or the calibration plate during actual shooting. The deviation between the actual image point and the ideal image point, and the deviation between the ideal optical center and the actual optical center position still exist. In view of this, the three-dimensional reconstruction algorithm proposed in the embodiment of the present invention requires that the test piece and the calibration plate plane are always closely parallel, and based on the principle of the cone method, the conversion relationship from the camera coordinate system to the world coordinate system is solved. Using this transformation, the actual image point is projected onto the calibration plate plane through the optical center. The geometric relationship between the projected point and the image point is used to determine the world coordinates of the projected point. Since the calibration plate and specimen are nearly parallel, their positional relationship can be considered as a difference in Z distance. The calibration plate thickness is added to the projected point coordinates in the Z direction to obtain the actual position of the object point on the specimen surface. This algorithm is independent of the alignment between the actual calibration plate and the camera, and is unaffected by depth of field, displacement, or Z distance during capture.

[0219] As mentioned in 1.2 Camera Extrinsic Parameter Calibration, the conversion relationship between each point on the image from the camera coordinate system to the world coordinate system is determined through camera extrinsic parameter calibration. Its expression is as follows:

[0220] Q=R * C+T * (22)

[0221] Among them, Q represents the position of the image point in the world coordinate system, and C represents the position of the image point in the camera coordinate system.

[0222] Assuming that the coordinates of a certain image point p(x,y) in the camera coordinate system are p'(x',y',z'), the following coordinate transformation relationship exists:

[0223]

[0224] In the above formula, w and H represent the width and height of the camera chip respectively. From formula (22), we can see that the point p'(x',y',z') corresponds to the projection point P(X1,Y1,Z1) in the world coordinate system, that is:

[0225] P=R * p'+T * (twenty three)

[0226] Solving equation (23), we can obtain the projection coordinates P(X1, Y1, Z1) of the image point p(x, y) in the world coordinate system. The projection coordinates of the optical center O in the world coordinate system are O'(O1, O2, O3), where O = [O1, O2, O3] T .

[0227] Assume that the object point corresponding to the image point p(x,y) is Q(X,Y,Z), then Figure 16 It can be seen that there is a directed line The intersection point of the calibration plate plane is the object point Q. The normal vector of the calibration plate plane W is The plane W can be expressed as:

[0228] n1x+n2y+n3z+d=0

[0229] Here d is a constant term in the spatial plane expression.

[0230] Since point Q satisfies both the line and plane equations, let but:

[0231]

[0232] in:

[0233]

[0234] because is the distance from point P to plane W, and the calculation formula is as follows:

[0235]

[0236] in addition:

[0237]

[0238] It can be expressed as the unit normal vector of plane W, and the coordinates are expressed as:

[0239]

[0240]

[0241] In summary, we can get:

[0242]

[0243] Formula (18) can be expressed as:

[0244] (X, Y, Z) = (X1, Y1, Z1) + h* (T1-X1, T2-Y1, T3-Z1) (25)

[0245] At this point, the intersection Q, that is, the coordinates of the object point Q, are solved. In actual measurement, the specimen is required to be close to the back of the ruler. After the camera external parameters are calibrated, the ruler is removed, the specimen strain image is collected, and speckle tracking and three-dimensional reconstruction are performed. Considering that it is difficult to ensure that the position of the ruler is consistent with the position during the external parameter calibration during the actual specimen strain process, there is often a thickness difference a of the calibration plate in the Z direction. Therefore, the position coordinates of the real point Q should be Q(X,Y,Z+a). The speckle tracking of the image point is completed through the above Section 2.1, and the strain displacement of the image point is determined; based on the three-dimensional reconstruction theory in this section, the specific strain displacement of the real object point can be determined in the world coordinate system. At this point, the conversion from two-dimensional image point strain to three-dimensional object point strain is completed, and the measurement process of the real strain of the specimen is realized.

[0246] 4. Multi-camera point crossing algorithm unit (multi-camera array stitching with high precision and large field of view)

[0247] 4.1 Overview of Multi-Camera Stitching Methods

[0248] The aforementioned method realizes the measurement of the true strain of the specimen under a fixed field of view through the parameter calibration unit, the feature marking unit and the three-dimensional reconstruction unit. However, in actual engineering, the problem of large-field-of-view strain measurement is often encountered. The field of view expansion range is ultimately limited by simply replacing the lens. To more effectively solve the problem of full-field strain measurement in an actual large field of view, an embodiment of the present invention further provides a method for full-field strain measurement under a large field of view by splicing multiple camera arrays. This method ensures that all states of scattered spots within the field of view can be reconstructed in three dimensions without strain displacement loss.

[0249] 4.2 Overview of the Lossless Traversal Algorithm

[0250] Although industrial cameras have a fixed field of view, if multiple industrial cameras are placed at a fixed spacing to capture the strain process within a large field of view, it is not difficult to find that by adjusting the spacing, adjacent cameras share a common field of view. First, the strain process of the points in the field of view of each camera is calculated. For all points within the common field of view, the two adjacent cameras generate their own two-dimensional coordinates of the image points. To ensure the uniqueness of the image point coordinates, the embodiments of the present invention propose the theory of the point crossing algorithm. The multi-camera point crossing algorithm unit is configured to splice the image point trajectories of the multi-camera array according to the following process: first, set a boundary for the specimen image taken by each camera, and the boundary is the effective image area of the camera; then, for the current camera, move the trajectory to the outside of the boundary or the image point just on the boundary is considered to be in the effective image area of the next camera, and record this type of image point as the first image point; with the first image point coordinates obtained by the current camera as a reference, determine the first real object point position corresponding to the first image point through the three-dimensional reconstruction unit; further, based on the coordinates of the first real object point position, use the reprojection method to find the corresponding image point of the first real object point in the next camera, and record it as the second image point; finally, define the image point coordinates of the first real object point using the coordinates of the second image point obtained by the reprojection method. Figure 17 FIG2 is a schematic diagram of a lossless traversal algorithm for moving point trajectories according to an embodiment of the present invention. The upper portion (a) of the figure shows a schematic diagram of the field of view of multiple cameras, and the lower portion (b) shows a schematic diagram of the moving point trajectories on two adjacent cameras. Figure 17 In Figure (a), Stage 1 to Stage 4 represent the four positions of the speckle patch in the camera image during the tensile strain process of the specimen, and C1 to C4 represent Figure 14 Four cameras were involved in the shoot; Figure 17 In Figure (b), from left to right, Stage1, Stage2, and Stage3 represent the three positions of the speckle patch in the camera image during the tensile strain process of the specimen. Figure 17 As shown in Figure (a), in a wide field of view scenario, strain tracking and 3D reconstruction of each image point can be completed within the field of view of each camera using a single-camera strain measurement process. For two adjacent cameras, each image point in their shared image area has two sets of coordinates, which is the key issue addressed in this section. Based on the point coordinate conversion process within the camera system, the 2D images of all image point trajectories across the entire wide field of view are fully connected. Through the translation and matrix relationship between each camera coordinate system and the world coordinate system, the strain displacement of all real object point trajectories across the wide field of view can be determined. This achieves the goal of complete tracking of trajectory points across a wide field of view.

[0251] First, define the lower left corner of the first image taken by the first camera as the origin of the world coordinate system. Based on the principle of the cone method, it is determined that there is a set of rotation and translation matrix relationships between any camera Ci and the world coordinates, namely:

[0252] W i =R i *C i +T i (26)

[0253] In formula (26), i represents the i-th camera.

[0254] Combining the 3D reconstruction theory, the coordinate transformation relationship of formula (26) can be used to determine the object point coordinates corresponding to each camera image point. As for how to 3D reconstruct the image points in the common area, the following is a detailed description using camera C1 and camera C2 as an example:

[0255] In the image coordinate system, the size of the characteristic speckle patch is generally around 20-25 pixels. Assuming that the image has a valid boundary, the distance from the valid boundary to the actual edge of the image is set to 100 pixels. For the image point trajectory tracking in this distance including the valid boundary, refer to the following.

[0256] like Figure 17 As shown in Figure (b), it is the movement trajectory of the center point of the patch captured by cameras C1 and C2. The left side is the image of camera C1, and the right side is the image of camera C2. When the patch moves to Stage 2, the position of its midpoint is exactly on the effective boundary of camera C1; because camera C2 also captures the center point of Stage 2, and its image in C2's image is within the effective boundary.

[0257] First, the central image point p(x, y) of Stage 2 is 3D reconstructed under the C1 camera to obtain the world coordinates P(X, Y, Z) of the center point. Then, according to Equation (26), the following relationship exists between the C2 camera and the world coordinate system:

[0258] W2=R2*C2+T2 (27)

[0259] In the C2 camera coordinate system, there exists a camera coordinate P'(X', Y', Z') that satisfies Equation (27). Therefore, the center point world coordinate P(X, Y, Z) is substituted into Equation (27) to obtain the point coordinate P'(X', Y', Z') in the C2 camera coordinate system.

[0260] like Figure 18 As shown, point P'(X', Y', Z') is projected back to the image coordinate system of the C2 camera. Let the projected image point be p1(x', y'). Then we can see that the following relationship exists:

[0261]

[0262] By substituting the known point P'(X', Y', Z') back into Equation (28), we can solve for the projected image point p1(x', y'), which is also the central image point p1(x', y') of Stage 2. This process is called reprojection. Similarly, through the image point-3D reconstruction-object point-image point method, the image point is able to pass through each camera. This algorithm process ensures that the object point corresponding to the image point will not be offset, and also avoids matching errors and omissions, accurately and efficiently achieving the strain and displacement calculation of the speckle patch under a large field of view.

[0263] The beneficial effects brought about by the technical solutions of the embodiments of the present invention are as follows 1 to 3:

[0264] 1. Verification of the effect brought by the algorithm improvement of digital image correlation method

[0265] In this invention, an improved least squares method based on digital image correlation is used to track the trajectory of moving points. Compared with traditional matching methods, the speed is increased by more than 40%. Table 1 below shows the matching time of two speckle images using two different methods:

[0266] Table 1 Comparison of matching time of two methods

[0267]

[0268] 2. Verification of the effect of camera self-calibration output of three-dimensional position relationship

[0269] The camera self-calibration process proposed in the embodiments of the present invention includes camera intrinsic parameter calibration and camera extrinsic parameter calibration. The camera intrinsic parameter calibration uses a ten-parameter model to determine focal length, principal point deviation, and other parameters. The extrinsic parameter calibration, unlike traditional extensometers, determines the three-dimensional spatial position relationship between the camera and the scale, which is very important for the overall matching and tracking process to achieve three-dimensional reconstruction. The specific verification test is as follows:

[0270] 2-1. Test environment:

[0271] Working distance: 270mm

[0272] ◆Exposure time: 200μs

[0273] ◆Camera / Lens: 2.3M camera / 25mm lens

[0274] Camera field of view (FOV): 120mm

[0275] 2-2. Test process:

[0276] 2-2-1. Camera internal parameter calibration

[0277] refer to Figure 4, complete the camera internal parameter calibration operation process, the software interface outputs the camera internal parameter calibration results as follows Figure 19 As shown, the parameters in the interface are the relevant internal parameters in the ten-parameter calibration model;

[0278] 2-2-2. Camera external parameter calibration

[0279] Set the working distance to 270mm, place the ruler close to the front of the specimen, and take a ruler image with the camera. The specific test environment is as follows: Figure 20 As shown;

[0280] 2-2-3. Determination of position relationship

[0281] After the software solves, it outputs the final camera external parameter calibration results; the specific calibration results are as follows Figure 21 As shown in the figure, the parameters in the interface represent the relevant extrinsic parameters in the ten-parameter calibration model. As can be seen from the figure, the camera self-calibration method proposed in this invention determines the three-dimensional spatial relationship between the camera and the scale, rather than the 2D positional relationship of a traditional extensometer. This calibration method eliminates the limitation of traditional extensometers, which require the camera to be aligned with the scale, making it more convenient to use. It also avoids the influence of Z-distance and image plane displacement during 3D reconstruction.

[0282] 3. Feature tracking displacement accuracy verification

[0283] Use a calibrator to conduct an accuracy verification test on this video extensometer system. The operating procedures are as follows:

[0284] 3-1. Test environment:

[0285] Working distance: 270mm

[0286] ◆Exposure time: 200μs

[0287] Camera field of view (FOV): 120mm

[0288] ◆Camera / Lens: 2.3M camera / 25mm lens

[0289] Calibrator model: GWB-200JA (which complies with the requirements of JJF 1096-2002 and its allowable error also complies with the metrological characteristics requirements of the calibrator)

[0290] ◆Calibrator range: 0-25mm

[0291] ◆Resolution of calibrator: 0.2μm

[0292] 3-2. Test process:

[0293] 3-2-1. Set the measurement distance

[0294] Start the extensometer camera and blue light source. Taking an earthworm pattern with a line width of 0.25 mm as an example, set the extensometer gauge length to 25 mm, the speckle patch size to 15 pixels, and the z-axis offset to -5.3 mm (this value is determined by the thickness of the calibration plate used, with the positive z-axis pointing toward the camera along the calibration plate plane).

[0295] 3-2-2. Speckle tracking test

[0296] After completing the calibration process of the camera's internal and external parameters, start from the zero position and slowly rotate the calibrator until it reaches the maximum displacement for calibration (the maximum displacement of this test is 20mm); each time it rotates a certain distance, read the readings of different scale lines on the calibrator and the corresponding displacement readings displayed by the extensometer software. Repeat this three times, which is the process test of the extensometer; from the zero position, apply the maximum displacement of the calibration range point by point in the process, and then return to the zero position point by point in the return stroke. Perform three sets of tests in this way, which is the return stroke test of the extensometer.

[0297] Comparison indication deviation

[0298] According to the definition of each standard quantity in Table 2, the numerical value or range of each standard quantity of the extensometer is determined.

[0299] Table 2 Extensometer measurement characteristics

[0300]

[0301] Verification of the extensometer calibration instrument: For extensometers that can directly measure gauge length, the gauge length should be measured directly after completing the installation preparations. Each gauge length should be measured three times, and the relative error of the extensometer gauge length measured each time should meet the requirements of Table 2.

[0302] The relative error of the extensometer gauge length is calculated according to formula (29):

[0303]

[0304] Among them, q Le Indicates the relative error of the extensometer gauge length, expressed in %; L' e Indicates the measured value of the extensometer gauge length, unit: mm; L e Indicates the nominal value of the extensometer gauge length, in mm.

[0305] Resolution verification: In Table 2, the absolute resolution r is the minimum value that can be read from the extensometer's indicating device. The relative resolution is the difference between the minimum value r that can be read from the instrument and the displacement l indicated by the extensometer. i Visually inspect and calculate the resolution of the extensometer.

[0306] Verification of indication error: When the temperature is stable, use a calibrator to pre-apply at least two displacements corresponding to the verification range of the extensometer to the extensometer before verification. After checking that it returns to zero normally, apply a very small negative displacement to the extensometer and return it to zero, and re-zero the extensometer.

[0307] Accuracy verification:

[0308] 1) Process accuracy measurement: Three sets of measurements are required for calibration, and each set of measurements generally has no less than 10 measurement points (excluding the zero point). According to the selected calibration range, a calibrator is used to apply a given displacement to the extensometer point by point. When the maximum displacement of the calibration range is reached, it returns to zero. After the first set of measurements is completed, record the data and turn off the video extensometer; then use the above method and repeat the test twice. The detection points are allocated according to actual usage and are relatively evenly distributed. For a given displacement, the relative error of the extensometer indication is calculated according to formula (30):

[0309]

[0310] Where, σ represents the relative error of the extensometer indication, expressed in %; l i Indicates the displacement indicated by the extensometer, in mm; l t Indicates the displacement value given by the calibrator, in mm.

[0311] 2) Return stroke accuracy measurement: During calibration, apply the maximum displacement of the calibration range point by point from the zero position in the forward stroke, and then return to the zero position point by point in the return stroke. Perform three sets of tests in this way. The relative error u of the extensometer indication in the forward and return strokes is calculated according to formula (31):

[0312]

[0313] In formula (31), u represents the relative error of the extensometer indication in the forward and reverse strokes, expressed in %; l i 'Indicates the return displacement indicated by the extensometer at the same test point, unit: mm.

[0314] Figure 22 Schematic diagram of the resolution of a Class 1 extensometer according to an embodiment of the present invention; Figure 23 Schematic diagram of the limit of the system error of the level 1 extensometer in the embodiment of the present invention. Figure 22 It means that when the extensometer indicates displacement within 200 microns (inclusive), the resolution is in absolute value; when it exceeds 200 microns, the resolution is expressed as a percentage of the reading. Figure 23 It means that the actual displacement given by the calibrator is within 300 microns (including 300 microns), and the system error uses the absolute value; if it exceeds 300 microns, the system relative error is used.

[0315] The test data of the verification are shown in Table 3 below:

[0316] Table 3 Comparison results of extensometer indications

[0317]

[0318] Verification conclusion: According to the extensometer calibration characteristic requirements given in the national standard JJG 762-2007, the extensometer given in this article meets the extensometer accuracy requirements of Class 0.5, which is already a relatively high level in the current industry.

[0319] The above is a further detailed description of the present invention in conjunction with specific preferred embodiments, and the specific implementation of the present invention should not be considered to be limited to these descriptions. Those skilled in the art will recognize that several equivalent substitutions or obvious variations can be made without departing from the scope of the present invention, and that such variations, with the same performance or use, should be considered to fall within the scope of protection of the present invention.

Claims

1. A 3D video extensometer with a single CCD camera, characterized in that: include: A camera, a lens, a blue light source, an electrical control terminal and a computer, wherein the electrical control terminal is connected to the camera and the computer; the computer is equipped with a parameter calibration unit, a feature marking unit and a three-dimensional reconstruction unit; The parameter calibration unit is configured to perform camera self-calibration to determine the intrinsic and extrinsic parameters of the camera; The characteristic marking unit is configured to: prepare characteristic speckles on the surface of the specimen, and match and track the motion trajectory of the speckles of the specimen through a self-calibrated camera; The three-dimensional reconstruction unit is configured to: solve the transformation relationship between the camera coordinate system and the world coordinate system based on the principle of the pyramid method, and project the image point through the optical center onto the calibration plate plane based on the transformation relationship, solve the world coordinates of the projection point by using the geometric position relationship between the projection point and the image point, add the calibration plate thickness in the Z direction to the world coordinates of the projection point to obtain the position of the object point on the specimen surface, and complete the transformation of the two-dimensional image point strain to the three-dimensional object point strain; The feature marking unit is configured to match and track the speckle motion trajectory of the specimen according to the following process: optimizing the least squares method of digital image correlation to convert the mapping function from a nonlinear relationship to a linear relationship, and performing correlation calculation on two images of the specimen surface before and after deformation to solve the strain; The steps of converting the mapping function from a nonlinear relationship to a linear relationship include: 1) Selecting a rectangular area of a predetermined size centered at the point to be matched in the reference image as a reference image sub-area; the reference image is an image of the specimen without strain; 2) searching, in the image to be matched, for an image region having the greatest similarity to the subregion of the reference image as the target image subregion; the image to be matched is an image of the specimen generating strain; 3) using a first-order mapping function of a digital image correlation method to express a matching relationship between the reference image subregion and the target image subregion; 4) performing a reversible transformation on the matching relationship between the reference image sub-region and the target image sub-region to convert the nonlinear mapping relationship into a linear mapping relationship.

2. The 3D video extensometer with a single CCD camera as claimed in claim 1, characterized in that: The parameter calibration unit is configured to perform camera self-calibration according to the following process: Collect calibration plate images; The grayscale area of the calibration plate image is extracted and ellipse fitting and correction are performed to determine the intrinsic and extrinsic parameters of the camera. The intrinsic parameters of the camera include lens distortion and the principal point deviation and focal length caused by lens distortion.

3. The 3D video extensometer with a single CCD camera as claimed in claim 2, characterized in that: The parameter calibration unit is configured to perform camera internal parameter calibration according to the following process: 1) Using the camera to capture an image of a calibration plate with marker points, and performing marker point detection on the calibration plate image; 2) Determine the spatial coordinates of the center of each marker point, and estimate the initial values of the camera intrinsic parameters based on the image point coordinates and world coordinates of each marker point; 3) Use the bundle adjustment method to iterate the camera intrinsic parameters and the calibration plate image extrinsic parameters to make the first bundle adjustment to the estimated initial values of the intrinsic parameters; 4) Perform ellipse correction on each mark point on the calibration plate; 5) After completing the ellipse calibration of all the marking points on the calibration plate, perform the second bundling adjustment using the same method as the first bundling adjustment; 6) After completing the second bundle adjustment, combine photogrammetry to obtain the global points of the calibration plate, converge, and confirm the camera internal parameters.

4. The 3D video extensometer with a single CCD camera as claimed in claim 3, characterized in that: The parameter calibration unit is configured to perform the first bundling adjustment according to the following process: First, the camera intrinsic parameters and global point coordinates are fixed, and only the extrinsic parameters of each calibration plate image are adjusted; Secondly, the extrinsic parameters and global point coordinates of each calibration plate image are fixed, and only the intrinsic parameters of the camera are adjusted; Finally, the global point coordinates are fixed and the camera intrinsic parameters and the extrinsic parameters of each calibration image are adjusted simultaneously.

5. The 3D video extensometer with a single CCD camera as claimed in claim 2, characterized in that: The parameter calibration unit is configured to perform camera external parameter calibration according to the following process: 1) Using the cone method, the calibration plate image captured by the camera is used to solve the initial values of the rotation matrix and the translation matrix from the camera coordinate system to the world coordinate system, thereby completing the estimation of the initial values of the camera extrinsic parameters; 2) Use the bundle adjustment method to iteratively adjust the initial values of the camera extrinsic parameters to determine the final rotation matrix and translation matrix from the camera coordinate system to the world coordinate system, completing the camera extrinsic parameter calibration; During the camera external parameter calibration process, the calibration plate is always kept parallel to the specimen surface.

6. The CCD single-camera 3D video extensometer according to claim 1, characterized in that: The characteristic marking unit is configured to prepare characteristic speckles on the surface of the specimen by using earthworm spot stamps, earthworm spot stickers, paint pen spots, non-coding dot stickers or spray paint spots.

7. The CCD single-camera 3D video extensometer according to claim 1, characterized in that: The computer is further configured with: a multi-camera point crossing algorithm unit for realizing three-dimensional reconstruction of large field of view strain measured by a multi-camera array.

8. The CCD single-camera 3D video extensometer according to claim 7, characterized in that: The multi-camera point crossing algorithm unit is configured to stitch the image point trajectories of the multi-camera array according to the following process: First, a boundary is set for the specimen image captured by each camera, and the boundary is the effective image area of the camera; Then, for the current camera, the image points whose trajectory moves outside the boundary or just on the boundary are considered to be within the effective image area of the next camera, and such image points are recorded as first image points; using the coordinates of the first image point captured by the current camera as a reference, the position of the first real object point corresponding to the first image point is determined by the 3D reconstruction unit; Furthermore, based on the coordinates of the first real object point, a reprojection method is used to find the corresponding image point of the first real object point in the next camera, which is recorded as the second image point; Finally, the coordinates of the second image point obtained by the reprojection method define the image point coordinates of the first real object point.

Images