Multi-view digital image correlation measurement method based on mirror surface assisted full surface

Abstract

The invention relates to the technical field of panoramic morphology and deformation measurement, and particularly discloses a multi-view digital image correlation measurement method based on a mirror-assisted full surface, and the method comprises the steps: arranging and configuring a novel mirror-assisted multi-view digital image system of an optical structure; samples and calibration image pairs of a tested sample in a mirror surface and a prism before and after deformation are collected based on a system, system imaging parameters are calibrated based on the calibration image pairs, and a virtual curved surface, the three-dimensional shape of the mirror surface and three-dimensional coordinates of a real point and a virtual point of a calibration plate are reconstructed. And calculating the reconstructed data points to obtain a reflection transformation matrix of the mirror surface and the prism pair, and reflecting the virtual surface to a corresponding real position based on the matrix to obtain the panoramic contour of the tested sample in the initial and deformation states. And synchronous high-definition images of samples at different field depths can be obtained, and dynamic high-precision three-dimensional deformation and contour measurement of object parts in different field depth ranges can be realized.

Description

Technical Field

[0001] The present application relates to the technical field of panoramic morphology and deformation measurement, and in particular to a multi-view digital image correlation measurement method based on mirror-assisted full-surface. Background Art

[0002] The three-dimensional digital image correlation method is a non-contact full-field morphology and deformation optical measurement technology, which can be used for 360° panoramic or dual-surface morphology and deformation measurement of the object to be tested. In order to dynamically or statically measure the panoramic or dual-surface deformation in a simpler and more economical way, there is a related technology that uses a set of ordinary dual-camera three-dimensional digital image correlation systems and two plane mirrors fixed behind the sample to complete the panoramic / dual-surface morphology and deformation field measurement. In this method, the morphology and deformation field of the front surface of the tested sample can be directly measured by the three-dimensional digital image correlation system, while the two invisible rear surfaces are measured by reflection from two plane mirrors. Compared with the original multi-camera multi-view digital image correlation, the dual-plane mirror-assisted multi-view digital image correlation in the related technology has great advantages in the panoramic / dual-plane morphology and deformation measurement of conventional-sized samples.

[0003] However, for the mirror-assisted method in the related art, in some special measurement scenarios, such as when the size of the object to be measured is relatively large, due to the limitation of the limited depth of field of the camera, the multi-view digital image correlation measurement system assisted by a plane mirror cannot guarantee that objects or parts of objects with different depths of field in the field of view simultaneously obtain high image quality, and decorrelation is prone to occur during the matching process, resulting in a decrease in measurement accuracy and speed. At the same time, the classic panoramic measurement system method requires complex calibration work and cumbersome point cloud registration algorithms, which increases the complexity of the measurement process. Summary of the invention

[0004] The present application aims to solve at least one of the problems existing in the above-mentioned prior art. Based on this, a multi-view digital image correlation measurement method based on mirror-assisted full-surface is proposed to achieve simultaneous high-definition imaging of measured objects or different parts of objects with different depths of field, as well as high-precision real-time measurement of three-dimensional deformation and contour of objects or different parts of objects with different depths of field.

[0005] In a first aspect, the present application provides a multi-view digital image measurement method for full-surface three-dimensional measurement, comprising: Step S1: Arrange a set of new mirror-assisted multi-view digital image system with optical structure, produce speckles with black dots on a white background on the plane mirror surface of the system and the surface of the tested sample, and adjust the distance between the prism pairs, the direction of the light source and the intensity of the light source in the optical configuration of the system to obtain a high-definition image of each surface; Step S2: Based on the system, the sample images of the tested sample before and after deformation in the mirror and the prism and the calibration image pair for calibrating system parameters are collected; Step S3: Based on the calibration image pair, the intrinsic parameters and extrinsic parameters of the system imaging are accurately calibrated using a stereoscopic vision calibration method, and the three-dimensional shapes of the three virtual surfaces, the two mirror surfaces, and the three-dimensional coordinates of the real points and the corresponding virtual points of the calibration plate are reconstructed based on the triangulation principle using the collected sample images and the obtained system parameters; Step S4: performing plane fitting on the reconstructed mirror surface, and calculating the three-dimensional coordinates of the real points and imaginary points corresponding to the coordinates of the corner points in the calibration plate using the image pairs of the calibration plate collected before and after the prism is removed, so as to obtain the reflection transformation matrix of the mirror surface and the prism pair; Step S5: Based on the reflection transformation matrix, the virtual surface is reflected to the corresponding real position to obtain a 360° panoramic and / or double-sided profile of the tested sample in the initial state and the deformed state, the constructed initial state profile is subtracted from the deformed state profile to obtain the displacement field in each state, and the full-field 360° panoramic and / or double-sided strain components are calculated from the displacement field based on a point-by-point least squares fitting algorithm.

[0006] In some embodiments, in step S1, the novel mirror-assisted multi-view digital image system specifically includes a pair of binocular cameras for capturing digital image pairs, two plane mirrors for panoramic image imaging, a pair of right-angle prism pairs for adjusting the depth of field of the front surface image, and a displacement fine-tuning platform for adjusting the position of the right-angle prism pair.

[0007] In some implementations, the step S2 specifically includes: The sample images of the tested sample before and after deformation are collected in the mirror surface, and after the sample images are collected, multiple sets of calibration image pairs are collected respectively when the calibration plate is placed behind the prism, the prism is removed and the position of the calibration plate is kept unchanged, and the plane mirror is removed and the calibration plate is placed in different postures.

[0008] In some implementations, the step S3 specifically includes: An area of ​​interest is selected from the speckle pattern area on the plane mirror, and a group of calculation points are determined in the area of ​​interest. The relevant calculation points are matched in other images based on a digital image correlation algorithm, and the three-dimensional shapes of three virtual surfaces and two mirror surfaces are reconstructed based on the triangulation principle. The calibration image pairs taken before and after the prism is moved are subjected to corner point detection, and the three-dimensional coordinates of the real points of the feature points and the corresponding virtual points are calculated according to the obtained system parameters and in combination with the PnP algorithm.

[0009] In some embodiments, matching relevant calculated points in other images based on a digital image correlation algorithm further includes: Robust zero-mean normalized square error and related criteria are used to optimize the robust matching of the calculation point and the target point, where the calculation expression is: , Where p is the parameter vector of the motion and deformation from the reference subset to the deformed subset, M represents the total number of calculation points in the image, and are the points in the reference subset respectively. and deformation subset point The gray value of and are the average grayscale of the reference subset and the deformed subset, respectively.

[0010] In some implementations, in step S4, obtaining the reflection transformation matrix of the prism pair specifically includes: A rigid body change algorithm based on SVD decomposition is used to calculate the rotation matrix and translation vector between the corresponding point sets of the two calibration plates to calibrate the right-angle prism. The calibration plates are used to collect images with and without a right-angle prism pair respectively. The collected image pairs are used to identify the corner points based on the classic corner point extraction algorithm, and the three-dimensional coordinates of the real and imaginary points corresponding to the pixel coordinates of the corner points are solved using a triangulation algorithm based on the internal and external parameters of the camera.

[0011] In some implementations, in step S4, obtaining the reflection transformation matrix of the prism pair further includes: Step S41: Assume that the calculated real image point set and virtual image point set are and , calculate the centroid of the two sets of points respectively, where , In the formula, is the weight ratio of the i-th element in the real image point set and the virtual image point set, is the three-dimensional coordinate of the i-th element in the real image point set, is the three-dimensional coordinate of the i-th element in the virtual image point set; Step S42: Point set and Subtract the center of gravity from each element in to obtain two decentralized real image point sets and virtual image point sets Q1 and Q 1v ,in, , In the formula, is the decentralized reference point of the i-th element in the decentralized real image point set Q1, is the decentralized virtual image point set Q 1v The i-th element in is the decentralized target point; Step S43: Q1, Q 1v The elements in the point set form a d×n matrix, and the matrix , , and the weights of each pair of point sets are combined into a diagonal matrix W, where , , Step S44: Calculate the covariance matrix between the two new point sets and the weight matrix, where: , In the formula, For the matrix The transposed matrix of Step S45: Perform SVD decomposition on the covariance matrix H, where: , In the formula, is the left singular vector matrix, is a diagonal matrix of singular values, is the transposed matrix of the right singular vector matrix; Step S46: Calculate the rotation matrix R and the translation vector t, where: , , In the formula, is the centroid of the target point set, is the centroid of the reference point set, is the right singular vector matrix; Step S47: Calibrate the right-angle prism pair based on the rotation matrix and the translation vector to obtain the corresponding reflection transformation matrix.

[0012] In some implementations, in step S4, obtaining the reflection transformation matrix of the mirror surface specifically includes: Reconstruct the three-dimensional point cloud coordinates of the two plane mirror surfaces, and fit the plane equation to the three-dimensional point cloud coordinates to determine the unit normal vector of the plane mirror plane and the vertical distance from the coordinate origin corresponding to the binocular camera position to the plane; A reflection transformation matrix of the mirror is obtained according to the unit normal vector and the vertical distance.

[0013] Compared with the prior art, the technical solution provided in the first aspect of the present application includes at least the following beneficial effects or advantages: 1) Compared with the traditional mirror-assisted MV-DIC measurement system, which is only suitable for places with conventional-sized objects due to the influence of depth of field, the method of the present application adds a pair of right-angle prisms through innovative design and proposes a calibration process for the right-angle prism pair and a reflection transformation matrix solution process for the right-angle prism pair. Specifically, the rotation matrix and translation vector between the corresponding point sets of two sets of high-precision chessboard calibration plates collected by the rigid body change algorithm based on SVD decomposition are calculated to realize the calibration of the right-angle prism and the solution of the reflection transformation matrix. The obtained reflection transformation matrix is ​​used to reflect these virtual surfaces to their real positions, thereby obtaining a 360-degree panoramic view and / or double-sided profile of the test sample in the initial state and the deformed state. When measuring large-sized objects to be measured, the method of the present application can effectively avoid the problem of being unable to focus due to the limitation of limited depth of field, not only compensates for the error caused by the depth of field difference, but also can obtain clear imaging of different parts of the measured object in different depths of field ranges at one time, and realize dynamic high-precision three-dimensional deformation and profile measurement of object parts in different depths of field ranges, with higher image quality and measurement accuracy.

[0014] 2) Compared with the classic panoramic measurement system which requires complex calibration work or cumbersome point cloud registration algorithm, the method of the present application is easier to implement. The method only requires calibrating the mirror-assisted MV-DIC measurement system and using a camera to observe a set of feature point pairs (including real points and virtual points) of a calibration plate with or without a prism to produce a solution of the reflection matrix of the prism pair to realize the conversion and fusion of three-dimensional data obtained from multiple angles, so as to directly realize 360-degree three-dimensional full-surface reconstruction of dynamic scenes.

[0015] 3) The method of the present application can effectively solve the problem that different surfaces cannot be imaged simultaneously with high precision, so that the system can capture three-dimensional data of the sample surface within a larger depth of field, that is, to achieve panoramic three-dimensional reconstruction of larger samples, with a wider range of applications. Panoramic measurements can also be performed on samples of larger surface materials. At the same time, the system device corresponding to the method is more compact and has lower cost than the conventional multi-view DIC system which requires four or more cameras and complex synchronization equipment.

[0016] In a second aspect, the present application further provides an electronic device, including: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can perform the steps of the multi-view digital image correlation measurement method based on mirror-assisted full surface provided in the first aspect above.

[0017] In a third aspect, the present application also provides a computer-readable storage medium having computer program instructions stored thereon, which, when executed by a processor, implement the steps of the multi-view digital image correlation measurement method based on mirror-assisted full surface provided in the first aspect above.

[0018] It can be understood that the beneficial effects of the technical solutions provided in the second and third aspects can be found in the relevant description of the first aspect, and will not be repeated here.

[0019] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0021] Figure 1 is a flow chart of a multi-view digital image correlation measurement method based on mirror-assisted full-surface according to an embodiment of the present application; Figure 2 is a schematic diagram of a mirror-assisted multi-view digital imaging system according to an embodiment of the present application; Figure 3 is a schematic diagram of mirror imaging of a mirror-assisted multi-view digital imaging system according to an embodiment of the present application; Figure 4 is a schematic diagram of subset-based stereo digital image correlation analysis according to an embodiment of the present application; Figure 5 is a schematic diagram of a system capturing an image of a table tennis ball surface according to a prior art method according to an embodiment of the present application; Figure 6 It is a schematic diagram of a system capturing an image of a table tennis surface according to an embodiment of the present application based on a multi-view digital image measurement method for a full surface assisted by a mirror; Figure 7 It is a structural block diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0022] The embodiments of the present application are described in detail below. The embodiments described with reference to the accompanying drawings are exemplary. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0023] It should be noted that, unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein in the specification of this application are only for the purpose of describing specific implementations and are not intended to limit this application. The term "and / or" used herein includes any and all combinations of one or more of the related listed items.

[0024] See also Figures 1 to 5 This embodiment provides a multi-view digital image correlation measurement method based on mirror-assisted full-surface, the method comprising: Step S1: Arrange a set of new mirror-assisted multi-view digital image system with optical structure, produce speckles with black dots on a white background on the plane mirror surface of the system and the surface of the tested sample, and adjust the distance between the prism pairs, the direction of the light source and the intensity of the light source in the optical configuration of the system to obtain a high-definition image of each surface; In this step, a set of novel mirror-assisted multi-view digital image system with optical structure is arranged as the hardware support of the measurement method in this embodiment and the preparation work before measurement. Different from the existing mirror-assisted multi-view digital image correlation system, this embodiment innovatively proposes to introduce a pair of right-angle prisms and a manual displacement fine-tuning platform in the measurement system, and the right-angle prisms are placed opposite to each other on the manual displacement fine-tuning platform. The distance between the right-angle prisms is adjusted by the displacement fine-tuning platform so that the depth of field of the front part of the object closer to the camera can be the same as that of the imaging part of the plane mirror, thereby obtaining clear imaging of different parts of the measured object within different depths of field at the same time.

[0025] For example, see Figure 2 The mirror-assisted multi-view digital image system corresponding to the present implementation method mainly includes a conventional binocular stereo digital image correlation system (two synchronous cameras 1), a pair of right-angle prisms 2 for adjusting the depth of field of the front surface image, two plane mirrors for panoramic image imaging (a first plane mirror 3 and a second plane mirror 4), and a displacement fine-tuning platform 6. In order to increase the overlapping area of ​​the field of view of the dual synchronous cameras 1, the cameras are arranged vertically instead of being arranged horizontally as usual. The first plane mirror 3 and the second plane mirror 4 can be pre-made with black / white speckle patterns in the top area and firmly installed behind the test sample 5. Of course, the speckle patterns can also be pre-arranged in other edge areas of the first plane mirror 3 and the second plane mirror 4, and can be selectively adjusted according to actual needs.

[0026] Optionally, one edge of the first plane mirror 3 and the second plane mirror 4 contacts each other to form an angle, and the angle can be set to an angle of about 120°. Of course, it can also be other angles, such as 100°, 110° or 125°, etc., and can be selectively adjusted according to actual needs. The two plane mirrors are placed and fixed behind the test sample 5 to reflect the rear surface contour of the test sample 5. The displacement fine-tuning platform 6 can be a manual or automatic adjustment platform. For example, the displacement fine-tuning platform 6 can be a slide rail set at the bottom and a support structure slidably connected to the top surface of the slide rail. The right-angle prisms 2 are fixed at intervals on the surface of the supporting structure of the displacement fine-tuning platform 6, so that the displacement fine-tuning platform 6 can adjust the position of the right-angle prisms 2. At the same time, the distance between the right-angle prisms 2 is adjusted by the displacement fine-tuning platform 6 so that the depth of field of the object directly in front of the camera that is closer to the camera can be the same as the imaging part of the plane mirror, so that clear imaging of different parts of the measured object within different depths of field can be obtained at the same time. The two synchronous cameras 1 in the binocular stereo digital image correlation system can be arranged on a tripod vertically instead of horizontally as usual, so as to increase the overlapping area of ​​the field of view of the two cameras.

[0027] Of course, it should be noted that the hardware used in the above system can be obtained from the market, and the functional principles of its individual hardware are not described here. After simply arranging the mirror-assisted multi-view digital image system, the distance between the prism pairs, the direction of the light source and the intensity of the light source in the optical configuration of the system are preliminarily adjusted to ensure high-definition images of each surface. Finally, the new mirror-assisted perspective digital image correlation system can simultaneously capture two views of the back of the sample and one view formed by multiple reflections of the front surface of the sample by the right-angle prism pair. Through this arrangement, the new mirror-assisted multi-view digital image correlation system can simultaneously capture three effective surfaces of interest (SOI1-SOI3) within the same depth of field in a single shot, namely the virtual front surface and the two rear surfaces reflected by the plane mirror. In addition, in order to calibrate the reflection transformation matrix of the plane mirror, matte paint and markers can be used to make white-bottomed black spot speckles at appropriate locations on the plane mirror surface. The speckles on the mirror surface and the three surfaces of the sample are synchronously captured by the lens into the same image.

[0028] Understandably, see Figure 3 , the system can be observed as an equivalent configuration: virtual objects, virtual objects through the mirror reflection of real objects To indicate that, in this equivalent configuration, the virtual object and Relative to the mirror Symmetry, virtual objects Relative to the prism and the real object Moving back. The three virtual surfaces and the speckle patterns S1 and S2 on the mirror are captured simultaneously in one shot by a real stereo digital image correlation system synchronized camera. This arrangement has two significant advantages: (1) the three virtual sample surfaces are perpendicular to the camera plane and can therefore be imaged with maximum effective spatial resolution; (2) all SOIs are distributed within the same depth of field, which ensures clear imaging of all SOIs.

[0029] Step S2: Based on the system, the sample images of the tested sample before and after deformation in the mirror and the prism and the calibration image pair for calibrating system parameters are collected; In this step, in order to calibrate the mirror-assisted multi-view digital imaging system and reconstruct the surface part, the sample images of the test sample in the mirror before and after deformation are collected, and after the sample images are collected, multiple groups of calibration image pairs are collected when the calibration plate is placed behind the prism, the position of the calibration plate is kept unchanged after the prism is removed, and the plane mirror is removed and the calibration plate is placed in different postures. Specifically, the image acquisition first collects the images of the three surfaces of the test sample in the mirror before and after deformation, and then after collecting the sample images, a group of images are collected by placing a high-precision circular calibration plate behind the prism, and then a group of images are collected by removing the prism while keeping the calibration plate in place. Finally, the mirror is removed and the calibration plate is placed in the binocular vision system. High-definition acquisition can be no less than 20 groups of calibration image pairs under different postures for calibration system.

[0030] Step S3: Based on the calibration image pair, the intrinsic parameters and extrinsic parameters of the system imaging are accurately calibrated using a stereoscopic vision calibration method, and the three-dimensional shapes of the three virtual surfaces, the two mirror surfaces, and the three-dimensional coordinates of the real points and the corresponding virtual points of the calibration plate are reconstructed based on the triangulation principle using the collected sample images and the obtained system parameters; In this step, the imaging system needs to be calibrated and the surface parts need to be reconstructed. The region of interest is selected from the speckle pattern area on the plane mirror, and a set of calculation points are determined in the region of interest. The relevant calculation points are matched in other images based on the digital image correlation algorithm, and the three-dimensional shapes of the three virtual surfaces and two mirror surfaces are reconstructed based on the triangulation principle. The calibration image pairs taken before and after the moving prism are used for corner point detection. The three-dimensional coordinates of the real points of the feature points and the corresponding virtual points are calculated based on the obtained system parameters and combined with the PnP algorithm.

[0031] Specifically, the calibration image pairs are used to accurately calibrate the intrinsic parameters and extrinsic parameters (relative posture R / T) with the help of mature stereo vision calibration methods. Using the collected sample images and the obtained system parameters, a ROI is selected in each SOI in the upper reference image and a set of calculation points is specifically specified in each ROI. The relevant calculation points are matched in other images based on the digital image correlation (DIC) algorithm. The three-dimensional shapes of three virtual surfaces and two mirrors are reconstructed using the triangulation principle. Corner point detection is performed on the image pairs taken before and after the moving prism. According to the obtained system parameters, the three-dimensional coordinates of the real points of the feature points and the corresponding virtual points are calculated in combination with the PnP algorithm.

[0032] It needs to be explained that for the stereo vision calibration method, the stereo vision calibration (Stereo Vision Calibration) method is a key preprocessing step in the binocular camera system. It aims to obtain the internal parameters of the camera (intrinsic parameters) and the relative position relationship between the two cameras (extrinsic parameters) through calibration, so as to provide a basis for subsequent depth calculation and three-dimensional reconstruction. When calibrating the imaging system, the steps may include: using the same calibration plate to simultaneously shoot a binocular image pair (the image pair must include the same position of the calibration plate), detecting the coordinates of the corner points of the calibration plate in the two cameras, and calculating the rotation matrix and translation vector between the two cameras through the matching corner point coordinates, jointly optimizing the intrinsic and extrinsic parameters, and minimizing the reprojection error.

[0033] It is also necessary to explain that the digital image correlation (DIC) algorithm is a non-contact optical measurement technology that calculates the displacement field and strain field by analyzing the digital images of the object surface before and after deformation. It is mainly used in material mechanics, structural engineering, biomedicine and other fields, and is particularly suitable for measuring complex deformation, dynamic loading and high-resolution strain distribution.

[0034] As for the PnP algorithm, the PnP algorithm (Perspective-n-Point) is a classic problem in computer vision and three-dimensional geometry. It is used to solve the camera's pose (position and direction) through the correspondence between known 3D space points and their projection points in the 2D image. It is assumed that there are several known 3D points in the scene (such as the key points of the object). These points are projected onto the 2D image plane after being photographed by the camera. In this way, based on these 3D-2D matching point pairs, the camera's position (translation vector t) and orientation (rotation matrix R) in the world coordinate system are calculated.

[0035] See also Figure 4For each SOI, we first select a region of interest (ROI) of appropriate size in the reference image above. After the 3D digital image correlation system is calibrated, the 3D shape reconstruction of each ROI can be achieved by using the well-established standard DIC algorithm based on image subsets and classical triangulation P (X, Y, Z). Taking one of the ROI measurements as an example, it should be noted that the ROI measurement process for other surfaces is exactly the same. The measurement first specifies the calculation point grid in the ROI. For the calculation point P1 (x1, y1) of the given camera 1 image acquisition, the DIC method is used at P2 (x2, y2) matches the reference image (stereo matching), matches the deformed image (time matching) at P1'(x1', y1'), matches the deformed image (time matching) at P2'(x2', y2'), and achieves matching through robust zero mean normalized square difference (ZNSSD) related criterion optimization. Specifically, robust matching between the calculation point and the target point is performed using robust zero mean normalized square difference and related criterion optimization, where the calculation expression is: , Where p is the parameter vector of the motion and deformation from the reference subset to the deformed subset, M represents the total number of calculation points in the image, and are the points in the reference subset respectively. and deformation subset point The gray value of and are the average grayscale of the reference subset and the deformed subset, respectively. According to the reliability-oriented displacement tracking strategy, the calculation path of IC-GN is guided, and all the specified calculation points can be robustly matched with the target points.

[0036] After finding the correspondence of all calculated points in all image pairs, the 3D point cloud in the reference state and the deformed state is reconstructed using the triangulation method. The full-field displacement and strain distribution are obtained by comparing the 3D coordinates of each point before and after deformation. The above process is repeated for all four ROIs, and the 3D shape of the virtual surface and the mirror surface can be reconstructed in each state.

[0037] Step S4: performing plane fitting on the reconstructed mirror surface, and calculating the three-dimensional coordinates of the real points and imaginary points corresponding to the coordinates of the corner points in the calibration plate using the image pairs of the calibration plate collected before and after the prism is removed, so as to obtain the reflection transformation matrix of the mirror surface and the prism pair; In this step, in order to reconstruct the 360-degree full surface of the sample, each part of the sample surface should be reconstructed at its real position. Since the images captured by the camera in the plane mirror and the right-angle prism are virtual, they can all be transformed into their real positions through reflection transformation to obtain the world coordinates of the sample in real space.

[0038] In some embodiments, for the specular reflection of a plane mirror, consider any actual point P on the rear surface of the real sample r , due to the obstruction of the camera's line of sight, the 3D-DIC system cannot directly measure this point. v Can be directly observed and measured by 3D-DIC system. r and P v According to the existing derivation, the coordinate transformation between can be expressed as: , Here, the matrix T r is the reflection transformation matrix of the plane mirror, From the origin O to the actual point P of the real sample r The vector of From the origin O to the reflection point P in the plane mirror v The vector of the plane mirror is n, and d is the distance from the plane mirror to the origin O. It can be seen that the key to realizing the reflection transformation of the plane mirror is to obtain the unit normal vector n corresponding to the plane mirror plane and the distance d from the plane to the origin O of the coordinate system. These two key parameters can be estimated in several ways according to existing literature.

[0039] It is understandable that among these methods, pre-creating speckle patterns on the mirror surface to calibrate the reflection change matrix is ​​still the most commonly used method in mirror-assisted multi-view digital image correlation experiments. When using the black / white speckle pre-made on the plane mirror to calibrate the reflection transformation matrix, it is necessary to first use digital image correlation to calculate the plane mirror speckle collected in the same frame as the three specimen surfaces, reconstruct the three-dimensional point cloud coordinates of the two plane mirror surfaces, and fit the plane equation to the point cloud: ; Among them, A, B, and C are the parameters of the mirror fitting, which can be obtained by multivariate linear regression of the points in the ROI. Once these parameters are determined, n and d can be expressed as: According to the obtained n and d, the reflection transformation matrix is ​​obtained according to the reflection transformation relationship.

[0040] In some embodiments, the front surface of the object is reflected multiple times by a pair of right-angle prisms to finally form a virtual image moving backwards. The sampling method above is computationally complex. This embodiment innovatively calibrates the right-angle prism by calculating the rotation matrix and translation vector between the corresponding point sets of two sets of high-precision chessboard calibration plates collected by a rigid body change algorithm based on SVD decomposition. The high-precision chessboard calibration plate is used to collect images with and without a right-angle prism pair. The collected image pairs are identified by a classic corner point extraction algorithm. The three-dimensional coordinates corresponding to the pixel coordinates of the corner points are solved by using the calculated internal and external parameters of the camera using a triangulation algorithm. The specific calculation process of the reflection transformation matrix of the prism pair includes the following steps: Step S41: Assume that the calculated real image point set and virtual image point set are and , calculate the centroid of the two sets of points respectively, where In the formula, is the weight ratio of the i-th element in the real image point set and the virtual image point set, is the three-dimensional coordinate of the i-th element in the real image point set, is the three-dimensional coordinate of the i-th element in the virtual image point set; Step S42: Point set and Subtract the center of gravity from each element in to obtain two decentralized real image point sets and virtual image point sets Q1 and Q 1v ,in, , In the formula, is the decentralized reference point of the i-th element in the decentralized real image point set Q1, is the decentralized virtual image point set Q 1v The i-th element in is the decentralized target point; Step S43: Q1, Q 1v The elements in the point set form a d×n matrix, and the matrix , , and the weights of each pair of point sets are combined into a diagonal matrix W, where , , Step S44: Calculate the covariance matrix between the two new point sets and the weight matrix, where: , In the formula, For the matrix The transposed matrix of Step S45: Perform SVD decomposition on the covariance matrix H, where: In the formula, is the left singular vector matrix, is a diagonal matrix of singular values, is the transposed matrix of the right singular vector matrix; Step S46: Calculate the rotation matrix R and the translation vector t, where: , , In the formula, is the centroid of the target point set, is the centroid of the reference point set, is the right singular vector matrix; Step S47: Calibrate the right-angle prism pair based on the rotation matrix and the translation vector to obtain the corresponding reflection transformation matrix.

[0041] Step S5: Based on the reflection transformation matrix, the virtual surface is reflected to the corresponding real position to obtain a 360° panoramic and / or double-sided profile of the tested sample in the initial state and the deformed state, the constructed initial state profile is subtracted from the deformed state profile to obtain the displacement field in each state, and the full-field 360° panoramic and / or double-sided strain components are calculated from the displacement field based on a point-by-point least squares fitting algorithm.

[0042] In some embodiments, the virtual surfaces are reflected to their real positions using the obtained reflection transformation matrix, and a 360-degree panoramic view and / or two-sided profile of the test specimen in the initial state and the deformed state can be obtained. The displacement field in each state is obtained by subtracting the constructed reference state profile from the deformed state profile, and then the full-field 360-degree panoramic view and / or two-sided strain components can be calculated from the displacement field using a point-by-point least squares fitting algorithm.

[0043] In the method steps provided in the above embodiments, compared with the traditional mirror-assisted MV-DIC measurement system, which is only suitable for places with conventional-sized objects due to the influence of depth of field, the method of the present application adds a pair of right-angle prisms through innovative design and proposes a calibration process for the right-angle prism pair and a reflection transformation matrix solution process for the right-angle prism pair. Specifically, the rotation matrix and translation vector between the corresponding point sets of two sets of high-precision chessboard calibration plates collected by the rigid body change algorithm based on SVD decomposition are calculated to realize the calibration of the right-angle prism and the solution of the reflection transformation matrix. The obtained reflection transformation matrix is ​​used to reflect these virtual surfaces to their real positions, thereby obtaining a 360-degree panoramic view and / or a double-sided profile of the test sample in the initial state and the deformed state. When measuring a larger object to be measured, the method of the present application can effectively avoid the problem of being unable to focus due to the limitation of limited depth of field, and can simultaneously obtain clear imaging of different parts of the object to be measured within different depths of field at one time, with higher image quality and measurement accuracy.

[0044] For example, see Figure 5 and Figure 6 , Figure 5 A schematic diagram showing a system for capturing an image of a table tennis ball surface using a prior art method is shown; Figure 6 FIG. 1 shows a schematic diagram of a system for capturing an image of a table tennis surface using a multi-view digital image measurement method based on a mirror-assisted full surface provided by this embodiment; FIG. Figure 5 In the image, the ping-pong ball is blurred in the middle. Due to the limitation of the limited depth of field of the synchronous camera, the plane mirror-assisted multi-view digital image correlation measurement system in the prior art cannot ensure that objects or parts of objects with different depths of field in the field of view can obtain high image quality at the same time. Decorrelation occurs during the matching process, resulting in a decrease in measurement accuracy and speed. Figure 6 In the method, a pair of right-angle prisms is added to form a mirror-assisted multi-view digital image correlation system with adjustable depth of field. The method can convert the measured objects or different parts of the objects within different depths of field of the camera into the same depth of field, thereby realizing simultaneous high-definition imaging of the measured objects or different parts of the objects with different depths of field, and realizing high-precision real-time measurement of the three-dimensional deformation and contour of objects or different parts of the objects with different depths of field.

[0045] Moreover, the above-mentioned embodiment method is easier to implement than the classic panoramic measurement system which requires complex calibration work or cumbersome point cloud registration algorithm. The method only needs to calibrate the mirror-assisted MV-DIC measurement system and use a camera to observe a set of characteristic point pairs (including real points and virtual points) of a calibration plate with or without a prism to produce a solution of the reflection matrix of the prism pair to realize the conversion and fusion of three-dimensional data obtained from multiple angles, so as to directly realize 360-degree three-dimensional full-surface reconstruction of dynamic scenes; at the same time, it can effectively solve the problem that different surfaces cannot be imaged with high precision at the same time, so that the system can capture three-dimensional data of the sample surface within a larger depth of field, that is, to realize panoramic three-dimensional reconstruction of larger samples, and has a wider range of applications. Panoramic measurement can also be performed on samples of larger-sized surface materials. At the same time, the system device corresponding to the method is more compact and has lower cost than the conventional multi-view DIC system which requires four or more cameras and complex synchronization equipment.

[0046] See also Figure 7 , Figure 7 5 is a block diagram of a structure of an electronic device provided in an embodiment of the present application. The server 500 of the electronic device includes: a processor 501, a memory 502, and a computer program 503 stored in the memory 502 and executable on the processor 501, such as a program portion of a multi-view digital image correlation measurement method based on a mirror-assisted full surface. When the processor 501 executes the computer program 503, some steps of a multi-view digital image correlation measurement method based on a mirror-assisted full surface in the above-mentioned embodiments are implemented, such as Figure 1 For details of the steps S10 to S50 of the corresponding embodiment, please refer to Figure 1 The relevant descriptions in the corresponding embodiments are not repeated here.

[0047] Exemplarily, the computer program 503 may be divided into one or more units, one or more units are stored in the memory 502, and are executed by the processor 501 to complete the technical solution provided in the above embodiment. One or more units may be a series of computer program instruction segments that can complete specific functions, and the instruction segments are used to describe the execution process of the computer program 503 in the server 500.

[0048] The electronic device may include, but is not limited to, a processor 501 and a memory 502. Those skilled in the art will appreciate that Figure 7 It is only an example of the server 500 in the electronic device and does not constitute a limitation of the server 500. It may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, the turntable terminal device may also include an input and output terminal device, a network access terminal device, a bus, etc.

[0049] The processor 501 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0050] The memory 502 may be an internal storage unit of the server 500, such as a hard disk or memory of the server 500. The memory 502 may also be an external storage terminal device of the server 500, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the server 500. Further, the memory 502 may also include both an internal storage unit of the server 500 and an external storage terminal device. The memory 502 is used to store computer programs and other programs and data required by the turntable terminal device. The memory 502 may also be used to temporarily store data that has been output or is to be output.

[0051] In some embodiments, a computer-readable storage medium is also provided, which stores a computer program. When the computer program is executed by a processor, the steps of a multi-view digital image correlation measurement method based on mirror-assisted full surface as described in the above embodiment are implemented.

[0052] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0053] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Among them, the computer-readable storage medium can be non-volatile or volatile. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable storage medium may include: any entity or device capable of carrying computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electrical carrier signal, telecommunication signal and software distribution medium, etc.

[0054] The terms "first", "second", "third", etc. in the specification and claims of the present application and the drawings are used to distinguish different objects rather than to describe a specific order. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a series of steps or units are included, or optionally, steps or units not listed are included, or optionally, other steps or units inherent to these processes, methods, products or devices are included.

[0055] Only the part relevant to the present application is shown in the accompanying drawings, but not all of the content. Before discussing the exemplary embodiments in more detail, it should be mentioned that some exemplary embodiments are described as processing or methods depicted as flow charts. Although the flow chart describes each operation (or step) as a sequential process, many of the operations therein can be implemented in parallel, concurrently or simultaneously. In addition, the order of each operation can be rearranged. When its operation is completed, the process can be terminated, but it can also have additional steps not included in the accompanying drawings. The process can correspond to a method, function, procedure, subroutine, subprogram, etc.

[0056] The terms "component", "module", "system", "unit", etc. used in this specification are used to represent computer-related entities, hardware, firmware, a combination of hardware and software, software, or software in execution. For example, a unit can be, but is not limited to, a process running on a processor, a processor, an object, an executable file, an execution thread, a program, and / or distributed between two or more computers. In addition, these units can be executed from various computer-readable media having various data structures stored thereon. Units can communicate through local and / or remote processes, for example, based on signals having one or more data packets (e.g., data from a second unit interacting with another unit in a local system, a distributed system, and / or a network. For example, the Internet interacts with other systems via signals).

[0057] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "illustrative embodiments", "examples", "specific examples", or "some examples" means that the specific features, structures, materials, or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example.

[0058] Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Mentioning "embodiment" in this article means that the specific features, structures or characteristics described in conjunction with the embodiment may be included in at least one embodiment of the present embodiment application. The appearance of this phrase in various positions in the specification is not necessarily the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It can be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0059] Although the embodiments of the present application have been shown and described, those skilled in the art will appreciate that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present application, and that the scope of the present application is defined by the claims and their equivalents.

[0060] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the disclosure disclosed herein. The present application is intended to cover any modification, use or adaptation of the present application, which follows the general principles of the present application and includes common knowledge or customary techniques in the art that are not disclosed in the present application. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present application are indicated by the following claims.

Claims

1. A multi-view digital image correlation measurement method based on mirror-assisted full-surface, characterized in that: include: Step S1: Arrange a set of new mirror-assisted multi-view digital image system with optical structure, produce speckles with black dots on a white background on the plane mirror surface of the system and the surface of the tested sample, and adjust the distance between the prism pairs, the direction of the light source and the intensity of the light source in the optical configuration of the system to obtain a high-definition image of each surface; Step S2: Based on the system, the sample images of the tested sample before and after deformation in the mirror and the prism and the calibration image pair for calibrating system parameters are collected; Step S3: Based on the calibration image pair, the intrinsic parameters and extrinsic parameters of the system imaging are accurately calibrated using a stereoscopic vision calibration method, and the three-dimensional shapes of the three virtual surfaces, the two mirror surfaces, and the three-dimensional coordinates of the real points and the corresponding virtual points of the calibration plate are reconstructed based on the triangulation principle using the collected sample images and the obtained system parameters; Step S4: performing plane fitting on the reconstructed mirror surface, and calculating the three-dimensional coordinates of the real points and imaginary points corresponding to the coordinates of the corner points in the calibration plate using the image pairs of the calibration plate collected before and after the prism is removed, so as to obtain the reflection transformation matrix of the mirror surface and the prism pair; Step S5: Based on the reflection transformation matrix, the virtual surface is reflected to the corresponding real position to obtain a 360° panoramic and / or double-sided profile of the tested sample in the initial state and the deformed state, the constructed initial state profile is subtracted from the deformed state profile to obtain the displacement field in each state, and the full-field 360° panoramic and / or double-sided strain components are calculated from the displacement field based on a point-by-point least squares fitting algorithm.

2. The multi-view digital image correlation measurement method based on mirror-assisted full surface according to claim 1, characterized in that: In step S1, the novel mirror-assisted multi-view digital image system specifically includes a pair of binocular cameras for capturing digital image pairs, two plane mirrors for panoramic image imaging, a pair of right-angle prism pairs for adjusting the depth of field of the front surface image, and a displacement fine-tuning platform for adjusting the position of the right-angle prism pairs.

3. The multi-view digital image correlation measurement method based on mirror-assisted full-surface according to claim 1, characterized in that: The step S2 specifically includes: The sample images of the tested sample before and after deformation are collected in the mirror surface, and after the sample images are collected, multiple sets of calibration image pairs are collected respectively when the calibration plate is placed behind the prism, the prism is removed and the position of the calibration plate is kept unchanged, and the plane mirror is removed and the calibration plate is placed in different postures.

4. The multi-view digital image correlation measurement method based on mirror-assisted full-surface according to claim 1, characterized in that: The step S3 specifically includes: An area of ​​interest is selected from the speckle pattern area on the plane mirror, and a group of calculation points are determined in the area of ​​interest. The relevant calculation points are matched in other images based on a digital image correlation algorithm, and the three-dimensional shapes of three virtual surfaces and two mirror surfaces are reconstructed based on the triangulation principle. The calibration image pairs taken before and after the prism is moved are subjected to corner point detection, and the three-dimensional coordinates of the real points of the feature points and the corresponding virtual points are calculated according to the obtained system parameters and in combination with the PnP algorithm.

5. The multi-view digital image correlation measurement method based on mirror-assisted full surface according to claim 4, characterized in that: The matching of relevant calculation points in other images based on the digital image correlation algorithm also includes: Robust zero-mean normalized square error and related criteria are used to optimize the robust matching of the calculation point and the target point, where the calculation expression is: ， Where p is the parameter vector of the motion and deformation from the reference subset to the deformed subset, M represents the total number of calculation points in the image, and are the points in the reference subset respectively. and deformation subset point The gray value of and are the average grayscale of the reference subset and the deformed subset, respectively.

6. The multi-view digital image correlation measurement method based on mirror-assisted full-surface according to claim 1, characterized in that: In step S4, obtaining the reflection transformation matrix of the prism pair specifically includes: A rigid body change algorithm based on SVD decomposition is used to calculate the rotation matrix and translation vector between the corresponding point sets of the two calibration plates to calibrate the right-angle prism. The calibration plates are used to collect images with and without a right-angle prism pair respectively. The collected image pairs are used to identify the corner points based on the classic corner point extraction algorithm, and the three-dimensional coordinates of the real and imaginary points corresponding to the pixel coordinates of the corner points are solved using a triangulation algorithm based on the internal and external parameters of the camera.

7. The multi-view digital image correlation measurement method based on mirror-assisted full surface according to claim 6, characterized in that: In step S4, obtaining the reflection transformation matrix of the prism pair further includes: Step S41: Assume that the calculated real image point set and virtual image point set are and , calculate the centroid of the two sets of points respectively, where ， In the formula, is the weight ratio of the i-th element in the real image point set and the virtual image point set, is the three-dimensional coordinate of the i-th element in the real image point set, is the three-dimensional coordinate of the i-th element in the virtual image point set; Step S42: Point set and Subtract the center of gravity from each element in to obtain two decentralized real image point sets and virtual image point sets Q1 and Q 1v ,in, ， In the formula, is the decentralized reference point of the i-th element in the decentralized real image point set Q1, is the decentralized virtual image point set Q 1v The i-th element in is the decentralized target point; Step S43: Q1, Q 1v The elements in the point set form a d×n matrix, and the matrix , , and the weights of each pair of point sets are combined into a diagonal matrix W, where ， ， Step S44: Calculate the covariance matrix between the two new point sets and the weight matrix, where: ， In the formula, For the matrix The transposed matrix of Step S45: Perform SVD decomposition on the covariance matrix H, where: ， In the formula, is the left singular vector matrix, is a diagonal matrix of singular values, is the transposed matrix of the right singular vector matrix; Step S46: Calculate the rotation matrix R and the translation vector t, where: ， ， In the formula, is the centroid of the target point set, is the centroid of the reference point set, is the right singular vector matrix; Step S47: Calibrate the right-angle prism pair based on the rotation matrix and the translation vector to obtain the corresponding reflection transformation matrix.

8. The multi-view digital image correlation measurement method based on mirror-assisted full-surface according to claim 1, characterized in that: In step S4, obtaining the reflection transformation matrix of the mirror surface specifically includes: Reconstruct the three-dimensional point cloud coordinates of the two plane mirror surfaces, and fit the plane equation to the three-dimensional point cloud coordinates to determine the unit normal vector of the plane mirror plane and the vertical distance from the coordinate origin corresponding to the binocular camera position to the plane; A reflection transformation matrix of the mirror is obtained according to the unit normal vector and the vertical distance.

9. An electronic device, characterized in that: include: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can perform the steps of a multi-view digital image correlation measurement method based on mirror-assisted full surface as described in any one of claims 1-8.

10. A computer-readable storage medium having computer program instructions stored thereon, characterized in that: When the computer program instructions are executed by the processor, the steps of a multi-view digital image correlation measurement method based on mirror-assisted full surface as described in any one of claims 1 to 8 are implemented.

Images