Image processing apparatus, method and medium for measuring strain of an object

Abstract

An image processing method for measuring object displacement is provided. The method includes acquiring a first sequence of images and a second sequence of images, two adjacent images of the first sequence of images including a first overlap portion, two adjacent images of the second sequence of images including a second overlap portion, the first sequence of images corresponding to a first three-dimensional (3D) surface including a curved surface on an object in a first state, the second sequence of images corresponding to a second 3D surface on the object in a second state. The method also includes deblurring the first sequence of images and the second sequence of images based on a blind deconvolution method to obtain sharp focal plane images, stitching the sharpened first sequence of images and the sharpened second sequence of images into a first 3D image and a second 3D image based on camera pose estimation performed by solving a perspective-n-point (PnP) problem using a refined robustly weighted Levenberg-Marquardt (RRWLM) algorithm. The method also includes forming a first two-dimensional (2D) image and a second 2D image by unwrapping the first 3D image and the second 3D image, respectively, and generating a displacement map image from the first 2D and second 2D images by performing a two-dimensional digital image correlation (DIC) method.

Description

Technical Field

[0001] The present invention generally relates to devices and methods based on a fusion-based digital image correlation framework suitable for strain measurement. Background Technology

[0002] Strain measurement of materials subjected to loads or mechanical damage is an essential task in various industrial applications. For strain measurement, in addition to the widely used point-by-point strain gauge technique, digital image correlation (DIC), as a non-contact and non-interfering optical technique, has attracted considerable attention due to its ability to provide a full-field strain distribution of a surface using simple test assemblies. DIC is performed by comparing digital grayscale intensity images of the surface before and after deformation, taking the derivative of the pixel displacement as the degree of strain of the pixel.

[0003] In various applications, full-view 2D (DIC) DIC analysis of curved surfaces of large 3D objects is of great interest. DIC places stringent requirements on images captured before and after distortion to obtain accurate pixel displacements, such as image resolution, image registration, and camera lens distortion compensation, because the displacement of most industrial materials under strain is typically very subtle. Therefore, the requirements of the target scenario lead to two major limitations in existing 2D DIC analysis. First, DIC methods are usually limited to 2D planar object surfaces rather than 3D curved surfaces. Second, because DIC analysis has very high requirements for image pixel resolution, DIC methods are usually limited to small surfaces. Currently, much effort has been made to develop 3D DIC methods based on binocular stereo vision or multi-camera system environments involving precise calibration and image stitching, but these methods are difficult to implement in various scenarios.

[0004] This work stitches together images captured by a single, ordinary moving camera, rather than a well-calibrated multi-camera system. Summary of the Invention

[0005] In our proposed framework, we combine image fusion and camera pose estimation to automatically stitch together a large number of images of a measured curved surface. This work extends the application of image fusion and stitching to strain measurement in mechanical engineering.

[0006] The proposed framework decouples the image fusion problem into a series of well-known PnP problems, which have been extensively explored using both non-iterative and iterative methods. Some of these problems involve additional outlier rejection or contain observational uncertainties. The proposed image fusion method, combining the principles of bundle adjustment and iterative PnP, outperforms existing PnP methods and achieves suitable fusion accuracy.

[0007] This disclosure addresses the problem of enabling two-dimensional digital image correlation (DIC) for strain measurement of large 3D objects with curved surfaces. Obtaining a full-field-of-view qualified image of the surface required for DIC is challenging due to the blurring, distortion, and narrow field of view of the surface that a single image can cover. To overcome this problem, we propose an end-to-end DIC framework incorporating image fusion principles to achieve full-field-of-view strain measurement on curved surfaces. Taking a series of blurred images as input, we first recover sharp images using blind deconvolution, then project the recovered sharp images onto the curved surface using the camera pose estimated by our proposed perspective n-point (PnP) method, known as RRWLM. The images on the curved surface are stitched together and then unfolded for strain analysis using DIC. Numerical experiments using RRWLM validate our framework and compare it with existing methods.

[0008] Some embodiments of this invention propose an end-to-end fusion-based DIC framework to enable strain measurements along the curved surfaces of large 3D objects using a single camera. We first use a moving camera on the large 3D surface to acquire a series of blurred 2D images of the surface texture. Then, using these blurred observations, we recover the corresponding sharp images using blind deconvolution, and project the pixels from the sharp images onto the 3D surface using the camera pose estimated by our proposed robust perspective n-point (PnP) method for image fusion. The stitched 3D surface images before and after deformation are unfolded into two 2D fused images, converting the 3D strain measurements into 2D strain measurements for further DIC analysis. As previously mentioned, since displacements are very subtle (typically subpixel-level), their derivatives and corresponding strains are highly sensitive to the quality of the fused image. Therefore, the biggest challenge in the pipeline is the stringent accuracy requirements (at least subpixel-level) for image fusion methods to achieve accurate strain measurements.

[0009] Furthermore, according to some embodiments of the present invention, an image processing apparatus for measuring the strain of an object is provided. The image processing apparatus includes: an interface configured to acquire a first sequence of images and a second sequence of images, wherein two adjacent images of the first sequence of images include a first overlapping portion, and two adjacent images of the second sequence of images include a second overlapping portion, wherein the first sequence of images corresponds to a first three-dimensional (3D) surface on the object in a first state, and the second sequence of images corresponds to a second 3D surface on the object in a second state (the first state may be referred to as an initial condition, and the second state may be a state after an operation for a period of time); a memory for storing a computer-executable program, the computer-executable program including image deblurring methods, pose refinement methods, fusion-based correlation methods, strain measurement methods, and image correction methods; and a processor configured to execute... The computer-executable program is described, wherein the processor performs the following steps: deblurring the first sequence of images and the second sequence of images based on a blind kernel deconvolution method to obtain a sharp focal plane image; stitching the sharpened first sequence of images and the sharpened second sequence of images into a first sharp 3D image and a second sharp 3D image based on camera pose estimation by solving the perspective n-point (PnP) problem using a refined robust weighted Levenberg-Marquardt (RRWLM) algorithm; forming a first two-dimensional (2D) image and a second 2D image by unfolding the first sharp 3D image and the second sharp 3D image respectively; and generating displacement (strain) maps from the first 2D image and the second 2D image by performing a two-dimensional digital image correction (DIC) method.

[0010] Some embodiments of the present invention provide an end-to-end DIC framework incorporating image fusion for strain measurement pipelines. It extends the application of DIC-based strain measurement to curved surfaces of large 3D objects.

[0011] Furthermore, embodiments of the present invention provide an image processing method for measuring the strain of an object. This image processing method may include: acquiring a first sequence of images and a second sequence of images, wherein two adjacent images of the first sequence of images include a first overlapping portion, and two adjacent images of the second sequence of images include a second overlapping portion; the first sequence of images corresponds to a first three-dimensional (3D) surface on the object in a first state, and the second sequence of images corresponds to a second 3D surface on the object in a second state; deblurring the first sequence of images and the second sequence of images based on blind deconvolution to obtain a sharp focal plane image; stitching the sharpened first sequence of images and the sharpened second sequence of images into a first sharp 3D image and a second sharp 3D image based on camera pose estimation performed by solving the perspective n-point (PnP) problem using the refined robust weighted Levenberg-Marquardt (RRWLM) algorithm; forming a first two-dimensional (2D) image and a second 2D image by unfolding the first sharp 3D image and the second sharp 3D image respectively; and generating a displacement (strain) image from the first 2D image and the second 2D image by performing two-dimensional digital image correction (DIC).

[0012] However, some embodiments of the present invention provide a non-transitory computer-readable medium comprising program instructions that cause a computer to perform a method. In this context, the method may include the following steps: acquiring a first sequence of images and a second sequence of images, wherein two adjacent images of the first sequence of images include a first overlapping portion, and two adjacent images of the second sequence of images include a second overlapping portion, wherein the first sequence of images corresponds to a first three-dimensional (3D) surface on the object in a first state, and the second sequence of images corresponds to a second 3D surface on the object in a second state; deblurring the first sequence of images and the second sequence of images based on blind deconvolution to obtain a sharp focal plane image; stitching the sharpened first sequence of images and the sharpened second sequence of images into a first sharp 3D image and a second sharp 3D image, respectively, based on camera pose estimation by solving the perspective n-point (PnP) problem using the refined robust weighted Levenberg-Marquardt (RRWLM) algorithm; forming a first two-dimensional (2D) image and a second 2D image by unfolding the first sharp 3D image and the second sharp 3D image, respectively; and generating a displacement (strain) image from the first 2D image and the second 2D image by performing a two-dimensional digital image correction (DIC) method.

[0013] Another embodiment of the present invention proposes a two-stage method for image fusion based on the principles of PnP and bundle adjustment. Our method outperforms existing methods and achieves suitable image fusion accuracy for strain measurement through DIC analysis.

[0014] The accompanying drawings included in this application are used to provide a further understanding of the invention, illustrate embodiments of the invention, and, together with the specification, serve to explain the principles of the invention. Attached Figure Description

[0015] Figure 1 An example illustrating an image processing apparatus according to an embodiment of the present invention is shown.

[0016] Figure 2 A block diagram illustrating image processing steps for generating strain images according to an embodiment of the present invention is shown.

[0017] Figure 3A A block diagram is shown illustrating an image deblurring module used in an image processing apparatus according to an embodiment of the present invention.

[0018] Figure 3B A block diagram is shown illustrating an image stitching module used in an image processing apparatus according to an embodiment of the present invention.

[0019] Figure 4A A schematic diagram illustrating an image acquisition pipeline and a strain measurement framework according to an embodiment of the present invention is shown.

[0020] Figure 4B A schematic diagram illustrating an image acquisition pipeline and a strain measurement framework according to an embodiment of the present invention is shown.

[0021] Figure 5 An algorithm for a refined robust weighted LM (RRWLM) is illustrated according to an embodiment of the present invention.

[0022] Figure 6 The PSNR of the average error of camera pose estimation and the image fusion result according to an embodiment of the present invention is shown. The average error of camera pose estimation and the image fusion result are also shown. and The PSNR can be calculated using all 160 images or just the first 10 images in each sequence.

[0023] Figure 7 A comparison of small-area strain images according to an embodiment of the present invention is shown. The comparison of small-area strain images uses the following elements: (a) an ideal surface image; (b) a fused image obtained by the proposed RRWLM method; and (c) a fused image of the best baseline result obtained by OPnP+LM.

[0024] Figure 8 A comparison of surface images based on different methods according to embodiments of the present invention is shown. The surface image comparison includes: (a) an ideal image. (b) Fusion image obtained by RRWLM (c) The fused image obtained by OPnP+LM .

[0025] Figure 9 A comparison of large-area strain images according to an embodiment of the present invention is shown. The comparison of large-area strain images uses the following elements: (a) an ideal surface image. and (b) Fusion image obtained by RRWLM and . Detailed Implementation

[0026] Various embodiments of the present invention will now be described with reference to the accompanying drawings. It should be noted that the drawings are not drawn to scale, and elements with similar structures or functions are indicated by similar reference numerals in all the drawings. It should also be noted that the drawings are intended only to facilitate the description of specific embodiments of the invention. They are not intended as an exhaustive description of the invention or a limitation on its scope. Furthermore, aspects described in connection with specific embodiments of the invention are not necessarily limited to those embodiments and can be practiced in any other embodiment of the invention.

[0027] We consider strain measurement on the surface of a cylinder, which is of great interest in many applications. For example... Figure 4A and Figure 4B As shown, for image acquisition, the moving camera captures the surface texture of the cylinder before deformation. Capture a series of images After deformation capture Each sequence consists of p (or q) images, which overlap sequentially with their adjacent images. Without loss of generality, we show only the sequences in the following description. The model and analysis.

[0028] Since out-of-focus blur is a common image degradation phenomenon, we consider the point spread function (PSF) (blur kernel) of the camera lens. A six-degree-of-freedom (6-DOF) pinhole camera model, which is assumed to be a truncated Gaussian kernel:

[0029] (1)

[0030] in, It is the radius. It is used to guarantee PSF The energy normalization term. Then, the captured image can be processed. Modeling:

[0031] (2)

[0032] in, This represents the convolution operation. It is a sharp camera focal plane image, and p is the total number of images. Each pixel in According to the following formula, from the pixel u on the 3D surface... Projected:

[0033] (3)

[0034] in, and They depend on The rotation matrix and translation vector of the camera pose. It is a pixel-dependent scalar that projects pixels onto the focal plane, and It is the camera's perspective matrix.

[0035] It should be noted that each image in the sequence All covered the surface of the cylinder Narrow field. Our goal is based on... and The entire unfolded image of the curved surface is recovered so that strain on the cylindrical surface can be analyzed using 2D DIC. In the following description, we introduce our proposed framework, which includes image deblurring, image fusion, and DIC, such as... Figures 4A-4B As shown.

[0036] Image Deblurring

[0037] The objective of this module is to obtain fuzzy observations from (2). Simultaneously restore sharp focal plane images And an unknown, fuzzy kernel K. Therefore, we formulate the blind deconvolution problem as:

[0038] (4)

[0039] in, Denotes the Frobenius norm of a matrix. It is used to ensure The indicator function for the truncated Gaussian kernel. express and Pixels in these two directions place The derivative of, and It depends on the image. The weights of the noise level. The first term is the data fidelity term. The second term is the widely used regularization term Total Variation (TV) to maintain image sharpness. (4) By relative to and We solve this by alternating minimization. In particular, when we consider… Using recurrent convolution to update under the periodic boundary assumption This allows for rapid computation via FFT.

[0040] To obtain a good initial value for the fuzzy kernel We use Wiener filters in the possible region The normalized sparsity metric is minimized as follows:

[0041] (5)

[0042] in, It has a nucleus Filtered images , They represent and The derivative in the direction, and This refers to the number of images used.

[0043] Image fusion

[0044] In this module, we utilize deblurred image sequences. Reconstructing super-resolution textures on curved surfaces of 3D objects for DIC analysis.

[0045] Camera pose estimation

[0046] Without loss of generality, we consider deblurring the target image. Overlapping reference image with known camera pose Co-registration to estimate the target deblurred image Issues such as camera posture.

[0047] First, we obtain the target image. The well-known SIFT feature point set in and reference In Then, we search for a set of matching feature points. It satisfies:

[0048] (6)

[0049] in, Represents pixels SIFT feature vector at that location, It has been ruled out. of The set, and It is the constant selected for deleting feature outliers, usually 1. .

[0050] we will Each feature point in Projected onto a 3D surface and using (3) and The corresponding set is obtained by considering the pose and geometry of the object. Then, the camera pose estimation problem becomes the well-known PnP problem, in order to utilize point sets. To estimate the camera attitude.

[0051] The PnP problem can usually be expressed as a nonlinear sum of least squares problems. Considering Constraints We use Let these represent the unknown parameters of the camera pose. Then, the following equation can be solved to achieve the same result as... Related camera pose :

[0052] st (7)

[0053] in, It utilizes (3) relative to the camera pose From 3D points to the camera focal plane The projection results From the above Certainly, and for , This represents the reciprocal of the measurement error of the m-th feature pair, and is usually... .

[0054] To address this problem, we utilize the widely used Levenberg-Marquardt (LM) algorithm with the projection operator. The combination of these is used to preserve the orthogonality and normality of the rotation matrix R. Given the current estimate... According to the following formula, the one-step update of (7) by LM This can be viewed as an interpolation of greedy descent and Gauss-Newton update.

[0055] (8)

[0056] in, It is the Hessian matrix. ,and These are parameters that change with iteration and are used to determine the interpolation level accordingly.

[0057] Projection operator Define as , Orthogonal normalization. We allocate approximately half of the error to... and The method was modified to:

[0058] (9)

[0059] Make the output orthogonal normalized for .

[0060] For sequences Each image in Using the previous image As a reference image, we utilize (8) with the matching feature set And the subsequent projection operation And evaluation steps, iteratively updating its camera pose. This allows us to estimate the camera's orientation.

[0061] Camera pose refinement and image fusion

[0062] Driven by the principle of bundle adjustment, we propose further refining the camera pose estimation to utilize more useful matching feature pairs. Through this observation, for the i-th image... We search for feature pairs in all previous images and form a sequence with... Overlapping image index set Using the target image With index set By matching feature points between each indexed image under the same conditions in (6), we obtain the matching feature set. The union of . Figure 5 An algorithm for a refined robust weighted LM (RRWLM) according to an embodiment of the present invention is shown. The camera pose estimated according to (9) is utilized. To perform initialization, such as Figure 5 As summarized in the previous section, the proposed method RRWLM alternately updates one pose while keeping other poses fixed. Finally, it utilizes the image sequence... (After deformation) Based on the accurately estimated camera pose, we project all pixels from these images back onto the 3D surface and use linear interpolation to achieve super-resolution surface texture. ( ), and unfold it into the final 2D image. .

[0063] DIC

[0064] From the previous modules, we respectively used two sequences of images as input from a narrow field of view. and A large field of view reference for 3D surfaces was obtained. and deformed images The basic principle of DIC is to obtain the displacement by tracking selected points between two images recorded before and after deformation. Using feature tracking, sub-displacements can be calculated by tracking pixels in a sparse grid defined on a reference image. Assuming that displacements are small in most engineering applications, our DIC module can perform strain measurement calculations based on displacements at different smoothing levels, as programmed.

[0065] Numerical Experiment

[0066] Test setup

[0067] For the 3D surface being tested, such as Figures 4A-4B As shown, a moving camera captures two sequences of images before and after deformation, assuming the area outside the cylinder is black. For the 3D cylinder, its radius r = 500 mm and its height H = 80 mm. The camera's trajectory lies approximately on the surface of a coaxial virtual cylinder with a radius r² = 540 mm. Due to random perturbations, the camera pose for all captured images except the first image cannot be accurately determined.

[0068] For super-resolution reconstruction of surface texture, the camera moves in a serpentine scanning pattern, taking 5 images as it moves along the axis, then moving tangentially to take the next 5 images along the axis, and so on. For each sequence, we collect a total of p = 160 images of size m × n = 500 × 600. Both sequences cover the same region, a cylindrical surface of approximately 60 degrees, with the camera starting position differing slightly before and after deformation. This region extends directly to the 360° surface.

[0069] Implementation and evaluation

[0070] To examine our proposed framework and the necessary PnP methods for image fusion, we consider five baseline methods for rejecting outliers, including the typical iterative method LHM, and four state-of-the-art non-iterative methods EPnP + GN, OPnP + LM, ASPnP, and REPPnP. For comparison, we show the non-refined estimation processing using (9) as a robust weighted LM (RWLM) and... Figure 5The algorithm shown in Algorithm 1 is a refined robust weighted LM as RRWLM. All baseline methods use the same matching feature set. Both LHM and RWLM use their own camera pose estimates of the previous image as initial values ​​for the current image. RRWLM uses... The camera pose estimation was run with M=20 and other parameters. To ensure accuracy, we consider the actual situation. The rotation and translation errors are calculated as follows: And for the image stitching results and The widely used PSNR was calculated.

[0071] First, only the first 10 images from each image sequence are used. and For reference and deformable textures, we show the average camera pose estimation error and the stitched surface texture image. and The average PSNR, and with Figure 6 The three best baseline methods were compared. Figure 7 The results of strain analysis obtained from DIC are shown in the figure. We observe that when the number of images used for fusion is relatively small, the proposed method has competitive accuracy compared to existing methods.

[0072] Figure 6 The average error of camera pose estimation and image fusion results according to an embodiment of the present invention are shown. and The PSNR of the average error of camera pose estimation and the PSNR of the image fusion result were calculated using all 160 images or only the first 10 images in each sequence. The figure was rewritten to show the same number of images using all images in a sequence of size p=q=160. Compared with RWLM, the proposed RRWLM method improves performance through camera pose refinement and significantly outperforms the baseline methods mentioned above when stitching a large number of images. The main reason for the improvement is that the proposed RRWLM method reduces the accumulation of irreversible camera pose errors in the target scene.

[0073] For illustrative purposes, compared Figure 8 The ideal image shown in (a) and Figure 8 The optimal baseline method in (c) is OPnP+LM. Figure 8 (b) shows the reference image obtained by the proposed RRWLM. The image fusion results. Since the image fusion results obtained using existing methods are no longer suitable for reasonable strain measurements, therefore... Figure 9In this paper, we only compare the strain measurement results obtained by relying on DIC using RRWLM with the actual situation (due to space constraints, only the strain in the xx direction is shown). This means that even when a large number of images are being fused, the proposed framework can achieve at least sub-pixel accuracy of the image fusion results for strain measurement.

[0074] Therefore, some embodiments of the present invention provide an end-to-end fusion-based DIC framework for 2D strain measurement along the curved surfaces of large 3D objects. To address the challenge of narrow field of view of a single image, we incorporate image fusion principles and decouple the image fusion problem into a series of perspective n-point (PnP) problems. The proposed PnP method, combined with bundle adjustment, accurately recovers the 3D surface texture stitched from a large number of images and achieves suitable strain measurement through the DIC method. Numerical experiments demonstrate its superior performance compared to existing methods.

[0075] The embodiments of the present invention described above can be implemented in any of a variety of ways. For example, the embodiments can be implemented using hardware, software, or a combination thereof. When implemented in software, the software code can execute on any suitable processor or set of processors, whether provided in a single computer or distributed across multiple computers. Such a processor can be implemented as an integrated circuit having one or more processors within an integrated circuit assembly. However, the processor can be implemented using circuitry of any suitable format.

[0076] Similarly, embodiments of the invention can be embodied as a method, of which an example has been provided. Actions performed as part of the method can be ordered in any suitable manner. Therefore, embodiments can be constructed in which actions are performed in a different order than those described, and even actions shown as sequential in the illustrative embodiments may include the simultaneous execution of certain actions.

[0077] The use of serial numbers such as "first" and "second" to modify claim elements in the claims does not imply any priority, position, or order of one claim element relative to another, nor does it imply the chronological order of the actions of performing the method. Rather, it serves as a label to distinguish one claim element with a specific name from another element with the same name (but used in serial numbers) to differentiate claim elements.

[0078] Although the invention has been described by way of example of preferred embodiments, it should be understood that various other adjustments and modifications can be made within the concept and scope of the invention.

[0079] Therefore, the purpose of the appended claims is to cover all such variations and modifications that fall within the true concept and scope of the invention.

[0080] Figure 1 This is a schematic diagram illustrating a strain measurement system 100 for generating a displacement image of a surface of interest 140 according to an embodiment of the present disclosure. In some cases, the displacement image may be a strain image.

[0081] The strain measurement system 100 may include a network interface controller (interface) 110 configured to receive images from a camera / sensor 141 and display the images on a display 142. The camera / sensor 141 is configured to capture overlapping images of the surface of interest 140.

[0082] Furthermore, the strain measurement system 100 may include a memory / CPU unit 120 for storing a computer-executable program in the memory 200. The computer-executable program / algorithm may include an image deblurring unit 220, an image stitching unit 230, a digital image correlation (DIC) unit 240, and an image displacement mapping unit 250. The computer-executable program is configured to be connected to the memory / CPU unit 120, which accesses the memory 200 to load the computer-executable program.

[0083] Furthermore, the memory / CPU unit 120 is configured to receive images (data) from the camera / sensor 151 or the image data server 152 via the network 150 and perform the displacement measurement (system) 100 described above.

[0084] In addition, the strain measurement system 100 may include at least one camera arranged to capture images of the surface of interest 140, and the at least one camera may transmit the captured images to a display (device) 142 via an interface.

[0085] Figure 2A schematic diagram illustrating a memory 200 for generating displacement images from surface images captured by a camera / sensor, according to some embodiments of the present disclosure, is shown. The memory (module) 200 generates displacement images 250 using images captured before and after strain, respectively, using tags ending in A and B. First, a blurred overlapping image 215A is captured by pre-strain image acquisition (processing) 210A, and after image deblurring (processing) 220A, the image is sharpened into a sharp overlapping image 225A. Then, the sharp overlapping images are stitched together using image stitching (processing) 230A to form a large, sharp surface image 235A. Similarly, a blurred overlapping image 215B captured by post-strain image acquisition 210B is processed by image deblurring 220B (the image is sharpened into a sharp overlapping image 225B) and image stitching 230B to form a large, sharp surface image 235B. DIC analysis 240 is used to compare images 235A and 235B to generate a displacement image 250 that shows the strain received by the surface.

[0086] Figure 3A A schematic diagram of an image deblurring module 220, illustrating an embodiment of the present disclosure, for deblurring a blurred overlapping image 215 of a surface captured by a camera / sensor. First, an initial blur kernel (2201) is estimated using a Wiener filter by minimizing a normalized sparsity metric as indicated in (5). Then, the image is sharpened by solving an iterative blind deconvolution problem (4). In each iteration, after deconvolving the blur kernel with the captured image (2202), a sharpened image is generated and used to check convergence by comparing it with a previous sharpened image (2203). If their differences (or relative errors) are small (2204), indicating algorithm convergence, the image deblurring module 220 outputs the current sharpened image as a sharp overlapping image 225. Otherwise (2204), the blur kernel is updated by minimizing (4) (2205) and used for the next iteration of deconvolution processing (2202) until the algorithm converges.

[0087] Figure 3B A schematic diagram is shown illustrating an image stitching module 230 for stitching sharp overlapping images 225 into a large and sharp surface image according to some embodiments of the present disclosure. First, in order to stitch the i-th image with its neighboring image set L... i The j-th nearest neighboring image in the image is stitched together, where the camera position h of the j-th image is... j It is known that the matching point is A. j,i (2301) is determined using matched SIFT features. The known camera pose h is used. jProject the matching points on the j-th image onto the surface of the cylinder (2302). If the camera position of the i-th image is unknown (2303), then use Algorithm 1 to solve the PnP problem (2304) to estimate the camera pose h. i , through including h i Update the known camera pose set H (2305) and update the neighboring image set L by including the i-th image. i (2306). Then, the (i+1)th image is considered as stitched to its neighboring images. If the camera pose associated with all images is determined, it means h i If the unknown (2303) is not true, then the image is projected onto the cylindrical surface using their camera pose, and a large and sharp surface image 235 is generated by interpolation (2307).

[0088] The above embodiments of the present invention can be implemented using hardware, software, or a combination of hardware and software.

[0089] Similarly, embodiments of the invention can be embodied as a method, in which an example has been provided. Actions performed as part of the method can be ordered in any suitable manner. Thus, embodiments can be constructed in which actions are performed in an order different from the order described, and even actions shown as sequential in the illustrative embodiments may include the simultaneous execution of certain actions.

[0090] The use of serial numbers such as "first" and "second" to modify claim elements in the claims does not imply any priority, position, or order of one claim element relative to another, nor does it imply the chronological order of the actions of performing the method. Rather, it serves as a label to distinguish one claim element with a specific name from another element with the same name (but used in serial numbers) to differentiate claim elements.

Claims

1. An image processing apparatus for measuring the strain of an object, the image processing apparatus comprising: An interface configured to acquire a first sequence of images and a second sequence of images, wherein two adjacent images of the first sequence of images include a first overlapping portion, and two adjacent images of the second sequence of images include a second overlapping portion, wherein the first sequence of images corresponds to a first 3D surface on the object in a first state, and the second sequence of images corresponds to a second 3D surface on the object in a second state. The memory is used to store computer-executable programs, including image deblurring methods, pose refinement methods, image stitching methods, and displacement measurement methods. as well as A processor configured to execute the computer-executable program, wherein the processor performs the following steps: The first sequence image and the second sequence image are deblurred using blind deconvolution to obtain a sharpened image; Based on camera pose estimation by solving the perspective n-point problem, the first sharpened sequence of images and the second sharpened sequence of images are stitched together to form the first 3D image and the second 3D image, respectively. A first 2D image and a second 2D image are formed by respectively unfolding the first 3D image and the second 3D image; as well as A displacement map is generated from the first 2D image and the second 2D image by performing a two-dimensional digital image correlation method, wherein, The splicing process includes a first stage and a second stage. In the first stage, the initial camera pose of the first sharpened image sequence and the second sharpened image sequence is estimated. In the second phase, a pose is updated alternately by using other poses while keeping other poses fixed.

2. The image processing apparatus according to claim 1, wherein, The camera pose estimation is updated using a refined robust weighted Levenberg-Marquardt algorithm, which is based on the Levenberg-Marquardt algorithm and refines and makes the pose estimation robust by alternately updating one pose using other poses while keeping other poses fixed, and using the reciprocal of each measurement error to represent the weights used for stitching the image.

3. The image processing apparatus according to claim 1, wherein, The camera pose of at least the first image in the first image sequence is known, and the camera pose of at least the first image in the second image sequence is known.

4. The image processing apparatus according to claim 1, wherein, The first state is the initial condition of the object before it has been operated on during the initial time period, and the second state is the post-condition of the object after it has been operated on for a period of time.

5. The image processing apparatus according to claim 1, wherein, The image processing apparatus also includes analysis of local strain on the surface of the object using the displacement image.

6. The image processing apparatus according to claim 1, wherein, The perspective n-point problem uses matching points based on scale-invariant feature transformation features.

7. The image processing apparatus according to claim 1, wherein, In the two-dimensional digital image correlation method, the displacement image is calculated based on the feature tracking method.

8. The image processing apparatus according to claim 1, wherein, The first and second sequence images were acquired from the curved surface of the object.

9. The image processing apparatus according to claim 1, wherein, The object is cylindrical in shape.

10. The image processing apparatus according to claim 1, wherein, The first sequence of images was acquired before the object was deformed, and the second sequence of images was acquired after the object was deformed.

11. The image processing apparatus according to claim 1, wherein, The stitching is performed through super-resolution reconstruction.

12. An image processing method for measuring the strain of an object, the image processing method comprising: Acquire a first sequence of images and a second sequence of images, wherein two adjacent images of the first sequence of images include a first overlapping portion, and two adjacent images of the second sequence of images include a second overlapping portion, wherein the first sequence of images corresponds to a first 3D surface on the object in a first state, and the second sequence of images corresponds to a second 3D surface on the object in a second state; The first sequence image and the second sequence image are deblurred using blind deconvolution to obtain a sharpened image; Based on camera pose estimation by solving the perspective n-point problem, the first sharpened sequence of images and the second sharpened sequence of images are stitched together to form the first 3D image and the second 3D image, respectively. A first 2D image and a second 2D image are formed by respectively unfolding the first 3D image and the second 3D image; and A displacement map is generated from the first 2D image and the second 2D image by performing a two-dimensional digital image correlation method, wherein, The splicing process includes a first stage and a second stage. In the first stage, the initial camera pose of the first sharpened image sequence and the second sharpened image sequence is estimated. In the second phase, one pose is updated alternately while keeping other poses fixed.

13. A computer-readable medium, which is a non-transitory computer-readable medium, comprising program instructions for causing a computer to perform a method, the method comprising: Acquire a first sequence of images and a second sequence of images, wherein two adjacent images of the first sequence of images include a first overlapping portion, and two adjacent images of the second sequence of images include a second overlapping portion, wherein the first sequence of images corresponds to a first 3D surface on an object in a first state, and the second sequence of images corresponds to a second 3D surface on the object in a second state; The first sequence image and the second sequence image are deblurred using blind deconvolution to obtain a sharpened image; Based on camera pose estimation by solving the perspective n-point problem, the first sharpened sequence of images and the second sharpened sequence of images are stitched together to form the first 3D image and the second 3D image, respectively. A first 2D image and a second 2D image are formed by respectively unfolding the first 3D image and the second 3D image; as well as A displacement map is generated from the first 2D image and the second 2D image by performing a two-dimensional digital image correlation method, wherein, The splicing process includes a first stage and a second stage. In the first stage, the initial camera pose of the first sharpened image sequence and the second sharpened image sequence is estimated. In the second phase, one pose is updated alternately while keeping other poses fixed.

Images