Compact morphology measurement system and method based on field-of-view splicing digital holography

Abstract

The invention provides a compact morphology measurement system and method based on field-of-view splicing digital holography. The compact morphology measurement system mainly comprises a laser control and generation module, an off-axis holographic light path module, an image acquisition and transmission module, a numerical value reconstruction module and an image splicing module. The laser control and generation module generates laser of which the frequency is changed step by step; the off-axis holographic light path module divides the laser into a measuring light path and a reference light path, the measuring light path is irradiated on the surface of an object to generate diffuse reflection after being subjected to beam expansion, interferes with the reference light path subjected to collimation and spatial filtering in the image acquisition and transmission module, and is acquired by a camera and transmitted to a computer; the numerical value reconstruction module adopts a sweep frequency method to reconstruct the object surface morphology; and the image splicing module adopts a deep learning method to splice a plurality of small-view-field images to realize large-view-field measurement. The device has the advantages of compact structure, high stability, large measurement field of view, submicron non-contact morphology measurement and the like, and is suitable for the field of precision mechanical part measurement.

Description

Technical Field

[0001] The invention provides a compact shape measurement system and method based on field-of-view splicing digital holography, belonging to the technical field of digital holographic shape measurement. Background Art

[0002] With the continuous advancement of science and technology, higher requirements are put forward for the precise measurement of the surface morphology of parts in the field of industrial manufacturing. Surface morphology measurement is widely used in various fields such as aerospace, machinery manufacturing, medical manufacturing, integrated circuits, etc. According to whether the measuring probe is in contact with the surface of the part, surface morphology measurement is divided into contact measurement method and non-contact measurement method. The traditional contact measurement method generally uses a three-coordinate measuring machine to measure the surface morphology. This method not only has a low measurement point density and a long measurement time, but also easily damages the surface of the object. At the same time, the measurement time of this method increases rapidly with the increase of target complexity and measurement point density. The digital holographic technology developed in recent years has the advantages of high measurement accuracy, full data volume, fast speed, non-destructiveness, and easy acquisition of complex structure three-dimensional morphology. It is a hot technology in the field of morphology measurement.

[0003] Digital holography measurement technology (Digital holography) is divided into single wavelength digital holography (SWDH) and multi-wavelength digital holography (MWDH) according to the number of wavelengths of the light source used. Multi-wavelength digital holography technology includes dual-wavelength digital holography (DWDH) and frequency sweeping digital holography (FSDH). Single wavelength digital holography technology refers to the technology that uses a single wavelength illumination light source, uses laser to illuminate the surface of an object, and obtains the surface morphology of the object through the interference of the light reflected or transmitted by the object with the reference light. However, in the measurement of non-continuous surface morphology, the axial measurement range of single wavelength digital holography is generally less than half of the wavelength, and the measurement effect is poor for complex and high-roughness morphologies, and the deformation measurement range is small. Multi-wavelength digital holography uses the optical path of single-wavelength digital holography to obtain interference information at different wavelengths in turn. The measurement range is expanded from a few microns to tens of microns or even millimeters. At the same time, objects with high roughness and step morphology can be accurately measured. This expands the application field of digital holography technology from tiny-scale objects to the measurement of mechanical parts of conventional sizes.In 2018, Tobias Seyler et al. published a paper titled "Extending the Depth of Fieldbeyond Geometrical Imaging Limitations Using Phase Noise as a Focus Measurein Multiwavelength Digital Holography" in Applied Sciences, Vol. 8, No. 7, p. 1042. In this paper, multi-wavelength digital holography was used to measure the morphology of gear samples in industrial scenes. A focus morphology recovery algorithm based on phase noise was proposed, which achieved the extension of the depth of field beyond the geometric imaging limitation in multi-wavelength digital holograms. In 2017, Vit Ledl et al. published a paper titled "Surface topography measurement by frequency sweeping digital holography" in Applied Optics, Vol. 56, No. 28, pp. 7808-7814. Digital holography, which uses a frequency sweeping digital holography method, is suitable for surface topography measurement of mechanical parts with complex geometric shapes and large variations in measurement depth. It is superior to traditional three-dimensional coordinate measuring machine methods in terms of accuracy and dynamic range. In 2021, Kavan Frantisek et al. published a paper titled "Parameter optimization of frequency sweeping digital holography for measurement of ground optical surfaces" in Applied Optics, Vol. 60, No. 27, pp. 8368-8374. The paper compared the performance of frequency sweeping digital holography and dual-wavelength digital holography under different measurement parameters such as light intensity, tilt, and surface roughness. The frequency sweeping digital holography method achieved better measurement results than the dual-wavelength digital holography method when the object had a large tilt, large fluctuations in light intensity, and large steps.In 2023, Kavan Frantisek et al. published a paper titled "Multiple wavelength digital holography for freeform shape measurement and lens alignment" in Applied Optics, Vol. 62, No. 10, pp. D138-D145. The paper used dual-wavelength digital holography to measure free-form surfaces during and after grinding, and was also used to diagnose the accuracy of component placement in optical systems, such as the relationship between the optical axis and the mechanical axis. In 2023, Hamada Hikaru et al. published a paper titled "Two-wavelength digital holography using frequency-modulated continuous-wave technique for multiplexing in the time-frequency domain" in Optical Review, Vol. 30, No. 6, pp. 595-605. The paper combined frequency-modulated continuous wave technology with dual-wavelength digital holography technology, reducing phase errors and expanding the axial measurement range. Therefore, compared with other digital holographic technologies, the swept-frequency digital holographic technology expands the axial measurement depth and is more suitable for measuring the surface morphology of objects with large absolute changes on the surface. It has broad application prospects in the morphology measurement of larger mechanical structure parts.

[0004] In recent years, the digital holographic 3D shape measurement technology has been developing continuously, attracting many domestic and foreign scholars to invest in research in this field. Building an integrated and miniaturized system is an important step to meet the actual engineering needs, and it is also a research direction for researchers at present. In 2020, Tobias Seyler et al. published a paper titled "HoloPort-design and integration of a digital holographic 3-D sensor in machine tools" in the Journal of Sensors and Sensor Systems, Vol. 9, No. 1, pp. 33-41. The paper proposed a miniaturized digital holographic measurement system that can directly measure the complete shape of the machined parts with sub-micron accuracy in a 20mm×20mm area in the machine tool. At the same time, the processing time is short, only 3 seconds, of which the pure measurement time is less than 1 second, realizing the closed-loop quality control directly inside the machine tool for the first time. In 2021, Raj Kumar et al. published a paper titled "Portable digital holographic camera featuring enhanced field of view and reduced exposure time" in Optics and Lasers in Engineering, Volume 137, Page 106359. The paper proposed a portable digital holographic camera that uses a combination of concave and convex lenses to achieve a field of view that is 192% larger than that of traditional digital holographic systems. At the same time, the combination design of concave and convex lenses reduces the exposure time of the camera, thereby reducing the impact of vibration on the system. In 2020, Zuo Chao et al. published "An auto-focusing reflection-type lens-less digital holographic microscope" in the "Fourth International Conference on Photonics and Optical Engineering". They proposed an auto-focusing reflection-type lens-less digital holographic microscope system with a wide field of view, high precision and miniaturization. It can quickly and accurately perform three-dimensional reconstruction and quantitative measurement of small devices with reflective surfaces. The overall size is only 68mm×43mm×38mm, and the autofocus function is also greatly improved.From the above, we can see that there are integrated digital holographic measurement systems for different scenarios, but there are still problems such as low stability, large system size, and difficult optical path adjustment.

[0005] At present, the single measurement range of digital holography is small. It is expensive and difficult to manufacture to use large cameras to measure the full size of the object to be measured. The image stitching algorithm provides a method to significantly expand the measurement range without significantly increasing the difficulty and complexity of the measurement system. Therefore, in the actual measurement process, image stitching is often used to achieve large field of view measurement. Traditional image stitching methods mostly use sub-aperture stitching, phase correlation method, Harris method and other methods. In 2016, Pan Feng et al. published a paper titled "Subaperture stitching interferometry based on digital holography" on pages 228-235 of Volume 86 of Optics and Lasers in Engineering. By rotating the object, the sub-aperture map of the digital holography was obtained, and a sub-aperture stitching algorithm based on iterative optimization was used to achieve the full-aperture phase map, which was applied to the deformation measurement of the spherical surface. In 2021, Liu Wei et al. published a paper titled "Research on Phase Stitching Method Based on Multi-Algorithm Fusion in Holography" on page 0709001 of Volume 48, Issue 7 of "Chinese Lasers". The paper combines the phase correlation method, Harris algorithm and full matching search algorithm, and proposes a phase stitching method based on multi-algorithm fusion optimization in digital holography. As can be seen from the above, traditional stitching methods mostly use corner detection algorithms to extract corners in the entire image. There is unnecessary detection and extraction of corners, which increases the time consumption of registration and affects the matching accuracy. Moreover, after multiple stitching, the stitching efficiency of traditional methods will drop significantly as the size of the image increases, and mismatching is prone to occur in similar areas of the image.

[0006] With the continuous development of deep learning technology, it has acquired powerful feature extraction capabilities and has shown excellent robustness under complex conditions. Therefore, the research on deep learning algorithms in the field of image stitching is gradually becoming a new research hotspot. In 2016, Daniel DeTone et al. submitted the article "Deep Image Homography Estimation" with the number arXiv:1606.03798 to the arXiv preprint website. They proposed a deep convolutional neural network to estimate the homography matrix between images, using the 4-corner homography parameterization and end-to-end estimation, and found the possibility of using deep learning to estimate homography in image stitching. In 2020, Lin Chunyu et al. published a paper titled "A view-free image stitching network based on global homography" in the Journal of Visual Communication and Image Representation, Volume 73, Page 102950. They proposed a view-free image stitching network based on global homography, which was the first time that images of arbitrary views were successfully stitched in a complete deep learning framework. In 2021, Lin Chunyu et al. published a paper titled "Unsupervised Deep Image Stitching: Reconstructing Stitched Features to Images" in Volume 30, pages 6184-6197 of the IEEE Transactions on Image Processing, in which they first proposed the unsupervised deep image stitching framework UDIS; in 2023, Lin Chunyu et al. published a paper titled "Parallax-Tolerant Unsupervised Deep Image Stitching" at the 2023 IEEE / CVF International Conference on Computer Vision (ICCV), which further pushed the deep learning-based image stitching framework to a research climax.In 2024, Minsu Kim et al. published a paper titled "Implicit Neural Image Stitching With Enhanced and Blended Feature Reconstruction" at the 2024 IEEE / CVF Winter Conference on Applications of Computer Vision (WACV). They proposed a new implicit neural network stitching method, which effectively solved the problems of blurring artifacts, illumination, and depth level mismatch in existing methods, and improved image quality. As can be seen from the above, image stitching based on deep learning is developing rapidly, and deep learning frameworks are emerging in an endless stream, but it has not yet been widely used in digital holographic image stitching.

[0007] Image stitching methods based on deep learning usually include two main stages: deformation stage and stitching stage. In the stitching stage, in order to achieve more natural and seamless image fusion, the network structure often adopts or improves the existing segmentation network to achieve it. In 2015, Olaf Ronneberger et al. published a paper titled "U-Net: Convolutional Networks for Biomedical Image Segmentation. Medical Image Computing and Computer-Assisted Intervention" on pages 234-241 at the conference "Medical Image Computing and Computer-Assisted Intervention" (MICCAI2015). The U-Net network adopts an encoder-decoder architecture, and the network structure is U-shaped. It introduces skip connections, which improves the accuracy of image segmentation and has been widely used in image segmentation scenarios in many fields. In 2018, Liang-Chieh Chen et al. published a paper titled "Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation" on pages 833-851 of Volume 11211 of the 15th European Conference on Computer Vision (ECCV). The DeepLabv3+ network is a new network proposed by Liang-Chieh Chen et al. based on the DeepLabv1, DeepLabv2, and DeepLabv3 networks. The network adds a decoder module to DeepLabv3, and fuses the low-level features and high-level features in the encoder module, showing better performance when facing complex scenes and details. At the same time, the network uses atrous convolution to expand the receptive field and capture a wider range of contextual information, and performs well in processing complex scenes and aligning large field of view features.From the above, we can see that there are already a series of outstanding image segmentation network structures in the field of deep learning. They are applied to the stitching stage of image stitching, which can achieve image fusion and reduce the impact of seams. However, in the stitching of sub-micron surface morphology images, there is a lot of detailed texture information of the morphology of complex objects. At the same time, it is affected by the stability of the wavelength of the measurement system and the system speckle noise. As a result, the existing deep learning methods have uneven transitions at the seams during image stitching in this scenario, and there are chromatic aberrations and geometric misalignments. Therefore, the stitching method still needs to be further improved and optimized to make the stitching seams more natural.

[0008] Based on the above background, the present invention proposes a compact morphology measurement system and method based on field-of-view stitching digital holography for realizing large-field-of-view high-precision measurement of the surface of an object. In order to improve the stability and portability of the system, the spatial filter structure is optimized, and an integrated off-axis digital holographic measurement system is built and designed using a cage structure; the interference pattern collected by the camera is numerically reconstructed using a swept-frequency digital holographic method, which is more suitable for morphology measurement of objects with large absolute surface transformations; in order to achieve image stitching of submicron surface morphology, the network structure of the image stitching stage is optimized to make the seams more natural after stitching, thereby realizing fast and accurate morphology image stitching. The present invention combines digital holographic measurement technology, swept-frequency digital holography technology, and deep learning image stitching technology to develop a solution for accurately measuring the surface morphology of an object and realizing large-field-of-view measurement through image stitching, so as to solve the problems of poor system stability and small measurement range of existing solutions, which is helpful for measuring the surface morphology of mechanical parts with a large field of view and provides more options for surface morphology measurement. Summary of the invention

[0009] The purpose of the present invention is to solve the problems of poor stability and small measurement range of the existing measurement system, and to propose a compact shape measurement system and method based on field of view splicing digital holography, which belongs to the technical field of digital holographic shape measurement. The system includes five modules: laser control and generation module, off-axis holographic optical path module, image acquisition and transmission module, numerical reconstruction module and image splicing.

[0010] In order to achieve the above technical objectives, the technical solution adopted by the present invention is: a computer controls a tunable laser in a laser control and generation module to generate a laser with a fixed frequency interval step change, and the output laser is divided into two paths after passing through a fiber coupler in an off-axis holographic optical path module. One path is a measurement optical path, which is irradiated to the surface of the object to be measured fixed on the displacement table after passing through a laser beam expander, and enters the image acquisition and transmission module after diffuse reflection on the surface of the object; the other path is a reference optical path, which is focused by an aspheric focusing lens through a pinhole after passing through a fiber collimator and an adjustable attenuation plate, and then expanded and collimated by a plano-convex lens to filter the spatial noise in the light beam, and the spot size is adjusted by an adjustable aperture, and enters the image acquisition and transmission module after reflection by a reflector; the measurement light and the reference light entering the image acquisition and transmission module are After the light is combined by the beam splitter prism, interference is generated. The camera collects the interference pattern and transmits it to the computer. The object to be measured is fixed in the displacement platform. In the numerical reconstruction module, M interference patterns are obtained in one scanning cycle. Digital holographic numerical reconstruction and phase extraction are performed on them in the computer. After phase unwrapping and denoising, the surface morphology of the measurement area is obtained. The displacement platform is moved in sequence, m times horizontally and n times vertically, and the above steps are repeated to finally obtain m×n surface morphology images. The image stitching module adopts an unsupervised deep learning image stitching algorithm, which is sequentially deformed and stitched to achieve pairwise stitching of surface morphology images. Repeating the stitching operation can obtain multiple pairwise stitching images, and then the stitched images are further stitched to obtain a surface morphology measurement image with a large field of view.

[0011] The off-axis holographic optical path module and the image acquisition and transmission module use a cage structure to assemble various optical devices together, and are designed with an integrated support base and cover, which enhances the stability of the system, facilitates optical axis alignment, and realizes rapid installation and replacement, thus realizing the integration and compactness of the system. The spatial filter is composed of an adjustable attenuation plate, an aspheric focusing lens, a pinhole, a plano-convex lens, and an adjustable diaphragm in the off-axis holographic optical path module; the reference beam is collimated by a collimator and an adjustable attenuation plate, and then focused by an aspheric focusing lens, and then passes through a pinhole centered on the optical axis to block the noise ring in the beam, transfer most of the laser energy to generate a pure Gaussian beam, eliminate the spatial inhomogeneity of the beam, and improve the beam quality. Then, it is expanded and collimated by a plano-convex lens, and adjusted by an adjustable diaphragm to form a uniform beam of a certain size.

[0012] The spatial filter part has achieved structural optimization through precise calculation and device selection, and is more suitable for an integrated cage structure. An aspherical condensing lens is used instead of an objective lens and a spherical condensing lens, and the aspherical condensing lens is fixed in a Z-axis displacement platform. The Z-axis displacement platform is fixed in the cage system through cage rods. The Z-axis displacement platform accurately controls its position, and selects appropriate pinholes, plano-convex lenses and other components through precise calculation, thereby achieving precise beam focusing and eliminating the spatial inhomogeneity of the beam.

[0013] The size of the pinhole depends on the size of the incident beam and the focal length of the focusing lens. The focal size is calculated according to the diffraction-limited spot formula. Calculation, λ represents the wavelength, f represents the focal length, and r is 1 / e of the incident beam 2 radius; the laser after passing through the collimator is considered to be collimated within the Rayleigh length, so 1 / e of the laser 2 The radius is approximately the beam waist radius of the collimator; therefore, according to the formula Calculate the size D of the diffraction-limited spot of the focused light spot; the selection of the pinhole should take into account the passage of the main lobe of the focal spot, and ensure the beam quality while ensuring that more than 95% of the energy passes through. The actual pinhole size should be about 30% larger than the diffraction-limited spot size. Select a suitable pinhole according to the size of the pinhole model; at the same time, in order to ensure the minimum diameter of the system's output spot, calculate the theoretical minimum focal length of collimation based on similar triangles, and select a plano-convex lens with a suitable focal length according to the model of the plano-convex lens to expand and collimate the focused and filtered light beam, and add an adjustable aperture after the plano-convex lens to adjust the size of the spot to meet the needs of optical path adjustment.

[0014] The cage-type precision adjustable spatial filter is easier to align the optical axis, the system is more stable, and is more suitable for the adjustment of the cage system. It also greatly reduces the length of the optical path, significantly reduces the volume and weight of the measurement system, and makes the system more compact and portable.

[0015] The image stitching module uses an unsupervised deep learning method to stitch the surface morphology images obtained by numerical reconstruction over a large field of view. The method is divided into an image deformation stage and an image stitching stage.

[0016] In the image deformation stage, two images with overlapping areas are taken as input, which are denoted as reference images I and I respectively. re and the target image I ta , after semantic feature extraction, overall deformation and fine deformation, the deformed image I is obtained wr , I wt And black and white mask M wr , M wt .

[0017] In the image stitching stage, in order to solve the problem of insufficient processing of seam details in the image stitching process in complex scenes, an extended DeepLabv3+ network is proposed as the stitching network, which improves the network's ability to process seam details in complex scenes. wr ,I wt As its input, it is generated by the encoder, decoder and spliced ​​image in sequence, and the spliced ​​mask M is predicted sr ,M st, and seamlessly stitched into image I according to formula (1). The specific steps of the stitching stage are as follows:

[0018] I=M sr I wr +M st I wt (1)

[0019] Step 1: The encoder of the extended DeepLabv3+ network consists of a backbone network and ASPP (Atrous Spatial Pyramid Pooling). The backbone network consists of ResNet101, which is used to extract deep features of the input image. wr and I wt After 5 layers of network processing, information of different scales is obtained, namely LF r0 , LF r1 , LF r2 , LF r3 , LF r4 and LF t0 , LF t1 , LF t2 , LF t3 , LF t4 ; The ASPP module receives the output LF of the last layer of the backbone network r4 and LF t4 As input, it consists of 4 dilated convolution blocks with different expansion rates and 1 global average pooling block. These blocks work in parallel to generate 5 sets of feature maps. These feature maps are merged and processed by convolution to reduce the number of channels and obtain the deep feature HF. r and HF t ,These deep features are then fed into the decoder module;

[0020] Step 2: The decoder of the extended DeepLabv3+ network uses multiple upsampling and multiple feature fusions to restore the abstract feature information in the encoder to an output of image size, so that the network can make a refined prediction of the image at the pixel level; the decoder uses the input image I wr and I wt The difference between the low-level feature maps of different layers after the backbone network and the output of the ASPP module is used as input, i.e., LF r1 -LF t1 , LF r2 -LF t2 , LF r3 -LF t3 and HF r -HF t; Introduce multi-layer lateral connections, upsample the deep features output by the ASPP module step by step and fuse them with the shallow features output by the corresponding backbone network. The whole process is done through 3 upsampling and 3 feature fusions to get the final prediction result, i.e., the deep feature HF r -HF t And the shallow feature LF after 1×1 convolution r3 -LF t3 After merging, SF is obtained by 3×3 convolution feature fusion. 3 SF 3 After 2 times upsampling, it is combined with the shallow feature LF after 1×1 convolution r2 -LF t2 After merging, SF is obtained by 3×3 convolution feature fusion. 2 SF 2 After 2 times upsampling, it is combined with the shallow feature LF after 1×1 convolution r1 -LF t1 After merging, SF is obtained by 3×3 convolution feature fusion. 1 ; Finally, SF 1 Upsample by 4 times and output the prediction result S with the same size as the original out ; The optimized decoder realizes the efficient fusion of deep features and multi-level shallow features, which not only improves the network's ability to capture detailed features, but also further enhances the network's ability to recognize and express multi-scale features during multiple feature fusion processes;

[0021] Step 3: According to the decoder's prediction result S out , calculate the reference mask image M in the stitching stage sr , as shown in the following formula (2):

[0022] M sr =M wr -M wr M wt +M wr M wt S out (2)

[0023] Among them, M wr Indicates the reference mask image obtained in the deformation stage; M wr M wt represents the overlapping area of ​​the reference mask image and the target mask image obtained in the deformation stage; M wr M wt S out represents the part of the reference image predicted by the model in the overlapping area of ​​the reference mask image and the target mask image; similarly, the target mask image M in the stitching stage is calculated st , as shown in the following formula (3):

[0024] M st =M wt -M wr M wt +M wr M wt (1-S out ) (3)

[0025] Among them, M wt Indicates the target mask image obtained in the deformation stage; M wr M wt (1-S out ) represents the target image portion predicted by the model in the overlapping area of ​​the reference mask image and the target mask image; then, the combined mask image M obtained by the above calculation sr and M st , substitute into formula (1) to calculate the final stitched image I; similarly, repeat the above operation to obtain multiple two-by-two stitched images, and then continue to stitch the stitched images to obtain a stitched image with a larger field of view.

[0026] The beneficial effect of the present invention lies in proposing a compact morphology measurement system and method based on field of view stitching digital holography, adopting a cage structure design to build an integrated digital holographic morphology measurement system, optimizing the spatial filter structure to make it more suitable for cage system adjustment, and improving the stability of the system; adopting an unsupervised deep learning method to achieve fast and accurate field of view stitching, and proposing an extended DeepLabv3+ network as a stitching network, which significantly improves the ability of the stitching network to process seam details in complex scenes, and is more suitable for the stitching of sub-micron surface morphology images. This method develops a solution for image stitching to achieve large field of view measurement, which provides a new idea for realizing digital holographic large field of view surface morphology measurement. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 Schematic diagram of a compact topography measurement system based on field of view stitching digital holography

[0028] Figure 2 The surface morphology images of different fields of view reconstructed based on the swept frequency digital holography method

[0029] Figure 3 This is a flowchart of the image stitching method based on unsupervised deep learning

[0030] Figure 4 This is an image stitching network structure diagram based on the extended DeepLabv3+

[0031] Figure 5 This is a 2×2 multi-image splicing surface morphology measurement result image DETAILED DESCRIPTION

[0032] The technical solution of the present invention is further described below in conjunction with the accompanying drawings. The present invention proposes a compact shape measurement system and method based on field of view stitching digital holography. The schematic diagram of the compact shape measurement system based on field of view stitching digital holography is shown in the attached figure. Figure 1 As shown, the measurement system includes a laser control and generation module, an off-axis holographic optical path module, an image acquisition and transmission module, a numerical reconstruction module and an image stitching module; the laser control and generation module includes a computer 101, a tunable laser 102, etc.; the off-axis holographic optical path module includes a fiber coupler 103, a laser beam expander 104, a fiber collimator 105, an adjustable attenuation plate 106, an aspheric focusing lens 107, a pinhole 108, a plano-convex lens 109, an adjustable aperture 110 and a reflector 111, etc.; the image acquisition and transmission module includes a beam splitter prism 112, a camera 113, etc.

[0033] The computer 101 in the laser control and generation module controls the tunable laser 102 to generate laser light with a frequency stepping interval of 50 GHz. The output laser light is divided into two paths after passing through the fiber coupler 103 in the off-axis holographic optical path module. One path is the measurement optical path, which is irradiated to the surface of the commemorative medal 114 of the object to be measured fixed on the displacement stage 115 after passing through the laser beam expander 104, and enters the image acquisition and transmission module after diffuse reflection on the surface of the object; the other path is the reference optical path, which is passed through the fiber collimator 105 and the adjustable attenuator 102. After the light is transmitted through the sheet 106, it is first focused by the aspheric condensing lens 107 and passes through the pinhole 108, and then expanded and collimated by the plano-convex lens 109 to filter the spatial noise in the light beam, and the spot size is adjusted by the adjustable aperture 110. After being reflected by the reflector 111, it enters the image acquisition and transmission module; the measurement light and the reference light entering the image acquisition and transmission module are combined by the beam splitter prism 112 to generate interference, and the camera 113 collects the interference pattern and transmits it to the computer; the commemorative medal 114 of the object to be measured is fixed in the displacement platform 115.

[0034] In the numerical reconstruction module, 10 interference patterns are obtained in one scanning cycle, and digital holographic numerical reconstruction and phase extraction are performed on them in the computer. After phase unwrapping and denoising, a surface topography of the measurement field is obtained; the translation stage is moved horizontally twice and vertically twice, and the above steps are repeated to finally obtain 4 surface topography images, as shown in the attached figure. Figure 2 As shown in (a), (b), (c) and (d).

[0035] The image stitching module adopts an unsupervised deep learning image stitching algorithm. After deformation and stitching operations, the pairwise stitching of surface morphology images is realized. Repeating the stitching operation can produce multiple pairwise stitched images. Then the stitched images are further stitched to obtain a surface morphology measurement image with a large field of view.

[0036] The off-axis holographic optical path module and the image acquisition and transmission module use a cage structure to assemble various optical devices together, and are designed with an integrated support base and cover, which enhances the stability of the system, facilitates optical axis alignment, realizes rapid installation and replacement, and realizes the integration and compactness of the system. The adjustable attenuation plate 106, the aspheric focusing lens 107, the pinhole 108, the plano-convex lens 109, and the adjustable aperture 110 in the off-axis holographic optical path module constitute a spatial filter; after the reference beam is collimated by the optical fiber collimator 105 and the adjustable attenuation plate 106, it is focused by the aspheric focusing lens 107, and then passes through the pinhole 108 centered on the optical axis to block the noise ring in the beam, transfer most of the laser energy to generate a pure Gaussian beam, eliminate the spatial inhomogeneity of the beam, and improve the beam quality. Then, it is expanded and collimated by the plano-convex lens 109, and the adjustable aperture 110 is adjusted to form a uniform beam of a certain size.

[0037] The spatial filter part has achieved structural optimization through precise calculation and device selection, and is more suitable for an integrated cage structure. An aspherical condenser lens 107 is used instead of an objective lens or a spherical condenser lens, and is fixed in a Z-axis displacement platform. The Z-axis displacement platform is fixed in the cage system through cage rods. The Z-axis displacement platform accurately controls its position, and selects appropriate pinholes, plano-convex lenses and other components through precise calculation, thereby achieving precise light beam focusing and eliminating spatial non-uniformity of the light beam.

[0038] The size of the pinhole depends on the size of the incident beam and the focal length of the focusing lens. The focal size is calculated according to the diffraction-limited spot formula. Calculated, where λ represents the wavelength, f represents the focal length, and r is 1 / e of the incident beam 2 Radius; The laser in the system is set to a central wavelength of 1410nm. The laser after passing through the collimator is considered to be collimated within the Rayleigh length, so the 1 / e 2 The radius is approximately equal to the beam waist radius of the collimator, 1.48 mm; the focal length of the aspheric focusing lens is 10.5 mm; therefore, the size of the diffraction-limited spot of the focused light spot can be calculated to be D≈10.003 μm; the selection of the pinhole should take into account the passage of the main lobe of the focal spot, and the beam quality should be guaranteed while ensuring that more than 95% of the energy passes through. The actual pinhole size should be about 30% larger than the diffraction-limited spot size, so theoretically a pinhole larger than 13.004 μm should be used, and the system uses a pinhole of 15 μm; at the same time, to ensure that the diameter of the output light spot of the system is at least 15 mm, the collimation focal length is calculated to be at least 53.2 mm based on similar triangles, and the system uses a plano-convex lens with a focal length of 75 mm to expand and collimate the light beam after focusing and filtering, and an adjustable aperture is added after the plano-convex lens to adjust the size of the light spot to meet the needs of optical path adjustment.

[0039] The cage-type precision adjustable spatial filter is easier to align the optical axis, the system is more stable, and is more suitable for the adjustment of the cage system. It also greatly reduces the length of the optical path, significantly reduces the volume and weight of the measurement system, and makes the system more compact and portable.

[0040] The numerical reconstruction module uses the swept-frequency digital holography method to reconstruct the surface morphology of the object to be measured. The digital holographic measurement system uses the same swept-frequency wavenumber interval of 50GHz to scan the object to be measured, and 10 interference patterns are obtained in one scanning cycle; the numerical reconstruction module performs digital holographic numerical reconstruction and phase extraction on the object in the computer, and obtains the surface morphology of the measurement area after phase unwrapping and denoising; by moving the displacement platform horizontally and vertically, repeating the above steps, four surface morphology images are finally obtained, as shown in the attached figure. Figure 2 As shown in (a), (b), (c) and (d).

[0041] In the numerical reconstruction module, the Fresnel diffraction formula is used to calculate and reconstruct the diffracted light field Γ(ξ,η,n) after propagation d based on the collected interference pattern. The following formula (4) can be obtained:

[0042]

[0043] in, is the inverse Fourier transform, and the diffracted light is calculated using the fast Fourier transform. d is the distance between the object to be measured and the CCD phase plane. The object light amplitude I(ξ,η,n) and phase can be calculated according to the following formula (5): information:

[0044]

[0045] Wherein, 4πnΔkΔd mod(2π) represents the remainder of 2π when the phase 4πnΔkΔd is taken; the phase calculated by equation (5) is the wrapped phase, and the actual phase difference is obtained by phase unwrapping, as shown in equation (6):

[0046] Δφ=φ(ξ,η,N)-φ(ξ,η,0)=4π(N-1)ΔkΔd (6)

[0047] Where N represents the number of holographic interference patterns, and N-1 represents the total number of changes in the light wave number. Here, the light wave number changes 9 times in one frequency sweep cycle, and 10 holographic interference patterns are obtained. According to formula (6), the distance d from the object to the camera surface is calculated, that is, the surface morphology distribution of the object, as shown in the following formula (7):

[0048]

[0049] The image stitching module uses an unsupervised deep learning method to stitch the surface morphology images obtained by numerical reconstruction in a large field of view. This method first stitches two adjacent images, and then stitches the two stitched images together again, eventually stitching four images together. This method consists of two stages, namely the deformation stage and the stitching stage. In the deformation stage, the method takes two images with overlapping areas as input, which are denoted as reference images I and I respectively. re Target image I ta First, the image is deformed as a whole based on direct linear transformation, and then a fine transformation based on thin plate spline interpolation is used to obtain a robust and flexible deformation, and the deformed image I is obtained. wr and I wt ; In the stitching stage, an extended DeepLabv3+ network is proposed as the stitching network to transform the deformed image I wr , I wt As its input, it predicts the stitching mask M sr , M st The stitched image I can be seamlessly synthesized by the following formula (8). The flowchart of the image stitching method based on unsupervised deep learning is shown in the attached figure. Figure 3 As shown:

[0050] I=M sr I wr +M st I wt (8)

[0051] In the feature extraction stage, the reference image I is input re and target image i ta , ResNet50 is used as the semantic feature extraction module to extract the semantic features of the two pictures, and the sizes of the two original pictures are obtained. and The feature map of and

[0052] In the overall deformation stage, the CCL (Contextual Correlation Layer) algorithm is used to transform the feature map and The correlation between them is calculated to estimate the initial movement of the pixels in the image; then a regression network consisting of 3 convolutional blocks, 3 maximum pooling layers and 3 fully connected layers is used to estimate the parameterization of the homography matrix of the four corner points and predict the initial movement of the four corner points to obtain the homography matrix H of the four corner points 4point ; Finally, direct linear transformation is used to transform the four corner point homography matrix H 4point Convert to homography matrix H.

[0053] In the fine deformation stage, the homography matrix H is first used to transform the feature map Transform to get Then the CCL algorithm is used to and The correlation between them is calculated; 16×16 grid control points are defined and evenly distributed on the deformed image. The results of CCL calculation are passed through a regression network composed of 4 convolution blocks, 4 maximum pooling layers and 3 fully connected layers to predict the fine movement of each grid control point; then, the thin plate spline interpolation algorithm is used to calculate the movement of the remaining pixel points except the control points to obtain the deformed image.

[0054] Create a completely black mask based on the size of the image after the reference image and the target image are stitched together. The size of the mask is sufficient to contain the reference image and the target image to ensure that there is enough space during the stitching process. Place the reference image on the completely black mask and determine its position in the completely black mask based on its relative position in the stitched image to generate the corresponding reference image I wr And the corresponding black and white mask M wr Similarly, the target image is deformed to align it with the reference image, its position in the black mask is determined, and the corresponding deformed target image I is generated. wt And the corresponding black and white mask M wt The four images generated above I wr 、M wr ,I wt 、M wt As input to the stitching phase.

[0055] In the image stitching stage, in order to solve the problem of insufficient processing of seam details in the image stitching process under complex scenes, the present invention proposes an extended DeepLabv3+ network as a stitching network, and uses the difference between the features of the reference deformed image and the target deformed image obtained by the encoder as the input of the decoder to predict the stitched reference mask image; at the same time, the decoder module structure of DeepLabv3+ is optimized, which significantly improves the network's ability to process complex scenes and details. The network structure diagram of the image stitching stage based on the extended DeepLabv3+ is shown in the attached figure. Figure 4 shown.

[0056] The extended DeepLabv3+ network adopts an encoder-decoder framework. The encoder reduces the spatial resolution of the image, extracts the multi-scale features and abstract deep features of the input image, and expands the receptive field. The encoder consists of two parts: the backbone network and ASPP; the backbone network uses ResNet101 to extract the deep features of the input image. The input image I wr and I wt Passing through 5 layers of network L0 , L 1 , L 2 , L 3 , L 4 Processing to obtain information of different scales, namely LF r0 , LF r1 , LF r2 , LF r3 , LF r4 and LF t0 , LF t1 , LF t2 , LF t3 , LF t4 ; The ASPP module receives the output LF of the last layer of the backbone network r4 and LF t4 As input, it consists of 4 dilated convolution blocks with different expansion rates and 1 global average pooling block. These blocks work in parallel to generate 5 sets of feature maps. These feature maps are merged and processed by convolution to reduce the number of channels and obtain the deep feature HF. r and HF t ,These deep features are then input into the decoder module.

[0057] The decoder uses multiple upsampling and multiple feature fusions to restore the abstract feature information in the encoder to image size output, allowing the network to make detailed predictions on the image at the pixel level. wr and I wt The difference between the low-level feature maps of different layers after the backbone network and the output of the ASPP module is used as input, i.e., LF r1 -LF t1 , LF r2 -LF t2 , LF r3 -LF t3 and HF r -HF t The decoding process introduces multi-layer lateral connections, and the deep features output by the ASPP module are upsampled step by step and fused with the shallow features output by the corresponding backbone network. The whole process is completed after 3 upsampling and 3 feature fusions to obtain the final prediction result, that is, the deep feature HF r -HF t And the shallow feature LF after 1×1 convolution r3 -LF t3 After merging, SF is obtained by 3×3 convolution feature fusion. 3 SF 3 After 2 times upsampling, it is combined with the shallow feature LF after 1×1 convolution r2 -LF t2 After merging, SF is obtained by 3×3 convolution feature fusion.2 SF 2 After 2 times upsampling, it is combined with the shallow feature LF after 1×1 convolution r1 -LF t1 After merging, SF is obtained by 3×3 convolution feature fusion. 1 ; Finally, SF 1 Upsample by 4 times and output the prediction result S with the same size as the original out The optimized decoder achieves efficient fusion of deep features and multi-level shallow features, which not only improves the network's ability to capture detailed features, but also further enhances the network's ability to recognize and express multi-scale features during multiple feature fusion processes.

[0058] In the stitching image generation stage, the reference mask image M of the stitching stage is calculated according to the prediction result of the decoder. sr , as shown in the following formula (9):

[0059] M sr =M wr -M wr M wt +M wr M wt S out (9)

[0060] Among them, M wr Indicates the reference mask image obtained in the deformation stage; M wr M wt It represents the overlapping area of ​​the reference mask image and the target mask image obtained in the deformation stage, which is the key part in the stitching and determines how the two images transition smoothly; M wr M wt S out It represents the part of the reference image predicted by the model in the overlapping area of ​​the reference mask image and the target mask image. Similarly, the target mask image M in the stitching stage is calculated. st , as shown in the following formula (10):

[0061] M st =M wt -M wr M wt +M wr M wt (1-S out ) (10)

[0062] Among them, M wt Indicates the target mask image obtained in the deformation stage; M wr M wt (1-S out) represents the target image portion predicted by the model in the overlapping area of ​​the reference mask image and the target mask image. sr and M st , substitute into formula (8) to calculate the final stitched image I. Similarly, repeat the above operation to obtain multiple stitched images, and then continue to stitch the stitched images to obtain the attached Figure 2 The 2×2 stitching of the four images (a), (b), (c), and (d) is shown in the attached Figure 5 shown.

[0063] The above description of the present invention and its implementation method is not limited thereto, and the accompanying drawings are only one of the embodiments of the present invention. Without departing from the purpose of the present invention, any structure or embodiment similar to the scheme designed without creativity shall fall within the protection scope of the present invention.

Claims

1. A compact topography measurement system and method based on field of view stitching digital holography, characterized in that: The system includes a laser control and generation module, an off-axis holographic optical path module, an image acquisition and transmission module, a numerical reconstruction module and an image stitching module, etc.; the laser control and generation module includes a computer 101, a tunable laser 102, etc.; the off-axis holographic optical path module includes a fiber coupler 103, a laser beam expander 104, a fiber collimator 105, an adjustable attenuator 106, an aspheric focusing lens 107, a pinhole 108, a plano-convex lens 109, an adjustable aperture 110 and a reflector 111, etc.; the image acquisition and transmission module includes a beam splitter prism 112, a camera 113, etc.; Among them, the computer 101 in the laser control and generation module controls the tunable laser 102 to generate a laser with a fixed frequency interval step change. The output laser is divided into two paths after passing through the fiber coupler 103 in the off-axis holographic optical path module. One path is the measurement optical path, which is irradiated to the surface of the object to be measured 114 fixed on the displacement stage 115 after passing through the laser beam expander 104, and enters the image acquisition and transmission module after diffuse reflection on the surface of the object; the other path is the reference optical path, which is focused by the aspheric condenser lens 107 through the pinhole 108 after passing through the fiber collimator 105 and the adjustable attenuation plate 106, and then expanded and collimated by the plano-convex lens 109 to filter the spatial noise in the light beam, and the spot size is adjusted by the adjustable aperture 110, and enters the image acquisition and transmission module after reflection by the reflector 111; the reference optical path after entering the image acquisition and transmission module The measuring light and the reference light are combined by the beam splitter prism 112 to generate interference, and the camera 113 collects the interference pattern and transmits it to the computer; the object to be measured 114 is fixed in the displacement platform 115; in the numerical reconstruction module, M interference patterns are obtained in one scanning cycle, and digital holographic numerical reconstruction and phase extraction are performed on them in the computer. After phase unwrapping and denoising, the surface morphology of the measurement area is obtained; the displacement platform is moved in sequence, horizontally for m times and vertically for n times, and the above steps are repeated to obtain m×n surface morphology images; the image stitching module adopts an unsupervised deep learning image stitching algorithm, and after deformation and stitching operations, the pairwise stitching of the surface morphology images is realized. Repeating the stitching operation can obtain multiple pairwise stitching images, and then the stitched images are further stitched to obtain a surface morphology measurement image with a large field of view.

2. According to claim 1, a compact morphology measurement system and method based on field of view stitching digital holography is characterized in that: The off-axis holographic optical path module and the image acquisition and transmission module use a cage structure to assemble various optical devices together. The spatial filter improves the beam quality by filtering the spatial noise in the beam. An aspherical condenser lens is used instead of an objective lens or a spherical condenser lens to achieve beam focusing, and the lens is fixed in a Z-axis displacement platform, thereby avoiding the problem of difficult adjustment and large volume when using an objective lens to focus the beam in a cage structure, and at the same time overcoming the limitation of difficulty in achieving precise focusing when using a spherical condenser lens due to spherical aberration and large focal spot size; The adjustable attenuation plate 106, the aspheric focusing lens 107, the pinhole 108, the plano-convex lens 109, and the adjustable diaphragm 110 in the off-axis holographic optical path module constitute a spatial filter; after the reference light beam is collimated by the optical fiber collimator 105 and the adjustable attenuation plate 106, it is focused by the aspheric focusing lens 107, and then passes through the pinhole 108 centered on the optical axis to block the noise ring in the light beam, transfer most of the laser energy to generate a pure Gaussian light beam, and then expands and collimates through the plano-convex lens 109, and is adjusted by the adjustable diaphragm 110 to form a uniform light beam of a certain size; the aspheric focusing lens is fixed in the Z-axis displacement platform, and the Z-axis displacement platform is fixed in the cage system, and the Z-axis displacement platform accurately controls its position, and selects appropriate pinholes, plano-convex lenses and other components through precise calculations, so as to achieve accurate light beam focusing and eliminate the spatial non-uniformity of the light beam; The size of the pinhole depends on the size of the incident beam and the focal length of the focusing lens. The focal size is calculated according to the diffraction-limited spot formula. Calculation, λ represents the wavelength, f represents the focal length, and r is 1 / e of the incident beam 2 radius; the laser after passing through the collimator is considered to be collimated within the Rayleigh length, so 1 / e of the laser 2 The radius is approximately the beam waist radius of the collimator; therefore, according to the formula Calculate the size D of the diffraction-limited spot of the focused light spot; the selection of the pinhole should take into account the passage of the main lobe of the focal spot, and ensure the beam quality while ensuring that more than 95% of the energy passes through. The actual pinhole size should be about 30% larger than the diffraction-limited spot size. Select a suitable pinhole according to the size of the pinhole model; at the same time, in order to ensure the minimum diameter of the output light spot of the system, calculate the theoretical minimum focal length of collimation based on similar triangles, and select a plano-convex lens with a suitable focal length according to the model of the plano-convex lens to expand and collimate the beam after focusing and filtering, and add an adjustable aperture after the plano-convex lens to adjust the size of the light spot to meet the needs of optical path adjustment; The cage-type precision adjustable spatial filter is easier to align the optical axis, the system is more stable, and is more suitable for the adjustment of the cage system. It also greatly reduces the length of the optical path, significantly reduces the volume and weight of the measurement system, and makes the system more compact and portable.

3. According to claim 1, a compact morphology measurement system and method based on field of view stitching digital holography is characterized in that: In the image stitching module, what is to be stitched is a sub-micron surface morphology map. There is a lot of detailed texture information on the morphology of complex objects. An extended DeepLabv3+ network is used as the network for the image stitching stage, the decoder module structure of DeepLabv3+ is optimized, jump connections are introduced, multiple upsampling and multiple feature fusions are used, the segmentation accuracy of the stitching network is improved, the stitched seams are more natural, the color difference and geometric misalignment on both sides of the seams are minimized, and the ability of the stitching network to process seam details in complex scenes is significantly improved, and the traditional image stitching method of feature extraction and matching is prone to mismatch in this scene, and the problem of uneven image transition, color difference and geometric misalignment caused by insufficient pixel-level fusion at the seams is avoided; the image stitching module uses an unsupervised deep learning method to stitch the surface morphology images obtained by numerical reconstruction with a large field of view. The method is divided into an image deformation stage and an image stitching stage; In the image deformation stage, two images with overlapping areas are taken as input, which are denoted as reference images I and I respectively. re and the target image I ta , after semantic feature extraction, overall deformation and fine deformation, the deformed image I is obtained wr , I wt And black and white mask M wr , M wt ; In the image stitching stage, in order to solve the problem of insufficient processing of seam details in the image stitching process in complex scenes, an extended DeepLabv3+ network is proposed as the stitching network, which improves the network's ability to process seam details in complex scenes. wr ,I wt As its input, it is generated by the encoder, decoder and spliced ​​image in sequence, and the spliced ​​mask M is predicted sr ,M st , and the stitched image I is seamlessly synthesized according to the following formula (1). The specific steps of the stitching stage are as follows: I=M sr I wr +M st I wt (1) Step 1: The encoder of the extended DeepLabv3+ network consists of a backbone network and ASPP; the backbone network is composed of ResNet101, which is used to extract deep features of the input image. wr and I wt After 5 layers of network processing, information of different scales is obtained, namely, LF r0 , LF r1 , LF r2 , LF r3 , LF r4 and LF t0 , LF t1 , LF t2 , LF t3 , LF t4 ; The ASPP module receives the output LF of the last layer of the backbone network r4 and LF t4 As input, it consists of 4 dilated convolution blocks with different expansion rates and 1 global average pooling block. These blocks work in parallel to generate 5 sets of feature maps. These feature maps are merged and processed by convolution to reduce the number of channels and obtain the deep feature HF. r and HF t ,These features are then fed into the decoder module; Step 2: The decoder of the extended DeepLabv3+ network uses multiple upsampling and multiple feature fusions to restore the abstract feature information in the encoder to an output of image size, so that the network can make refined predictions on the image at the pixel level; The decoder takes an input image I wr and I wt The difference between the low-level feature maps of different layers after the backbone network and the output of the ASPP module is used as input, i.e., LF r1 -LF t1 , LF r2 -LF t2 , LF r3 -LF t3 and HF r -HF t ; Introduce multi-layer lateral connections, upsample the deep features output by the ASPP module step by step and fuse them with the shallow features output by the corresponding backbone network. The whole process is done through 3 upsampling and 3 feature fusions to get the final prediction result, i.e., the deep feature HF r -HF t And the shallow feature LF after 1×1 convolution r3 -LF t3 After merging, SF3 is obtained by 3×3 convolution feature fusion; SF3 is upsampled by 2 times and combined with the shallow feature LF after 1×1 convolution. r2 -LF t2 After merging, SF2 is obtained by 3×3 convolution feature fusion; SF2 is then upsampled by 2 times and combined with the shallow feature LF after 1×1 convolution. r1 -LF t1 After merging, SF1 is obtained by 3×3 convolution feature fusion; finally, SF1 is upsampled 4 times and the output is the prediction result S with the same size as the original out ; The optimized decoder realizes the efficient fusion of deep features and multi-level shallow features, which not only improves the network's ability to capture detailed features, but also further enhances the network's ability to recognize and express multi-scale features during multiple feature fusion processes; Step 3: According to the decoder's prediction result S out , calculate the reference mask image M in the stitching stage sr , as shown in the following formula (2): M sr =M wr -M wr M wt +M wr M wt S out (2) Among them, M wr Indicates the reference mask image obtained in the deformation stage; M wr M wt represents the overlapping area of ​​the reference mask image and the target mask image obtained in the deformation stage; M wr M wt S out represents the part of the reference image predicted by the model in the overlapping area of ​​the reference mask image and the target mask image; similarly, the target mask image M in the stitching stage is calculated st , as shown in the following formula (3): M st =M wt -M wr M wt +M wr M wt (1-S out ) (3) Among them, M wt Indicates the target mask image obtained in the deformation stage; M wr M wt (1-S out ) represents the target image portion predicted by the model in the overlapping area of ​​the reference mask image and the target mask image; then, the combined mask image M obtained by the above calculation sr and M st , substitute into formula (1) to calculate the final stitched image I; similarly, repeat the above operation to obtain multiple two-by-two stitched images, and then continue to stitch the stitched images to obtain a stitched image with a larger field of view.

Images