Vision Transform-based hologram depth prediction system
Through the Hologram Depth Prediction System based on Vision Transformer, the focus position is directly predicted, which solves the problem of insufficient focus accuracy and real-time performance in digital holographic microscopy, and achieves efficient hologram imaging optimization.
Patent Information
- Application Number
- CN202510586002.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-05-08
- Publication Date
- 2025-08-08
AI Technical Summary
In the digital holographic microscopy imaging, the accuracy and real-time performance of the focus position are insufficient, the calculation cost is high, and it is difficult to meet the needs of high precision and fast focus.
The hologram depth prediction system based on Vision Transformer is adopted, and the holograms are collected through digital holographic optical paths, data preprocessing is performed and ViT model is input. The Transformer encoder and regression head are used to directly predict the focus distance, and the imaging process is optimized.
It improves the clarity and accuracy of digital holographic microscopy, overcomes the shortcomings of traditional methods in high precision and real-time, and achieves fast and high-precision focusing.
Smart Images
Figure CN120451241A_ABST
Abstract
Description
(1) Technical field
[0001] The present invention belongs to the field of digital holographic microscopy and machine vision, and in particular relates to a hologram depth prediction system based on VisionTransformer. (2) Background technology
[0002] The quality of the reconstructed image of a digital hologram often determines the quality of subsequent 3D reconstruction, which in turn affects the results of 3D measurement. Therefore, improving the quality of the reconstructed image from a digital hologram has become a key research topic in digital holographic microscopy. In digital holographic microscopy, accurately determining the focal position of an object is crucial for holographic reconstruction. However, in practice, the focal position cannot be accurately determined directly and requires the use of a focusing algorithm. Traditionally, this approach involves first reconstructing a set of images, then calculating the sharpness of each reconstructed image using a focus metric (such as entropy or variance), with the position corresponding to the sharpest image or the image with the extreme value of the computational coefficient being used as the focal position. In practical applications, to improve focusing accuracy, after performing overall coarse focusing, the reconstruction distance is often shortened for fine focusing to obtain the final result. While effective, this approach is computationally expensive and time-consuming. Furthermore, for purely phase-sensitive objects, phase jumps in the reconstructed result are often misinterpreted by the algorithm as sharp structures, necessitating a post-reconstruction phase unwrapping step, further increasing the computational cost.
[0003] In recent years, the field of machine learning (ML) has seen rapid development. Unlike traditional computer algorithms that process data using well-defined patterns or rules, ML can perform specific tasks by learning patterns or rules directly from a set of data. Deep learning has also provided new insights into the imaging process and opened up new application prospects for digital holographic microscopy. Deep learning has now proven effective in addressing many problems involved in digital holographic microscopy, such as holographic reconstruction and automatic focusing. Compared to traditional methods, these approaches combined with deep learning have achieved higher imaging efficiency. This is because traditional methods process individual images in isolation and require prior knowledge, while deep learning methods generally do not require extensive prior knowledge. Deep learning-based reconstruction methods can leverage large datasets to impose implicit constraints on the reconstruction problem. Through a network learning optimization phase, deep learning-based methods can achieve real-time reconstruction, while traditional algorithms require iterative optimization for each new image. This demonstrates the high efficiency of deep learning methods.
[0004] The improved ViT model proposed in this paper can automatically predict the position of the focal plane and further optimize the entire imaging process. In this way, the clarity and accuracy of digital holographic microscopy imaging can be improved, overcoming the shortcomings of traditional methods in terms of high precision, real-time performance, and plane focusing accuracy. (3) Summary of the invention
[0005] The present invention proposes a hologram depth prediction system based on Vision Transformer, which aims to solve the problems of the accuracy of the focusing distance of the hologram and the slow prediction time.
[0006] To achieve the above objectives, the present invention provides a hologram depth prediction system based on Vision Transformer. The method comprises the following steps:
[0007] S1: Build a digital holographic optical path to measure the sample to be tested and record the hologram of the sample to be tested.
[0008] S2: Data preprocessing is performed on the obtained holograms, and images are uniformly cropped to 224×224 size. Then, data enhancement is performed by rotation and translation brightness enhancement. Finally, all images are normalized and fed into the ViT model.
[0009] S3: The preprocessed image is divided into a sequence of 16×16 tiles, which are converted into a 768-dimensional vector by linear projection and added with a learnable positional encoding. The [CLS] token inserted at the beginning and the 196 tiles are input into a 12-layer Transformer encoder.
[0010] S4: Extract the 768-dimensional global feature vector of the [CLS] token from the final Transformer layer, reduce the dimension layer by layer through a three-level fully connected network (384→196→1), apply LayerNorm and GELU activation functions to each layer, and finally denormalize the output to the physical depth value.
[0011] S5: Repeat steps S3 to S4, and use the training set to continuously update the model parameters until the model converges.
[0012] In the above-mentioned S1, the digital holographic optical path is a microscopic holographic optical path.
[0013] In S2, the image preprocessing method includes uniformly cropping the image into a picture of 224×224 size, and then performing translation and rotation on the cropped picture to enhance brightness.
[0014] In S4, the multi-head attention mechanism in the Transformer network can process different subspaces of the input information in parallel. The model can capture important features in the image from different angles, thereby improving focusing accuracy.
[0015] From a general perspective, the vector x in is its input, x outis its output. In the case of ViT, N is fixed for all Transformer Block layers. The input data will pass through different learning matrices are mapped to query vectors q i , key vector k i , value vector v i :
[0016]
[0017]
[0018]
[0019] For each head i , attention is given by the query vector q i , key vector k i , value vector v i calculate:
[0020] head i =Attention(Q,K,V)
[0021] Attention normalizes the query vector q by first applying Softmax i , key vector k i This output is then used as a weight to apply to the value vector v i , thus obtaining the final attention output:
[0022]
[0023] Output vector x out By Concat(head1…head h ) and a learnable matrix W o The vector product of is obtained:
[0024] x out =Concat(head1…head h )W o
[0025] The population can be expressed as:
[0026] x out =MHA(x in )
[0027] In S4, in the ViT regression task, after being processed by the Transformer encoder, the feature vector corresponding to the [CLS] token in the output feature sequence is regarded as the global representation of the entire input hologram. The [CLS] feature is input into a regression head. Finally, the feature is compressed into a one-dimensional continuous value through the last layer of linear transformation, and the corresponding physical focus distance is output. Throughout the entire process, the model only uses [CLS]Token as a global feature carrier to directly complete the mapping from image to continuous scalar.
[0028] The beneficial effects of the present invention are:
[0029] This paper designs a hologram depth prediction system based on Vision Transformer. This method uses a deep learning model to accurately predict the optimal focus position. The model automatically predicts the position of the focal plane and further optimizes the entire imaging process. This approach improves the clarity and accuracy of digital holographic microscopy imaging, overcoming the limitations of traditional methods in terms of high precision, real-time performance, and focusing accuracy. (IV) Description of the accompanying drawings
[0030] Figure 1 It is a flow chart of a hologram depth prediction system based on Vision Transformer of the present invention;
[0031] Figure 2 This is the ViT regression network architecture diagram proposed in the present invention; (V) Specific implementation methods
[0032] The following will be combined with the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements with the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present invention, and should not be understood as limiting the present invention.
[0033] See also Figure 1 The present invention provides a hologram depth prediction system based on Vision Transformer, comprising the following steps:
[0034] S1: Build a digital holographic optical path to measure the sample to be tested and record the hologram of the sample to be tested.
[0035] S2: Data preprocessing is performed on the obtained holograms, and images are uniformly cropped to 224×224 size. Then, data enhancement is performed by rotation and translation brightness enhancement. Finally, all images are normalized and fed into the ViT model.
[0036] S3: The preprocessed image is divided into a sequence of 16×16 tiles, which are converted into a 768-dimensional vector by linear projection and added with a learnable positional encoding. The [CLS] token inserted at the beginning and the 196 tiles are input into a 12-layer Transformer encoder.
[0037] S4: Extract the 768-dimensional global feature vector of the [CLS] token from the final Transformer layer, reduce the dimension layer by layer through a three-level fully connected network (384→196→1), apply LayerNorm and GELU activation functions to each layer, and finally denormalize the output to the physical depth value.
[0038] S5: Repeat steps S3 to S4, and use the training set to continuously update the model parameters until the model converges.
[0039] The following is a further explanation based on the specific implementation steps:
[0040] In step S2, the image preprocessing method includes cropping the image into a 224×224 size image, and then performing translation and rotation on the cropped image to enhance the brightness.
[0041] In step S4, the multi-head attention mechanism in the Transformer network can process different subspaces of the input information in parallel. The model can capture important features in the image from different angles, thereby improving focusing accuracy.
[0042] From a general perspective, the vector x in is its input, x out is its output. In the case of ViT, N is fixed for all Transformer Block layers. The input data will pass through different learning matrices are mapped to query vectors q i , key vector k i , value vector v i :
[0043]
[0044]
[0045]
[0046] For each head i , attention is given by the query vector q i , key vector k i , value vector v i calculate:
[0047] head i =Attention(Q,K,V)
[0048] Attention normalizes the query vector q by first applying Softmax i , key vector k i This output is then used as a weight to apply to the value vector v i , thus obtaining the final attention output:
[0049]
[0050] Output vector x out By Concat(head1…head h ) and a learnable matrix W o The vector product of is obtained:
[0051] x out =Concat(head1…head h )W o
[0052] The population can be expressed as:
[0053] x out =MHA(x in )
[0054] In step S4, in the ViT regression task, after processing by the Transformer encoder, the feature vector corresponding to the [CLS] token in the output feature sequence is regarded as the global representation of the entire input hologram. The [CLS] feature is input into a regression head. Finally, the feature is compressed into a one-dimensional continuous value through the last layer of linear transformation, and the corresponding physical focus distance is output. Throughout the entire process, the model only uses the [CLS] Token as a global feature carrier to directly complete the mapping from image to continuous scalar.
[0055] The above disclosure is only a preferred embodiment of the present invention, and certainly cannot be used to limit the scope of the rights of the present invention. Ordinary technicians in this field can understand that all or part of the processes of the above embodiment and equivalent changes made in accordance with the claims of the present invention are still within the scope of the invention.
Claims
1. A hologram depth prediction system based on Vision Transformer, characterized in that: The method comprises the following steps: S1: Build a digital holographic optical path to measure the sample to be tested and record the hologram of the sample to be tested. S2: Data preprocessing is performed on the obtained holograms, and images are uniformly cropped to 224×224 size. Then, data enhancement is performed by rotation and translation brightness enhancement. Finally, all images are normalized and fed into the ViT model. S3: The preprocessed image is divided into a sequence of 16×16 tiles, which are converted to a 768-dimensional vector via linear projection and then added with a learnable positional encoding. The [CLS] token inserted at the beginning and the 196 tiles are fed into a 12-layer Transformer encoder. S4: Extract the 768-dimensional global feature vector of the [CLS] token from the final Transformer layer, reduce the dimension layer by layer through a three-level fully connected network (384→196→1), apply LayerNorm and GELU activation functions to each layer, and finally denormalize the output to the physical depth value. S5: Repeat steps S3 to S4, and use the training set to continuously update the model parameters until the model converges.
2. The hologram depth prediction system based on Vision Transformer according to claim 1, characterized in that: In the above-mentioned S1, the digital holographic optical path is a microscopic holographic optical path.
3. The hologram depth prediction system based on Vision Transformer according to claim 1, characterized in that: In S1, the sample to be measured is any microstructure used for holographic imaging, and is a transmission type sample or a reflection type sample.
4. The hologram depth prediction system based on Vision Transformer according to claim 1, characterized in that: In S2, the image preprocessing method includes uniformly cropping the image into a picture of 224×224 size, and then performing translation and rotation on the cropped picture to enhance brightness.
5. The hologram depth prediction system based on Vision Transformer according to claim 1, characterized in that: In S4, the multi-head attention mechanism in the Transformer network can process different subspaces of the input information in parallel. The model can capture important features in the image from different angles, thereby improving focusing accuracy.
6. The hologram depth prediction system based on Vision Transformer according to claim 1, characterized in that: In S4, in the ViT regression task, after being processed by the Transformer encoder, the feature vector corresponding to the [CLS] token in the output feature sequence is regarded as the global representation of the entire input hologram. The [CLS] feature is input into a regression head. Finally, the feature is compressed into a one-dimensional continuous value through the last layer of linear transformation, and the corresponding physical focus distance is output. Throughout the entire process, the model only uses [CLS]Token as a global feature carrier to directly complete the mapping from image to continuous scalar.
7. The hologram depth prediction system based on Vision Transformer according to claim 1, characterized in that: The hologram depth prediction system of the Vision Transformer combined with the regression head significantly improves the final prediction accuracy.