Calculation ghost imaging reconstruction method based on Vision Transform and U-Net network

Abstract

The invention discloses a calculation ghost imaging reconstruction method based on a Vision Transform and a U-Net network, and the method comprises the following steps: 1, enabling a Hadamard matrix of a cut cake sequence to serve as a phase mask, sequentially embedding the Hadamard matrix into a spatial light modulator, sequentially projecting the Hadamard matrix to a target image, and collecting a total light intensity sequence through a barrel detector; 2, partitioning the Hadamard matrix of the cake cutting sequence and the light intensity sequence, performing linear projection on each block to obtain a high-dimensional vector token, adding a position code to each high-dimensional vector token, inputting the high-dimensional vector token into a Transform encoder for processing, and converting the output into a two-dimensional matrix; and step 3, inputting the two-dimensional matrix obtained in the step 2 into the U-Net network to enhance transmission and fusion of feature information until a loss function of the ViT-U-Net network is kept stable, and obtaining a result of calculating ghost imaging. The image reconstructed by the method is high in quality and high in reconstruction efficiency.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of image processing, and particularly relates to a computational ghost imaging reconstruction method based on a Vision Transformer and a U-Net network. BACKGROUND The reconstruction method of traditional computational ghost imaging is mainly realized through a second-order correlation algorithm or a compressed sensing technology. The compressed sensing technology is used for reconstruction by utilizing the sparse characteristics of a signal, but the reconstruction effect is still not ideal in some cases, which also causes problems such as low reconstruction efficiency and poor quality of the computational ghost imaging technology in actual application. In order to overcome these shortcomings, researchers are constantly exploring new reconstruction algorithms to improve the practicability of the computational ghost imaging technology. In recent years, with the development of deep learning technology, convolutional neural networks and fully convolutional neural networks have made significant progress in the field of image processing. However, these methods have problems such as lack of sensitivity to details and lack of spatial consistency when processing complex images, resulting in the quality of the processed images still not reaching the ideal state. SUMMARY

[0002] The purpose of the application is to provide a computational ghost imaging reconstruction method based on a Vision Transformer and a U-Net network, which solves the problem of poor image quality reconstruction of the existing ghost imaging method.

[0003] The technical solution adopted by the application is a computational ghost imaging reconstruction method based on a Vision Transformer and a U-Net network, which is implemented according to the following steps: Step 1, embed the Hadamard matrix of the cut cake sequence as a phase mask into the spatial light modulator in turn and project it onto the target image , collect the total light intensity sequence through a bucket detector ; Step 2, divide the Hadamard matrix of the cut cake sequence and the light intensity sequence into blocks, linearly project each block to obtain a high-dimensional vector token, add a position code to each high-dimensional vector token and input it into a Transformer encoder for processing, and convert the output into a two-dimensional matrix; Step 3, input the two-dimensional matrix obtained in step 2 into a U-Net network to enhance the transmission and fusion of feature information until the loss function of the ViT-U-Net network remains stable, and obtain the result of the computational ghost imaging. The application is characterized in that The specific process of step 1 is: Step 1.1, use a 2-order basic block to construct a Hadamard matrix of a natural sequence of any order, and the 2-order basic block is represented as: (1) provided with The Hadamard matrix of natural sequence of order n is expressed as: (2) Suppose the target image has a size of pixels and satisfies , where is an integer, then the size of the Hadamard matrix of natural sequence is also ; Step 1.2, using the cake cutting strategy, the Hadamard matrix of obtained in step 1.1 is rearranged row by row to integrate into matrices of pixels, and these matrices are reordered in ascending order of the number of connected domains, and each matrix is tiled into row vectors to form a cake-cutting sequence Hadamard matrix, which is used as a phase mask embedded in a spatial light modulator in the calculation of ghost imaging; Step 1.3, the cake-cutting sequence Hadamard matrix obtained in step 1.2 is sequentially embedded into the spatial light modulator in the actual calculation of ghost imaging system and projected onto the target image in order, and the light intensity sequence of the target image is obtained by detecting with a bucket detector.

[0004] In step 1.3, the expression of the light intensity sequence is: (3) In equation (3), denotes the horizontal and vertical coordinates of the target image, denotes the th sampled cake-cutting sequence Hadamard matrix, denotes the horizontal and vertical coordinates of the cake-cutting sequence Hadamard matrix element.

[0005] The specific process of step 2 is as follows: Step 2.1, the light intensity sequence is converted into a two-dimensional matrix by row-column conversion to obtain the converted light intensity sequence, and the cake-cutting sequence Hadamard matrix and the converted light intensity sequence is divided into fixed-size blocks, each block is flattened into a one-dimensional vector, and the one-dimensional vector is converted into a high-dimensional vector token through a linear layer (also known as an embedding layer), a position encoding is added to each high-dimensional vector token to obtain a feature vector; Step 2.2, input all the feature vectors obtained in step 2.1 into the encoder of the Transformer, process through the multi-head self-attention mechanism, and obtain the final output , and then input the final output into a fully connected feedforward neural network layer for processing to obtain a higher layer feature vector, and convert the higher layer feature vector into a two-dimensional matrix as the input of the subsequent U-net network.

[0006] In step 2.1, the expression of the feature vector is: (4) In formula (4), is the feature vector; is the weight matrix; is the size of the block, is the dimension of the high-dimensional vector token; is the position encoding vector of the th high-dimensional vector token; is the th high-dimensional vector token.

[0007] The specific process of step 2.2 is as follows: Step 2.2.1, after the feature vector obtained in step 2.1 is linearly transformed by the weight matrix obtained by learning, a query matrix , a key matrix , and a value matrix are generated. (6) In formula (6), indicates the th attention head; , and respectively indicate the weight matrix corresponding to the query matrix, the key matrix, and the value matrix; indicates the feature vector; The attention score matrix of each attention head is calculated, and the expression is: (7) In formula (7), is the dimension of the key vector; The attention score matrix is multiplied by the value matrixsoftmax After normalization, the attention weight matrix is obtained , and the expression is: (8) The value matrix is multiplied by the attention weight matrix to obtain the weighted sum of each attention head , that is, the output of each attention head, and the expression is: (9) The weighted sum of each attention head is merged along the last dimension to obtain the output of the self-attention layer , and the expression is: (10) Subsequently, a linear transformation is applied to the output of the self-attention layer to obtain the final output , and the expression is: (11) In equation (11), represents the weight matrix of the fusion linear transformation; Step 2.2.2, the obtained final output is sent to a fully connected feedforward neural network layer for processing, and a higher layer feature vector is output; Step 2.2.3, the obtained higher layer feature vector is converted into a two-dimensional matrix as the input of the subsequent U-net network.

[0008] In step 2.2.2, the expression of the fully connected feedforward neural network layer processing is: (12) In equation (12), is an activation function; , is the weight , is the bias.

[0009] In step 3, the loss function of the ViT-U-Net network is: (13) In equation (13), is the reconstructed image predicted by the ViT-U-Net network, is the target image, are the height and width of the image, respectively.

[0010] The beneficial effects of the present application are: the present application is based on the computational ghost imaging reconstruction method of the Vision Transformer and the U-Net network, which combines the feature extraction ability of the Vision Transformer with the powerful image reconstruction ability of the U-Net, aims to efficiently and accurately restore high-quality ghost imaging results by significantly reducing the sampling times; the Vision Transformer is used as the core processing unit, which can effectively capture the global information in the image, has high processing speed, and can process image data of different sizes and resolutions without manually designing the convolution structure, on this basis, further integrate the U-Net network, use its symmetrical encoder-decoder structure and jump connection, effectively improve the fineness and edge preservation ability of image reconstruction, so that clear and accurate reconstructed images can still be generated under the condition of low sampling rate. BRIEF DESCRIPTION OF DRAWINGS

[0011] Figure 1 The flowchart of the present application based on the computational ghost imaging reconstruction method of the Vision Transformer and the U-Net network; Figure 2 The structure diagram of the present application based on the computational ghost imaging reconstruction method of the Vision Transformer and the U-Net network; Figure 3 The Hadamard matrix of the 16x16 cake sequence obtained in the present application based on the computational ghost imaging reconstruction method of the Vision Transformer and the U-Net network; Figure 4 The processing flowchart of the ViT model in the present application based on the computational ghost imaging reconstruction method of the Vision Transformer and the U-Net network; Figure 5 The training flowchart of the U-net network in the present application based on the computational ghost imaging reconstruction method of the Vision Transformer and the U-Net network; Figure 6 The result comparison diagram of the present application based on the computational ghost imaging reconstruction method of the Vision Transformer and the U-Net network, the traditional ghost imaging reconstruction method, and the TVAL3 algorithm of compressed sensing for reconstructing binary image "6" under different sampling rates; Figure 7 The reconstruction result diagram of the present application based on the computational ghost imaging reconstruction method of the Vision Transformer and the U-Net network, the traditional ghost imaging reconstruction method, and the TVAL3 algorithm of compressed sensing for the images of numbers "0-9" in the training set under the sampling rate of 3.12%. DETAILED DESCRIPTION

[0012] The present application will be described in detail below in combination with the drawings and specific embodiments.

[0013] Example 1 This invention presents a computational ghost imaging reconstruction method based on Vision Transformer and U-Net networks, such as... Figure 1 and Figure 2 As shown, please follow these steps: Step 1: The Hadamard matrix of the cake-cutting sequence is used as a phase mask and sequentially embedded into the spatial light modulator and projected onto the target image. Above, a bucket detector is used to collect the total light intensity sequence. ; Step 2, combine the Hadamard matrix of the cake-cutting sequence and the light intensity sequence. The process involves dividing the data into blocks, linearly projecting each block to obtain a high-dimensional vector token, adding a positional code to each high-dimensional vector token, and then inputting it into a Transformer encoder for processing. The output is then converted into a two-dimensional matrix. Step 3: Input the two-dimensional matrix obtained in Step 2 into the U-Net network to enhance the transmission and fusion of feature information until the loss function of the ViT-U-Net network remains stable, and obtain the result of calculating ghost imaging.

[0014] Example 2 Based on Example 1, the specific process of step 1 is as follows: Step 1.1: Construct the Hadamard matrix of a natural sequence of arbitrary order using 2-order basic blocks. The 2-order basic block is then represented as: (1) have The Hadamard matrix of a natural sequence of orders is represented as: (2) Assuming the target image The size is 100 pixels, and satisfy 1000 pixels ,in If the integer is an integer, then the size of the Hadamard matrix of the natural sequence is also . ; Step 1.2, using the cake cutting (CC) strategy, first cut the cake obtained in step 1.1... The Hadamard matrix is ​​rearranged row by row and integrated into indivual A matrix of pixels, sorted in ascending order by the number of connected components. Reorder the matrices, then flatten each matrix. The Hadamard matrix of the cake-cutting sequence is formed by a number of row vectors, and the Hadamard matrix of the cake-cutting sequence is used as the phase mask embedded in the spatial light modulator (SLM) in the computation of ghost imaging. The Hadamard matrix of a natural sequence contains only two values: +1 and -1. Its spatial distribution is square and symmetrical. Furthermore, the Hadamard matrix of a natural sequence is an orthogonal matrix, meaning that any two columns or rows selected are orthogonal. The key to its effectiveness as a measurement matrix and its excellent reconstruction results lies in its two important properties: column incoherence and row orthogonality. If the Hadamard matrix is ​​arranged in ascending order of the number of connected components, the coherent regions of each speckle will naturally exhibit a transition from wide to narrow. Therefore, by imagining each unsorted Hadamard matrix as a cake, one can count how many slices the cake needs to be cut into, with each slice considered a connected region. This method is called the cake-cutting strategy. Step 1.3: The Hadamard matrices of the cake-cutting sequence obtained in Step 1.2 are sequentially embedded into the spatial light modulator (SLM) of the actual computational ghost imaging system and then projected onto the target image in sequence. Above, a bucket detector is used to obtain the target image. Light intensity sequence ; The expression is: (3) In equation (3), Represents the x and y coordinates of the target image. This represents the Hadamard matrix of the cake-cutting sequence sampled in the i-th iteration. The x and y coordinates of the Hadamard matrix elements representing the cake-cutting sequence; light intensity sequence. The light intensity sequence is obtained by taking the Hadamard matrix corresponding to the i-th cake-cutting sequence.

[0015] Example 3 Based on Example 2, like Figure 4 As shown, the specific process of step 2 is as follows: Step 2.1, sequence the light intensity The rows and columns are converted into a two-dimensional matrix to obtain the converted light intensity sequence. Then, the Hadamard matrix of the cake-cutting sequence is used. The converted light intensity sequence is divided into fixed-size blocks. Each block is flattened into a one-dimensional vector. This one-dimensional vector is then transformed into a high-dimensional vector token through a linear layer (also known as an embedding layer). The high-dimensional vector token has a predefined dimension, which is the embedding dimension. A positional encoding is added to each high-dimensional vector token to preserve the positional information of each patch in the original image. The dimension of the positional encoding is the same as the predefined dimension, resulting in a feature vector. The expression for obtaining the eigenvector is: (4) In equation (4), For feature vectors; This is the weight matrix; For the size of the block, The dimension of the high-dimensional vector token; For the first The position encoding vector of a high-dimensional vector token; For the first A high-dimensional vector token.

[0016] Step 2.2: Input all the feature vectors obtained in Step 2.1 into the Transformer encoder, and process them through a multi-head self-attention mechanism to obtain the final output. Then the final output The vectors are fed into a fully connected feedforward neural network layer for processing to obtain higher-level feature vectors. These higher-level feature vectors are then converted into two-dimensional matrices and used as input to the subsequent U-Net network.

[0017] Example 4 Based on Example 3, the specific process of step 2.2 is as follows: The expression for obtaining the relationship between all feature vectors using a multi-head self-attention mechanism is as follows: (5) In equation (5), It is a query matrix, composed of all query vectors; It is a key matrix, composed of all key vectors; It is a value matrix, composed of all value vectors; It is the dimension of the key vector, used to scale the dot product attention to avoid computational errors. The gradient vanishing problem caused by excessively high dimensionality; Step 2.2.1, the specific implementation process of the multi-head self-attention mechanism is as follows: The feature vector obtained in step 2.1 is linearly transformed through the weight matrix obtained from learning to generate... , , ; (6) In equation (6), Indicates the first One point of attention; , and These represent the weight matrices corresponding to the query matrix, key matrix, and value matrix, respectively. Represents the eigenvector; Calculate the attention score matrix for each attention head. The expression is: (7) Attention score matrix conduct softmax After normalization, the attention weight matrix is ​​obtained. The expression is: (8) Value matrix With attention weight matrix Multiply to obtain a weighted sum for each attention head. That is, the output of each attention head, expressed as: (9) Weighted sum of each attention point By merging along the last dimension, we obtain the output of the self-attention layer. The expression is: (10) Subsequently, the output of the self-attention layer Applying a linear transformation, we obtain the final output. The expression is: (11) In equation (11), Represents the weight matrix of the fused linear transformation; In this way, the multi-head self-attention mechanism achieves parallel information extraction by processing multiple attention heads simultaneously, each focusing on different features or aspects of the input sequence. Ultimately, the outputs from these different attention heads are integrated to form the final output of the model.

[0018] Step 2.2.2, the final output obtained The data is fed into a fully connected feedforward neural network (FFN) layer for processing, and outputs feature vectors for higher layers. The expression is: (12) In equation (12), For activation functions; , For weight , For bias.

[0019] The structure of a fully connected feedforward neural network (FFN) consists of a first fully connected layer, a ReLU activation layer, and a second fully connected layer. It aims to utilize features extracted by the multi-head attention mechanism and further explore the nonlinear relationships between these features through the nonlinear FFN, thereby enhancing the expressive power of the features. Step 2.2.3: Convert the obtained higher-level feature vectors into two-dimensional matrices, which will then be used as input to the subsequent U-Net network.

[0020] Example 5 Based on Example 4, in step 3, the loss function of the ViT-U-Net network is: (13) In equation (13), This is the reconstructed image predicted by the ViT-U-Net network. It is the target image. These are the image's height and width, respectively.

[0021] In the U-Net network, spatial resolution is gradually reduced through convolutional and pooling layers during the encoding stage to extract more refined feature information; while in the decoding stage, spatial resolution is gradually restored through deconvolutional layers, and mechanisms such as skip connections are used to enhance the transmission and fusion of feature information, thereby ensuring the accuracy and detail of image reconstruction.

[0022] During the training of the ViT-U-Net network, the changes in the loss function are closely monitored, serving as a key indicator to measure the difference between the network output and the true value, guiding the network's learning and optimization. By continuously adjusting the network parameters, the ViT-U-Net network loss function gradually converges and stabilizes, thereby enabling the output of high-quality computational ghost imaging results.

[0023] This invention combines the architectures of Vision Transformer and U-Net, enabling the model to maintain reconstruction quality while optimizing computational efficiency. The parallel processing capabilities of Vision Transformer make the model more efficient during training and inference. Furthermore, by reducing the number of channels or layers in U-Net, computational speed can be further improved without significantly reducing reconstruction quality, thus meeting the needs of real-time applications.

[0024] This invention not only provides an efficient alternative to traditional correlation imaging and compressed sensing in the field of computational ghost imaging, but also significantly optimizes reconstruction results and effectively suppresses noise. This allows for high-quality image reconstruction even under resource-constrained conditions (such as a limited number of samples), demonstrating significant application potential and practical value. Furthermore, the model's innovation and efficiency lay a solid foundation for its application in extended fields such as image encryption and digital watermarking. For example, in image encryption, this model is expected to achieve more complex and secure encryption algorithms through its powerful image processing capabilities; in the field of digital watermarking, its refined image reconstruction capabilities can be used to embed and extract subtle yet robust watermark information, protecting the copyright of digital content. Therefore, this invention not only has significant advantages in the field of computational ghost imaging but also demonstrates broad application prospects in extended fields.

[0025] Example 6 In this embodiment, handwritten digit images from the MNIST image test database are used as target images. The MNIST database is a widely used image dataset containing various types of handwritten digits. Although it mainly focuses on digit images, its image types are also quite diverse, which can be used to preliminarily test and verify the basic performance of image processing algorithms.

[0026] This embodiment is implemented through MATLAB simulation. The specific process is as follows: Step 1.1: Obtain the Hadamard matrix of the natural sequence using formulas (1) and (2); Assuming the target image The size is 100 pixels, and satisfy 1000 pixels ,in If the integer is an integer, then the size of the Hadamard matrix of the natural sequence is also . ; Step 1.2, using the cake cutting (CC) strategy, first cut the cake obtained in step 1.1... The Hadamard matrix is ​​rearranged row by row and integrated into indivual A matrix of pixels, sorted in ascending order by the number of connected components. Reorder the matrices, then flatten each matrix. Each row vector forms a shape like... Figure 3 The Hadamard matrix of the cake-cutting sequence shown is used as a phase mask embedded in the spatial light modulator (SLM) in computational ghost imaging. When performing full sampling, it is necessary to... Then, for the next sampling, the number of Hadamard matrices for the corresponding cake-cutting sequence will also need to be at least [number missing]. indivual; Step 1.3: The Hadamard matrices of the cake-cutting sequence obtained in Step 1.2 are sequentially embedded into the spatial light modulator (SLM) of the actual computational ghost imaging system and then projected onto the target image in sequence. The target image is obtained by using a bucket detector according to formula (3). Light intensity sequence ; Step 2.1: Based on the MNSIT dataset, after processing through steps 1.1 to 1.3, the required dataset is obtained. The dataset is divided into a training set of 12,000 images, a validation set of 2,000 images, and a test set of 2,000 images. The ViT-U-Net network structure is built, and all parameters in the network are initialized. like Figure 3 As shown in step 2.2, the light intensity sequence in the training set is... The rows and columns are converted into a two-dimensional matrix. The two-dimensional matrix and the Hadamard matrix of the cake-cutting sequence in the training set are divided into m blocks. Each block is flattened into a one-dimensional vector. This one-dimensional vector is converted into a high-dimensional vector token through a linear layer. A position code is added to each high-dimensional vector token. The feature vector is obtained through formula (4). Step 2.3: Input all the feature vectors obtained in Step 2.2 into the Transformer encoder, and process them through a multi-head self-attention mechanism to obtain the final output. Then the final output The vectors are fed into a fully connected feedforward neural network layer for processing to obtain higher-level feature vectors. These higher-level feature vectors are then converted into two-dimensional matrices as input to the subsequent U-net network. The expression for the relationship between all feature vectors obtained by processing through a multi-head self-attention mechanism is shown in Equation (5); The specific process is as described in steps 2.2.1 to 2.2.3 of Example 5; Step 3: Input the two-dimensional matrix obtained in step 2 into the U-Net network to enhance the transmission and fusion of feature information until the loss function of the ViT-U-Net network (the loss function is shown in formula (13)) remains stable, and obtain the result of calculating ghost imaging; The specific process is as follows: like Figure 5 As shown, the U-Net network structure is built and all parameters in the U-Net network are initialized; Step 3.1: The two-dimensional matrix obtained in step 2 is processed through five consecutive convolutional modules, and the output of the last convolutional module is processed by max pooling to reduce the spatial dimension and retain important features. Each convolutional module consists of convolutional layers and activation functions; the kernel size of each convolutional layer is [size missing]. The step size is 1, and the number of feature maps is 64. Step 3.2: Process the output of the previous step with five consecutive convolutional modules, and the output of the last convolutional module is processed by max pooling to further compress the data; Each convolutional module in this step consists of a convolutional layer and an activation function; the kernel size of each convolutional layer is [missing information]. The step size is 1, and the number of feature maps is 128. Step 3.3: Process the output of the previous step with five consecutive convolutional modules, increasing the number of feature maps to 256. Each convolutional module in this step consists of a convolutional layer and an activation function; the kernel size of each convolutional layer is [missing information]. The step size is 1; Step 3.4: To perform feature expansion, deconvolution (transposed convolution) is applied to the output of the previous step for upsampling, and the upsampled output is concatenated with the 128 feature maps obtained in step 3.2. Step 3.5: Process the output of the previous step with five consecutive convolutional modules again, and adjust the number of feature maps to 128. Each convolutional module in this step consists of a convolutional layer and an activation function; the kernel size of each convolutional layer is [missing information]. The step size is 1; Step 3.6: Apply deconvolution (transposed convolution) to the output of the previous step for upsampling, and then concatenate the upsampled output with the 64 feature maps obtained in step 3.1 to enhance the diversity of features; Step 3.7: Process the output from the previous step using five consecutive convolutional modules, adjusting the number of feature maps to 64. Each convolutional module in this step consists of a convolutional layer and an activation function; the kernel size of each convolutional layer is [missing information]. The step size is 1; Step 3.8: Process the output from the previous step using a convolutional module to compress the number of feature maps to 1, thus producing the final output of the network. ; Each convolutional module in this step consists of a convolutional layer and an activation function; the kernel size of this convolutional layer is [size missing]. The step size is 1; After training the training set, training the validation set begins. Specifically: First, a digital image is selected from the validation set as the input image. Using the Hadamard matrix of the cake-cutting sequence generated by computer simulation, the intensity sequence of the bucket detector under this mask is obtained. Then, the intensity sequence is converted into a two-dimensional matrix. Both the two-dimensional matrix and the generated Hadamard matrix of the cake-cutting sequence are then divided into... Each block is flattened into a one-dimensional vector, and this one-dimensional vector is transformed into a high-dimensional vector token through a linear layer. A positional encoding is added to each high-dimensional vector token, and a feature vector is obtained through formula (4). The feature vector is then fed into the ViT-U-Net network, which outputs a prediction result. To evaluate the accuracy of the prediction, we use a loss function to calculate the difference between the output result and the real target image. Based on this difference, we execute the backpropagation algorithm to optimize and adjust the parameters inside the network until the loss function value tends to stabilize, indicating that the network performance has reached a relatively ideal state.

[0027] Next, to verify the network's generalization ability, we input images from the test set into the trained network and compare the network's output with real image data. The purpose of this step is to evaluate the accuracy of the network's predictions when faced with unseen data through practical testing, thereby comprehensively measuring the network's generalization performance.

[0028] To demonstrate the practicality of the method of this invention, the following three evaluation metrics are needed to verify the feasibility of the system: root mean square error (RMSE), peak signal-to-noise ratio (PSNR), and structural similarity (SSIM) analysis of the target image and the reconstructed image. Their corresponding mathematical formulas can be expressed as follows:

[0029] In the formula, This is the reconstructed image predicted by the ViT-U-Net network. It is the target image. These are the image's height and width, respectively.

[0030]

[0031] The parameters in the formula are interpreted the same as those in the formula above.

[0032]

[0033] In the formula, They are 、 The average value, They are 、 variance It is covariance; It is a constant used to maintain stability.

[0034] This embodiment analyzes the RMSE, PSNR, and SSIM of three different computational ghost imaging reconstruction algorithms—the traditional computational ghost imaging method, the TVAL3 algorithm based on compressed sensing, and the method of this invention—on the test set reconstructed images at different sampling rates: 1%, 3%, 5%, 10%, 15%, 20%, 25%, 50%, and 75%. Figure 6 It can be seen that the method of the present invention has wide applicability and superiority in calculating ghost image reconstruction. As shown in Table 1, by comparing the reconstruction results under different sampling rates, it can be clearly seen that in the lower sampling rate range (such as 1% to 25%), the method of the present invention shows a more superior reconstruction quality compared with the traditional method and the TVAL3 algorithm, demonstrating the powerful ability of the algorithm under low sampling conditions.

[0035] Table 1

[0036] At the same time, Figure 6 This paper compares the performance of three different computational ghost imaging reconstruction algorithms at different sampling rates: the traditional computational ghost imaging reconstruction method, the TVAL3 algorithm based on compressed sensing, and the method of this invention. The figure focuses on the reconstruction effect of the digit "6", with sampling rates from low to high: 1%, 3%, 5%, 10%, 15%, 20%, 25%, 50%, and 75%. The figure shows the significant advantage of the method of this invention under lower sampling rate conditions. At extremely low sampling rates (e.g., 1% to 5%), the images reconstructed by the traditional method and the TVAL3 algorithm often suffer from severe blurring, distortion, and noise, making it difficult to accurately identify the outline and details of the original pattern "6". However, at the same sampling rate, the method of this invention exhibits a clearer and more detailed reconstruction effect, not only preserving the basic shape of the pattern but also recovering some detail information. This fully demonstrates its powerful feature extraction and reconstruction capabilities under low sampling rate conditions.

[0037] exist Figure 7This paper visually demonstrates the comparison of reconstruction performance of three different computational ghost imaging reconstruction algorithms for the ten digits "0-9" under a low sampling rate of 5%. By comparing the reconstruction results, it is clear that at a low sampling rate of 5%, traditional second-order correlation algorithms and compressed sensing-based algorithms exhibit image blurring, loss of detail, and some digits being difficult to identify when reconstructing the "0-9" digits. The method of this invention successfully recovers the content of the ten digits "0-9" almost completely, maintaining high clarity and recognizability for each digit, with rich detail and low noise. This illustrates the powerful reconstruction capability of the method of this invention at low sampling rates. By integrating the global feature extraction advantages of Vision Transformer and the fine image restoration capabilities of U-Net, the method of this invention achieves efficient reconstruction of complex image content, maintaining high reconstruction quality even under extremely low sampling conditions.

Claims

1. A computational ghost imaging reconstruction method based on Vision Transformer and U-Net network, characterized in that, Follow these steps: Step 1: The Hadamard matrix of the cake-cutting sequence is used as a phase mask and sequentially embedded into the spatial light modulator and projected onto the target image. Above, a bucket detector is used to collect the total light intensity sequence. ; Step 2, combine the Hadamard matrix of the cake-cutting sequence and the light intensity sequence. The process involves dividing the data into blocks, linearly projecting each block to obtain a high-dimensional vector token, adding a positional code to each high-dimensional vector token, and then inputting it into a Transformer encoder for processing. The output is then converted into a two-dimensional matrix. Step 3: Input the two-dimensional matrix obtained in Step 2 into the U-Net network to enhance the transmission and fusion of feature information until the loss function of the ViT-U-Net network remains stable, and obtain the result of calculating ghost imaging.

2. The computational ghost imaging reconstruction method based on Vision Transformer and U-Net network according to claim 1, characterized in that, The specific process of step 1 is as follows: Step 1.1: Construct the Hadamard matrix of a natural sequence of arbitrary order using 2-order basic blocks. The 2-order basic block is then represented as: （1） have The Hadamard matrix of a natural sequence of orders is represented as: （2） Assuming the target image The size is 100 pixels, and satisfy 1000 pixels ,in If the integer is an integer, then the size of the Hadamard matrix of the natural sequence is also . ; Step 1.2, using the cake-cutting strategy, first cut the cake obtained in Step 1.1... The Hadamard matrix is ​​rearranged row by row and integrated into indivual A matrix of pixels, sorted in ascending order by the number of connected components. Reorder the matrices, then flatten each matrix. The Hadamard matrix of the cake-cutting sequence is formed by a number of row vectors. The Hadamard matrix of the cake-cutting sequence is used as the phase mask embedded in the spatial light modulator in the calculation of ghost imaging. Step 1.3: The Hadamard matrices of the cake-cutting sequence obtained in Step 1.2 are sequentially embedded into the spatial light modulator of the actual computational ghost imaging system and then projected onto the target image in sequence. Above, a bucket detector is used to obtain the target image. Light intensity sequence .

3. The computational ghost imaging reconstruction method based on Vision Transformer and U-Net network according to claim 2, characterized in that, In step 1.3, the light intensity sequence The expression is: （3） In equation (3), Represents the x and y coordinates of the target image. Indicates the first The Hadamard matrix of the cake-cutting sequence after subsampling. The x and y coordinates of the elements of the Hadamard matrix representing the cake-cutting sequence.

4. The computational ghost imaging reconstruction method based on Vision Transformer and U-Net network according to claim 2, characterized in that, The specific process of step 2 is as follows: Step 2.1, sequence the light intensity The rows and columns are converted into a two-dimensional matrix to obtain the converted light intensity sequence. Then, the Hadamard matrix of the cake-cutting sequence is used. The converted light intensity sequence is divided into blocks of fixed size. Each block is flattened into a one-dimensional vector. This one-dimensional vector is then converted into a high-dimensional vector token through a linear layer. A positional code is added to each high-dimensional vector token to obtain a feature vector. Step 2.2: Input all the feature vectors obtained in Step 2.1 into the Transformer encoder, and process them through a multi-head self-attention mechanism to obtain the final output. Then the final output The vectors are fed into a fully connected feedforward neural network layer for processing to obtain higher-level feature vectors. These higher-level feature vectors are then converted into two-dimensional matrices and used as input to the subsequent U-Net network.

5. The computational ghost imaging reconstruction method based on Vision Transformer and U-Net network according to claim 4, characterized in that, In step 2.1, the expression for the feature vector is obtained as follows: （4） In equation (4), For feature vectors; This is the weight matrix; For the size of the block, The dimension of the high-dimensional vector token; For the first The position encoding vector of a high-dimensional vector token; For the first A high-dimensional vector token.

6. The computational ghost imaging reconstruction method based on Vision Transformer and U-Net network according to claim 4, characterized in that, The specific process of step 2.2 is as follows: Step 2.2.1: After performing a linear transformation on the feature vector obtained in Step 2.1 using the weight matrix obtained through learning, a query matrix is ​​generated. Key matrix Value matrix ; (6） In equation (6), Indicates the first One point of attention; , and These represent the weight matrices corresponding to the query matrix, key matrix, and value matrix, respectively. Represent the feature vector; calculate the attention score matrix for each attention head. The expression is: Calculate the attention score matrix for each attention head. The expression is: （7） In equation (7), It is the dimension of the key vector; Attention score matrix conduct softmax After normalization, the attention weight matrix is ​​obtained. The expression is: （8） Value matrix With attention weight matrix Multiply to obtain the weighted sum of each attention head. That is, the output of each attention head, expressed as: （9） Weighted sum of each attention point By merging along the last dimension, we obtain the output of the self-attention layer. The expression is: (10） Subsequently, the output of the self-attention layer Applying a linear transformation, we obtain the final output. The expression is: （11） In equation (11), Represents the weight matrix of the fused linear transformation; Step 2.2.2, the final output obtained The data is fed into a fully connected feedforward neural network layer for processing, and outputs a higher-level feature vector. Step 2.2.3: Convert the obtained higher-level feature vectors into two-dimensional matrices, which will then be used as input to the subsequent U-net network.

7. The computational ghost imaging reconstruction method based on Vision Transformer and U-Net network according to claim 6, characterized in that, In step 2.2.2, the expression processed by the fully connected feedforward neural network layer is: （12） In equation (12), For activation functions; , For weight , For bias.

8. The computational ghost imaging reconstruction method based on Vision Transformer and U-Net network according to claim 6, characterized in that, In step 3, the loss function of the ViT-U-Net network is: （13） In equation (13), This is the reconstructed image predicted by the ViT-U-Net network. It is the target image. These are the image's height and width, respectively.

Images