A hybrid network-based ghost imaging reconstruction and classification method and device

Abstract

The application discloses a kind of ghost imaging reconstruction and classification method and equipment based on hybrid network, belong to the field of computational imaging technology, will be based on Hadamard matrix generation light speckle sequence formed by fixed light speckle sequence;Then, based on this light speckle sequence sampling expanded MNIST data set, generate light intensity sequence, light speckle sequence data;Then, construct and train GRU-CNN network model;Finally, using the model trained to calculate correlation imaging reconstruction and target identification classification.The conventional ghost imaging technique is often faced with the problem of poor image quality under low sampling conditions.In order to solve this problem, the application proposes a GRU-CNN network (GCGI) model combined with deep learning, which can efficiently reconstruct high-quality images and accurately identify and classify targets under very low sampling rates by fusing light intensity sequence and light speckle information, and has strong practical value.

Description

Technical Field

[0001] This invention belongs to the field of computational imaging technology, specifically relating to a method and device for ghost imaging reconstruction and classification based on hybrid networks. Background Technology

[0002] Ghost imaging, also known as correlation imaging, is an unconventional imaging technique that indirectly reconstructs images by utilizing the higher-order correlation properties of light fields. The core of this imaging method lies in the fact that it does not directly acquire an image from the light field of the object being imaged. Instead, it reconstructs the image by measuring the correlation between the light field after it interacts with the object and another light field that does not interact with the object. In ghost imaging, the light emitted by the light source is split into two paths: one path passes through the object and is collected by a bucket detector without spatial resolution, called the signal light path or object arm; the other path does not contact the object and is directly acquired by a detector with spatial resolution (such as a CCD camera), called the reference light path or reference arm. The image is reconstructed by calculating the intensity correlation between these two light paths. Ghost imaging can utilize the second-order intensity correlation properties of the light field to obtain image information of the object, giving it a significant advantage in noise resistance, especially under the influence of perturbation and attenuation of the light signal by scattering media. Ghost imaging also has non-locality, meaning that the light does not need to directly contact the target object to obtain an image of it.

[0003] Two key factors affecting image quality in ghost imaging are the measured illumination speckle pattern and the image reconstruction association algorithm. Current illumination speckle methods include orthogonalized sinusoidal speckle and orthogonal Hada matrix illumination speckle methods. Ghost image reconstruction algorithms include differential ghost imaging and normalized ghost imaging. These proposed illumination speckle and reconstruction algorithms have significantly improved the reconstruction quality of ghost imaging. However, under low sampling conditions, the image quality reconstructed by these traditional ghost imaging algorithms remains poor. Summary of the Invention

[0004] The purpose of this invention is to provide a method and device for ghost imaging reconstruction and classification based on hybrid networks, which can reconstruct high-quality images and complete target recognition and classification at extremely low sampling rates.

[0005] To achieve the above objectives, the present invention adopts the following technical solution: A method for ghost imaging reconstruction and classification based on hybrid networks includes the following steps: An illumination speckle sequence is generated based on the Hadamard matrix, and the generated illumination speckle is used to sample the object to obtain an intensity sequence. The illumination speckle sequence and light intensity sequence are input into the trained image reconstruction and classification model to obtain the corresponding images and image categories; The image reconstruction and classification model adopts a GRU-CNN hybrid network. The training and testing datasets for the image reconstruction and classification model include illumination speckle sequence, light intensity sequence, fusion data of illumination speckle sequence and light intensity sequence up-dimensional splicing, original image and original image label. The GRU-CNN hybrid network includes a light intensity sequence feature extraction module, a light intensity sequence and speckle fusion data feature extraction module, and an image classification and image reconstruction module, wherein: Light intensity sequence feature extraction module: used to extract temporal features from the input light intensity sequence; Fusion data feature extraction module: used to receive and fuse the measured light speckle sequence and the features extracted by the light intensity sequence feature extraction module, and perform multi-scale feature extraction and output; Image classification and image reconstruction module: used to reconstruct and classify images based on the multi-scale features output by the light intensity sequence and speckle fusion data feature extraction module.

[0006] Furthermore, the process of generating the light speckle sequence is as follows: Recursively construct a one-dimensional node tree and recursively generate the left and right subtrees; Perform a matrix dot product operation between the row vector of the left leaf node and the column vector of the right leaf node to generate the original Hadamard matrix, and then convert it to obtain the binary form of the Hadamard transformation matrix. The Hadamard transformation matrix is ​​traversed in a zigzag pattern to obtain the illumination speckle sequence.

[0007] Furthermore, the formula for calculating light intensity is:

[0008] in, This represents the total light intensity value. Let be the transmittance function of the object under test. The light field distribution on the surface of the target object is shown, and the subscript i indicates the i-th sampling.

[0009] Furthermore, the construction of the training and testing datasets for the image reconstruction and classification model includes: From the illumination speckle sequence, the first 3M illumination speckles are selected in sequence and loaded onto a digital micromirror device in sequence to control the light field. The light intensity of the illuminated image is collected by a barrel detector to obtain the light intensity sequence. The obtained light intensity sequence is multiplied by the corresponding M illumination speckle sequences to obtain M-channel fused data; The dataset includes: the last 2M illumination speckles, intensity sequences, M-channel fused data, original images, and original image labels.

[0010] Furthermore, the light intensity sequence feature extraction module includes two GRU-LayerNorm modules, which are sequentially connected via nn.ModuleList(). Each GRU-LayerNorm module includes: A GRU layer: used for time series modeling of the input data; A LayerNorm layer: used to normalize the output of the GRU layer, wherein the output dimension of the LayerNorm layer is the same as the output dimension of the GRU layer.

[0011] Furthermore, the light intensity sequence and speckle fusion data feature extraction module includes: Multi-scale gated fusion module: Through a series of convolutional layers and gating mechanisms, combined with features extracted by multi-scale deep convolutional modules, the final fused feature output is generated; BatchNorm2d layer: Normalizes the input for each batch in a convolutional neural network.

[0012] Furthermore, the multi-scale feature extraction process includes: The input feature data undergoes a first-stage 1×1 convolution process to map the number of channels in the input data to a predetermined dimension. By using a multi-scale depth convolution module, depthwise separable convolution operations with different dilation factors are employed to extract multi-scale features from the input data and generate feature maps at multiple scales. The feature maps at each scale are weighted and fused. The weighting coefficients are based on the channel allocation ratio, and the fused feature map is obtained by weighted summation of each channel. The fused feature map is input into the gated layer, which uses a 1×1 convolution to generate a gated signal. The gated signal is processed using the SiLU activation function to obtain the gate weight for each channel; The gated signal is multiplied channel by channel with the fused feature map; The second stage involves 1×1 convolution to adjust the number of channels, and the result is used as the final output.

[0013] Furthermore, the image classification and image reconstruction module includes an image classification submodule and an image reconstruction submodule. The image classification submodule includes: Several convolutional-batch normalization blocks and pooling layers are used to progressively extract image features and reduce spatial dimensionality; An adaptive average pooling layer is used to transform the input two-dimensional feature map into an output of a specified size using an adaptive average pooling method. Fully connected layers are used to flatten the feature maps after convolution and pooling and output the classification results; The image reconstruction submodule includes: Several convolutional-batch normalization blocks: used to progressively extract features and reduce the number of channels; Output convolutional layer: used to generate the final image output; The Sigmoid activation function is used to generate normalized image output.

[0014] In a second aspect, a terminal device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the above-described method.

[0015] Thirdly, a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the above-described method.

[0016] Compared with the prior art, the present invention has the following beneficial technical effects: This invention presents a ghost imaging reconstruction and classification method based on a hybrid network, specifically a GRU-CNN hybrid network-based method for ghost imaging reconstruction and target recognition and classification, effectively improving the reconstruction quality and accuracy of target recognition and classification. This invention provides a novel approach to ghost imaging and target recognition and classification by utilizing one-dimensional light intensity sequences and two-dimensional spatial illumination speckle fusion for simultaneous object imaging and classification. Experimental results show that, under the same sampling rate, this method achieves higher PSNR, SSIM, and classification accuracy for reconstructed target objects; at a sampling rate of 1.56%, the target object recognition and classification accuracy reaches 95.27%. This invention offers advantages such as high efficiency, high accuracy, and versatility, overcoming the hardware limitations that hinder target object reconstruction and classification in the ghost imaging field. Attached Figure Description

[0017] Figure 1 This is a flowchart of a ghost imaging reconstruction and classification method based on a hybrid network in an embodiment of the present invention; Figure 2 Here is a flowchart of the GRU-CNN hybrid network; Figure 3 This is a schematic diagram of the light intensity sequence feature extraction module. Figure 4 This is a schematic diagram of the multi-scale gating fusion module structure; Figure 5 This is a schematic diagram of the structure of a multi-scale depthwise convolutional module; Figure 6This is a schematic diagram of the convolutional batch normalized activation module structure. Figure 7 This is a schematic diagram of the image classification module structure; Figure 8 This is a schematic diagram of the image reconstruction module structure; Figure 9 The graph shows the change in classification accuracy of the object classification and recognition on the MNIST test set with the number of training batches when the sampling rate is 1.56%, the learning rate is 0.0005, the epoch is 5000, and the early stopping method is used. Figure 10 The image shows a comparison of the results of reconstructing the digit 8 at sampling rates of 0.78%, 1.56%, 3.12%, 6.25%, and 12.50% between the present invention and the traditional computational ghost imaging (CGI) algorithm and the physically guided single-layer neural network embedded with differential ghost imaging (DGI-SLNN) algorithm. Figure 11 This is a comparison chart showing the results of reconstructing the digit 8 at a sampling rate of 1.56% using the present invention, the traditional computational ghost imaging (CGI) algorithm, and the physically guided single-layer neural network with embedded differential ghost imaging (DGI-SLNN) algorithm, with additional sampling rates of 30dB, 20dB, 10dB, 5dB, and 1dB. Detailed Implementation

[0018] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0019] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0020] It should be noted that the terms "first" and "second" in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0021] The present invention will now be described in further detail with reference to the accompanying drawings: like Figure 1 As shown, a ghost imaging reconstruction and classification method based on hybrid networks includes the following steps: Step 1: Generate an illumination speckle sequence based on the Hadamard matrix, and use the generated illumination speckle to sample the object to obtain an illumination intensity sequence; Specifically, it includes: Step 1.1: Enhance the 70,000 data points in the MNIST dataset by rotating, mirroring, and other operations, and uniformly expand them into N*N size images; Step 1.2: Construct an N*N illumination speckle sequence, including the following steps: Step 1.2.1: Generate the column vectors of the Hadamard matrix by sorting; First, a sequence of column vectors is constructed. In this step, the construction process starts with [1] as the root node and uses a recursive method to generate the value of each column. With [1] as the root node, left and right nodes are generated: the value of the right node is formed by concatenating the values ​​of the root node, that is, [1] concatenated with [1] to get [1, 1]. The value of the left node is formed by concatenating the value of the root node with -1 * [1], that is, [1] concatenated with [-1] to get [1, -1]. As the recursion continues, other levels of the tree are constructed. For example, the root node [1, 1] will be further recursively split into: concatenating the right node [1, 1] to get [1, 1, 1, 1]; concatenating the left node -1 * [1, 1] to get [1, 1, -1, -1]; The process continues recursively until the length of the nodes reaches N, thus completing the construction of the column vectors. After all column vectors are generated, they are sorted from right to left as needed.

[0022] Step 1.2.2: Generation of Hadamard matrix row vector sorting The generation of row vectors is similar to that of column vectors, the only difference being that this time the row vectors are constructed from left to right. The root node is [1], and the left and right nodes are generated: the left node is [1, 1] by concatenating [1] with [1]; the right node is [1, -1] by concatenating [1] with -1 * [1]. Recursively generate deeper levels: the left node is [1, 1], which is further recursively split into: the left node is concatenated with [1, 1] to get [1, 1, 1, 1]; the right node is concatenated with -1 * [1, 1] to get [1, 1, -1, -1]; Similar to column vectors, the generation process continues recursively until the length reaches N, until leaf nodes are formed. After all row vectors are generated, they are sorted from left to right.

[0023] Step 1.2.3: Generation of the N*N illumination speckle matrix After steps 1.2.1 and 1.2.2, the column vectors and row vectors of the Hadamard matrix have been generated. Next, the illumination speckle matrix is ​​generated by multiplying each element of the sorted column vector with each element of the sorted row vector. The product of each element is equivalent to performing a dot product (i.e., inner product) between the column and row vectors of the Hadamard matrix. The result of the dot product is usually 1 or -1, representing each element of the illumination speckle matrix; For example, assuming the column vector is [1, -1, 1] and the row vector is [1, 1, -1], their dot product is: 1 × 1 + ( 1)×1+1×( 1)=1 1 1= 11×1+( 1)×1+1×( 1)=1 1 1= The result obtained is -1; Replacing all -1 values ​​in the matrix with 0 generates a matrix containing only 0s and 1s, which forms the final speckle matrix.

[0024] Step 1.2.4: Zigzag scanning and sorting In this step, the matrix is ​​sorted using a zigzag scan, starting from the top left corner and scanning diagonally throughout the matrix: Initially, the scan begins with the first row of the matrix, followed by the second and third rows, and so on. During the scan, the direction of scan is determined by whether the sum of the row and column of the current element is even: if even, the scan direction is the upper right corner; if odd, the scan direction is the lower left corner. This scanning method traverses all elements in the matrix, resulting in a zigzag arrangement.

[0025] Step 1.3: Image data acquisition from 70,000 data points in the MNIST dataset, including the following steps: Step 1.3.1: Calculate the light intensity collected by the bucket detector for each measurement in the correlated imaging. The formula for calculating the light intensity is as follows:

[0026] Where T(x, y) is the transmittance function of the object to be measured. The barrel detector measures the total light intensity as the light field distribution on the surface of the target object. The subscript i indicates the i-th sample; Step 1.3.2: Set the sampling rate β. If the image size is N*N, then the total number of samples T = N*N. If M is the number of samples, then the sampling rate β = M / T. Step 1.3.3: Determine the number of samplings M based on the sampling rate β in step 1.3.2, and take 3*M light speckles from the light speckle sequence generated in step 2.3 from front to back; Step 1.3.4: The first M illumination speckles are sequentially loaded onto the digital micromirror device to control the light field. A bucket detector is used to collect the light intensity of the illuminated image; the total collected light intensity is denoted as... (i represents the i-th sampling), the image is illuminated by M measurements, and the collected light intensity sequence is: I = [ , , ...., ] Step 1.3.5: Multiply the light intensity sequence obtained in step 1.3.4 with the corresponding M light speckle sequences to obtain M-channel speckle light intensity fusion data.

[0027] The dataset consists of the last 2*M speckles obtained in step 1.3.3, the light intensity sequence obtained in step 1.3.4, the M-channel speckle light intensity fusion data obtained in step 1.3.5, and the corresponding images and image labels in the dataset.

[0028] Step 2: Input the illumination speckle sequence and light intensity sequence into the trained image reconstruction and classification model to obtain the corresponding image and image category; In detail, the image reconstruction and classification model adopts a GRU-CNN hybrid network. The GRU-CNN hybrid network includes a light intensity sequence feature extraction module, a light intensity sequence and speckle fusion data feature extraction module, and image classification and image reconstruction modules, such as... Figure 2 As shown, where: Light intensity sequence feature extraction module: used to extract temporal features from the input light intensity sequence; Fusion data feature extraction module: used to receive and fuse the measured light speckle sequence and the features extracted by the light intensity sequence feature extraction module, and perform multi-scale feature extraction and output; Image classification and image reconstruction module: used to reconstruct and classify images based on the multi-scale features output by the light intensity sequence and speckle fusion data feature extraction module.

[0029] Specifically, the input light intensity sequence first passes through the light intensity sequence feature extraction module, the structure of which is as follows: Figure 3 The diagram shows a two-layer time-series processing block, each consisting of two main components: a GRU layer for processing time-series data and capturing the dynamic relationships between time steps, and a LayerNorm layer for normalizing the features at each time step to stabilize the training process. The specific processing steps are as follows: Step 2.1.1: Input data: The shape of the light intensity sequence (temporal_data) obtained in step 1.3.4 is (batch_size, seq_length, input_feature_dim).

[0030] Step 2.1.2: After passing through the first timing processing layer: The first timing processing block contains two layers: a GRU layer and a LayerNorm layer; The input feature dimension of the GRU layer is input_feature_dim, the hidden state dimension is also input_feature_dim, and the output dimension of the GRU layer is input_feature_dim, which is the state dimension of the GRU hidden layer. The GRU layer updates the hidden state step by step based on the input temporal_data and the previous hidden state. At each time step, GRU updates the current hidden state based on the current input and the hidden state of the previous time step. The output temporal_data after GRU processing has the shape of (batch_size, seq_length, input_feature_dim), and this output contains the hidden state information of each time step; The LayerNorm layer standardizes the input temporal data, making the mean of the input_feature_dim feature at each time step 0 and the variance 1 in each batch. This avoids the impact of numerical differences between different features on the training process, thereby improving the stability of training. After LayerNorm, temporal_data still has the shape (batch_size, seq_length, input_feature_dim), but its values ​​are normalized, making the feature distribution at each time step more consistent.

[0031] Step 2.1.3: After the second timing processing block: The second timing processing block has the same structure as the first timing processing block, but its input is the output of the first timing processing block, and the output dimension has changed. The second timing processing block also contains two layers: a GRU layer and a LayerNorm layer. The input feature dimension of the GRU layer is input_feature_dim, which is the output dimension from the first temporal block. The output dimension, which is also the hidden state dimension, is twice the input_feature_dim. The shape of the output temporal_data after processing by the second GRU layer is (batch_size, seq_length, 2*input_feature_dim); After the second timing processing block, the final output temporal_data has the shape of (batch_size, seq_length, 2*input_feature_dim).

[0032] The above steps complete the feature extraction of the light intensity sequence.

[0033] The next step is to perform element-wise multiplication and fusion of the extracted light intensity sequence features with the last 2*M speckles obtained in step 3.3 to obtain 2*M channel speckle light intensity fusion data. This data is then spliced ​​with the M channel speckle light intensity fusion data obtained in step 3.5 in the channel dimension. The spliced ​​data (spatial_data) is then sent to the light intensity sequence and speckle fusion data feature extraction module.

[0034] The feature extraction module for light intensity sequence and speckle fusion data includes a multi-scale gated fusion module and batch normalization (BatchNorm).

[0035] The multi-scale gating fusion module comprises numerous modules, and its structure is as follows: Figure 4 As shown, the specific process is as follows: Step 2.2.1: The shape of the input data spatial_data is (batch_size, input_channels, H, W); Step 2.2.2: The spatial_data first passes through a convolutional layer with a kernel size of 1x1. The number of input channels is input_channels, and the number of output channels is the same as the number of input channels. This layer does not change the spatial dimension (H, W), and the shape of the output feature is (batch_size, input_channels, H, W). Step 2.2.3: The data x after convolutional transformation will pass through a global average pooling layer. The purpose of this step is to compress the spatial dimensions (H and W) of each channel to 1. Therefore, each channel retains only one global feature value. After this step, the input shape of the data changes from (batch_size, input_channels, H, W) to (batch_size, input_channels, 1, 1). Step 2.2.4: The pooled feature map is then compared with the original input feature spatial_data (the original input spatial_data minus the pooled result), and scaled using feature_scaler. feature_scaler is a learnable transformation module used to smooth and adjust feature differences, and the output shape remains (batch_size, input_channels, H, W). Step 2.2.5: Add the data from Step 2.2.2 to the adjusted data and perform non-linear activation. Here, the SiLU activation function (Sigmoid-Linear Unit) is used. It is a smooth and efficient activation function that is often used to enhance the performance of the network while keeping the output shape unchanged. Step 2.2.6: The spatial_data obtained in step 2.2.5 is passed into the convolutional layer. The convolutional kernel size is 1x1, the number of input channels is input_channels, and the number of output channels is the same as the number of input channels. This layer does not change the spatial dimension (H, W), and the shape of the output feature is (batch_size, input_channels, H, W). Step 2.2.7: Input the data obtained in step 2.2.5 into the multi-scale deep convolution module. This module extracts contextual information at different scales through various dilation convolution operations, thereby better capturing multi-scale features. The data input shape and output shape are both (batch_size, input_channels, H, W). Through multi-scale convolution, the shape of the feature map remains consistent. Step 2.2.8: The data obtained in Step 2.2.6 and Step 2.2.7 are respectively fed into the SiLU activation function to obtain two activated outputs. Then, the two activated feature maps are multiplied element-wise to represent weighted fusion. Step 2.2.9: Next, the fused feature maps will undergo further linear transformation through convolutional layers. Similar to the convolution in Step 2.2.2, the goal of this step is to fine-tune the fused features, while keeping the input and output shapes unchanged (batch_size, input_channels, H, W). Step 2.2.10: The data from Step 2.2.9 is passed through the Dropout layer and then added to the initial input data to form a residual connection. Residual connections help the network better propagate gradients, avoid the gradient vanishing problem, and keep the input and output shapes unchanged. The final output feature map shape is still (batch_size, input_channels, H, W).

[0036] The main steps of this module are: convolution, pooling, difference adjustment, activation, convolution, multi-scale extraction, weighted fusion, convolution, and residual connection. Output feature shape: The final output shape is consistent with the input feature shape, i.e., (batch_size, input_channels, H, W).

[0037] Among the multi-scale gated fusion modules, a particularly important module is the multi-scale deep convolution module, whose structure is as follows: Figure 5 As shown, the internal operation is as follows: Step 2.3.1: The shape of the input data spatial_data is (batch_size, input_channels, H, W); Define dilations=[1, 2, 3] and channel_split_ratios=[1, 3, 4] for subsequent operations.

[0038] Step 2.3.2: Next, calculate split_factors, which represent the proportion of each ratio in channel_split_ratios relative to the total. Then, calculate the dimension split_channel_dims of each channel block based on these ratios.

[0039] Step 2.3.3: The input data spatial_data passes through the first depthwise convolutional layer to obtain depthwise_output_1. The input and output channels of the first convolutional layer are both input_channels. The kernel size is 5x5, the dilation rate is dilations[0], and the convolution operation is a depthwise convolution based on all input channels.

[0040] Step 2.3.4: depthwise_output_1 is divided into three parts based on split_channel_dims calculated in step 2.3.2, and the second and third channels are processed by the second and third depthwise convolutional layers respectively: The second part of the channel data is processed through the second convolutional layer. The input and output channels of the second convolutional layer are both split_channel_dims[1] (calculated in step 2.3.2). The kernel size is 5x5 and the dilation rate is dilations[1], resulting in depthwise_output_2.

[0041] The third part of the channel data is processed through the third convolutional layer. The input and output channels of the third convolutional layer are both split_channel_dims[2] (calculated in step 2.3.2). The kernel size is 7x7 and the dilation rate is dilations[2], resulting in depthwise_output_3.

[0042] Step 2.3.5: The first part of the channels in depthwise_output_1 (the split_channel_dims[0] channel data obtained through slicing operation) is concatenated with depthwise_output_2 and depthwise_output_3 to obtain a new feature map output_features with the shape (batch_size, input_channels, H, W), where each part of the channels has been integrated according to the number of input channels and the allocation ratio; Step 2.3.6: The output_features will be processed through a 1x1 pointwise convolutional layer to uniformly adjust the number of output channels. The shape of the output is still (batch_size, input_channels, H, W).

[0043] The main steps of this module are: the input feature map is processed by a deep convolutional layer, divided into different parts and convolved with different dilation rates, and then the results of each part are concatenated. The number of output channels is unified by pointwise convolution, and the shape of the output is consistent with the shape of the input, i.e. (batch_size, input_channels, H, W).

[0044] Following the multi-scale gating fusion module is the BatchNorm module. BatchNorm is a very effective technique that reduces instability during training by standardizing the input of each layer, accelerating the convergence speed of the network, and improving training stability.

[0045] The multi-scale gating fusion module and the BatchNorm module are stacked to form a sequence container. The input data enters the classification module after passing through a two-layer sequence container consisting of a two-layer multi-scale gating fusion module and a BatchNorm module.

[0046] The classification module includes multiple operations, such as Figure 7 As shown, the specific process is as follows: Step 2.4.1: The input data (input_data) after passing through the two-layer light intensity sequence and speckle fusion data feature extraction module has the shape (batch_size, input_channels, H, W); Step 2.4.2: Using the function compute_channel_sizes to calculate the channel size, calculate the channel size list channel_sizes[input_channels, input_channels / 2, input_channels / 4] based on the number of input channels. These parameters control the number of output channels for each convolutional layer. Step 2.4.3: Input data input_data is passed through the first convolution batch normalization activation module, with input channel channel_sizes[0] and output channel channel_sizes[1] passed in, and output conv_output_1 is obtained; Convolutional normalization activation module, such as Figure 6 As shown, the specific modules included are as follows: The first convolutional layer has a kernel size of 3x3, a stride of 1, padding of 1, input channels of in_channels, and output channels of out_channels; The second layer, BatchNorm2d, takes out_channels as its input parameter. The third layer is the Leaky ReLU activation function layer (Leaky Rectified Linear Unit); Step 2.4.4: Perform pooling operation on conv_output_1 and downsample using MaxPool2d: Spatial downsampling of the feature map is performed by max pooling (MaxPool2d) to reduce the spatial size (H and W) of the feature map and output pool_output_1; Step 2.4.5: The data pool_output_1 is passed through the second convolution normalization activation module, with input channel channel_sizes[1], output channel channel_sizes[2], and normalization group number group_norm_groups[1], to obtain the output conv_output_2; Step 2.4.6: After two convolutional normalization activation module operations, the feature map is typically three-dimensional (batch_size, channels, height, width). To pass it to the fully connected layer (i.e., the final output in the classification task), the feature map needs to be flattened into a one-dimensional vector; The feature maps are input into an adaptive average pooling layer, which adjusts the size of each input two-dimensional feature map (i.e., height x width) to 1x1 through an average pooling operation.

[0047] Step 2.4.7: The flattened feature vector will pass through a fully connected layer and finally be mapped to the class space. The input is the flattened feature vector (batch_size, flattened_size), and the output is a tensor of shape (batch_size, num_classes). Each row represents the class probability distribution of an image sample. Specifically, for each sample, the elements of the output vector represent the probability that the sample belongs to different classes.

[0048] After passing through two layers of multi-scale gated fusion modules and BatchNorm modules, the data, in addition to entering the classification module, must pass through a sequence container composed of three layers of multi-scale gated fusion modules and BatchNorm modules before being sent to the image reconstruction module.

[0049] The image reconstruction module includes several operations, such as Figure 8As shown, the specific process is as follows: Step 2.5.1: Input the data (input_data) after passing through the five-layer light intensity sequence and speckle fusion data feature extraction module. The shape of the input data is (batch_size, input_channels, H, W). Step 2.5.2: Input the data into the first convolutional normalization activation module, pass in the parameters channel_sizes[0], output channel_sizes[0], and group_norm_groups[0], and get the output conv_output_1; Step 2.5.3: Input conv_output_1 into the second convolution normalization activation module, and pass in the parameters channel_sizes[0], channel_sizes[1], and group_norm_groups[1] to obtain the output conv_output_2.

[0050] Step 2.5.4: Input conv_output_2 into the third convolution normalization activation module, and pass in the parameters channel_sizes[1], channel_sizes[2], and group_norm_groups[2] to obtain the output conv_output_3.

[0051] Step 2.5.5: After three convolutional normalization activation module operations, the number of channels and spatial size of the feature map have changed. The final step is to map the processed features to the final image space; The last convolution operation, with a 3x3 kernel, a stride of 1, and padding of 1, converts the number of channels from channel_sizes[2] to output_channels.

[0052] Step 2.5.6: Apply the Sigmoid activation function to the output of the last convolutional layer to constrain the pixel values ​​of the output image to the range [0, 1], thereby generating the final image. This ensures that the pixel values ​​of the generated image fall within a reasonable range.

[0053] The output image has a shape of (batch_size, output_channels, H, W), and the value of each pixel in the image is restricted to the range [0, 1].

[0054] Training a GRU-CNN hybrid network model involves the following steps: Step 3.1: Divide the image dataset obtained in Step 1 into a training set and a test set in a 6:1 ratio; Step 3.2: The loss function of the GRU-CNN hybrid network adopts a dynamic weighted loss calculation method for multi-task learning, which includes the following steps: Step 3.2.1: Initialize the weight parameters, which represent the initial importance of each task. The weight parameters are trainable tensors of size num_tasks and are initialized on the device to ensure that the weights of each task are positive and can be updated during training. Step 3.2.2: Based on the outputs and objectives of multiple tasks, calculate the loss for each task, where: For classification tasks, the cross-entropy loss function is used to calculate the loss. For regression tasks, the mean squared error loss function (MSELoss) is used to calculate the loss; Step 3.2.3: Normalize the task weight parameters initialized in Step A using the softmax function to ensure that the weights of each task are positive and the sum is 1; Step 3.2.4: Weighted total loss. Multiply the loss of each task by the corresponding normalized weight and sum them to obtain the final weighted total loss.

[0055] Specifically, the formula for calculating the loss function is as follows:

[0056] Where task_weight1 and task_weight2 are the weights of the two loss functions, n is the size of the image, and i is the pixel position in the image. Let i be the position of pixel i in the original image. Let be the position of pixel i in the training result; c is the total number of categories, and k is the number of samples calculated. For sample category labels, Predict labels for categories.

[0057] Step 3.3: The AdamW optimizer is used for training, and the ReduceLROnPlateau learning rate scheduling method is used.

[0058] AdamW is a variant of the Adam optimizer with weight decay. The hyperparameters of the AdamW optimizer are determined by grid search, with a learning rate of 0.0005 and a weight decay of 0.0001.

[0059] The learning rate scheduling method used is ReduceLROnPlateau, which is a learning rate scheduling method based on validation loss. When the validation loss no longer decreases within a certain period of time, it will reduce the learning rate. The mode of ReduceLROnPlateau is set to min, which monitors the validation loss. Each time, the learning rate is reduced by 10% of the original value. If the validation loss does not improve, it will wait for a maximum of 5 epochs. Step 3.4: The quality assessment of the reconstructed image by the model uses structural similarity, mean squared error, and peak signal-to-noise ratio.

[0060] Structural similarity (SSIM) refers to the degree of similarity between the generated image and the real image, and is calculated as follows:

[0061] and Let x and y represent the average values, respectively. and y and x represent the standard deviations of x and y, respectively. This represents the covariance of x and y. , , These are constants to avoid system errors caused by a denominator of 0.

[0062] Mean Squared Error (MSE) is a commonly used error metric in regression analysis, used to assess the difference between model predictions and actual values. Its calculation formula is as follows:

[0063] The original image is in position Pixel value at that location, It is to restore the image in position The pixel value at the location is m, where m is the width of the image (number of pixel columns) and n is the height of the image (number of pixel rows).

[0064] Peak signal-to-noise ratio (PSNR) is the ratio between the maximum signal strength and the noise intensity. A higher PSNR value indicates less noise interference to the image. The mathematical expression is as follows:

[0065] Where max is the maximum possible pixel value in the image, and MSE is the mean square error between the original image and the reconstructed image.

[0066] Step 3.5: The noise added during the model reconstruction process is SNR noise, and the specific process is as follows: Given a normalized image I={I(x,y)} with pixel values ​​ranging from [0,1], the signal power (P_((signal))) can be represented by the mean square value of the image pixels:

[0067] Where: N is the total number of pixels in the image, I(x,y) is the pixel value at position (x,y), and E[ ] represents the mean value for all pixel values; Signal-to-noise ratio (SNR) is usually expressed in decibels (dB), which is defined as the ratio of signal power to noise power. Given an SNR in decibels, it needs to be converted into a linear ratio to calculate the noise power.

[0068] Noise power This can be obtained from the relationship between SNR and signal power:

[0069] The generated noise then has a mean of zero and a variance of . For a Gaussian distribution, the standard deviation of the noise is the square root of the noise power.

[0070] Therefore, the generated noise field At each pixel, samples are independently taken from the following Gaussian distribution:

[0071] That is, noise At each pixel location, the mean is 0 and the variance is... It follows a normal distribution.

[0072] Adding noise to the image results in a noisy image.

[0073]

[0074] in These are the pixel values ​​of the original image. At pixel position Noise added at the location.

[0075] Since image pixel values ​​should be limited to the range [0,1], the final noisy image undergoes a cropping operation to ensure that each pixel value remains within this range:

[0076] here The function will valuea Limits to between min and max; Step 3.6: Save the GRU-CNN hybrid network model. The GRU-CNN hybrid network model is saved as GCGI.pth. The saved model is used to load models for real-world scenarios and reconstruct images.

[0077] To demonstrate the versatility of the ghost imaging reconstruction algorithm based on the GRU-CNN hybrid network of this invention, comparative examples are provided: Example 1: Using existing conventional computational ghost imaging reconstruction methods, the image digit "8" was reconstructed at sampling rates of 0.78%, 1.56%, 3.12%, 6.25%, and 12.50%, respectively. Example 2: Using existing technology to physically guide a single-layer neural network and embed the Differential Ghost Imaging (DGI-SLNN) algorithm, the image digit "8" was reconstructed at sampling rates of 0.78%, 1.56%, 3.12%, 6.25%, and 12.50%, respectively. Example 3: Using the ghost imaging reconstruction method based on GRU-CNN hybrid network of the present invention, the image digit "8" was reconstructed at sampling rates of 0.78%, 1.56%, 3.12%, 6.25%, and 12.50%, respectively; The average numerical results of the reconstructed images on the test set at different sampling rates are compared with those of the traditional computational ghost imaging (CGI) algorithm and the physically guided single-layer neural network with embedded differential ghost imaging (DGI-SLNN) algorithm. Table 1 shows that as the sampling rate increases, all three indicators of GCGI improve. Compared with CGI and DGI-SLN, at low sampling rates (below 12.5%), the ghost imaging reconstruction method based on the GRU-CNN hybrid network of this invention has higher indicators than both the computational ghost imaging (CGI) algorithm and the physically guided single-layer neural network with embedded differential ghost imaging (DGI-SLNN) algorithm. This indicates that at lower sampling rates, GCGI has a significant advantage over the other two methods.

[0078] Table 1 Comparison of Average Values

[0079] The object classification accuracy of this invention on the MNIST test set with a sampling rate of 1.56%, a learning rate of 0.0005, an epoch of 5000, and using the early stopping method, is shown in the following curve: Figure 9 As shown, the recognition accuracy steadily increases with the increase of training batches.

[0080] The results of the image classification and recognition accuracy of the test set under different sampling rates are shown in Table 2 below. As can be seen from Table 2, the image classification and recognition accuracy of GCGI improves with the increase of sampling rate; at a lower sampling rate, the image classification and recognition accuracy of MMTS is still relatively high, and GCGI has a great advantage in object recognition and classification under low sampling rate.

[0081] Table 2. Image classification and recognition accuracy results for the test set.

[0082] Figure 10 The results are shown in the comparison chart of Examples 1-3. It can be seen from the chart that even at a low sampling rate, the ghost imaging reconstruction method based on the GRU-CNN hybrid network of the present invention in Example 3 has a significant advantage over the other two methods.

[0083] Figure 11 This image compares the results of reconstructing the digit 8 using the present invention, the traditional computational ghost imaging (CGI) algorithm, and the physically guided single-layer neural network embedded with differential ghost imaging (DGI-SLNN) algorithm at a sampling rate of 1.56% with 30dB, 20dB, 10dB, 5dB, and 1dB added. The noisy image is the original image after adding different levels of noise, and the recovered image is the image reconstructed by each algorithm. Figure 11 It can be seen that the reconstruction effect of the present invention after adding noise is better than the two existing methods.

[0084] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can be implemented in one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROMs) containing computer-usable program code. The form of a computer program product implemented on ROM (optical memory, etc.).

[0085] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0086] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0087] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0088] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit its scope of protection. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that after reading the present invention, they can still make various changes, modifications or substitutions to the specific implementation of the invention, but these changes, modifications or substitutions are all within the scope of protection of the pending claims of the invention.

Claims

1. A ghost imaging reconstruction and classification method based on hybrid networks, characterized in that, Includes the following steps: An illumination speckle sequence is generated based on the Hadamard matrix, and the generated illumination speckle is used to sample the object to obtain an intensity sequence. The illumination speckle sequence and light intensity sequence are input into the trained image reconstruction and classification model to obtain the corresponding images and image categories; The image reconstruction and classification model adopts a GRU-CNN hybrid network. The training and testing datasets for the image reconstruction and classification model include illumination speckle sequence, light intensity sequence, fusion data of illumination speckle sequence and light intensity sequence up-dimensional splicing, original image and original image label. The GRU-CNN hybrid network includes a light intensity sequence feature extraction module, a light intensity sequence and speckle fusion data feature extraction module, and an image classification and image reconstruction module, wherein: Light intensity sequence feature extraction module: used to extract temporal features from the input light intensity sequence; Fusion data feature extraction module: used to receive and fuse the measured light speckle sequence and the features extracted by the light intensity sequence feature extraction module, and perform multi-scale feature extraction and output; Image classification and image reconstruction module: used to reconstruct and classify images based on the multi-scale features output by the light intensity sequence and speckle fusion data feature extraction module.

2. The ghost imaging reconstruction and classification method based on hybrid networks according to claim 1, characterized in that, The process of generating the light speckle sequence is as follows: Recursively construct a one-dimensional node tree and recursively generate the left and right subtrees; Perform a matrix dot product operation between the row vector of the left leaf node and the column vector of the right leaf node to generate the original Hadamard matrix, and then convert it to obtain the binary form of the Hadamard transformation matrix. The Hadamard transformation matrix is ​​traversed in a zigzag pattern to obtain the illumination speckle sequence.

3. The ghost imaging reconstruction and classification method based on hybrid networks according to claim 1, characterized in that, The formula for calculating light intensity is: in, This represents the total light intensity value. Let be the transmittance function of the object under test. The light field distribution on the surface of the target object is shown, and the subscript i indicates the i-th sampling.

4. The ghost imaging reconstruction and classification method based on hybrid networks according to claim 1, characterized in that, The construction of the training and testing datasets for the image reconstruction and classification model includes: From the illumination speckle sequence, the first 3M illumination speckles are selected in sequence and loaded onto a digital micromirror device in sequence to control the light field. The light intensity of the illuminated image is collected by a barrel detector to obtain the light intensity sequence. The obtained light intensity sequence is multiplied by the corresponding M illumination speckle sequences to obtain M-channel fused data; The dataset includes: the last 2M illumination speckles, intensity sequences, M-channel fused data, original images, and original image labels.

5. The ghost imaging reconstruction and classification method based on hybrid networks according to claim 1, characterized in that, The light intensity sequence feature extraction module includes two GRU-LayerNorm modules, which are sequentially connected via nn.ModuleList(). Each GRU-LayerNorm module includes: A GRU layer: used for time series modeling of the input data; A LayerNorm layer: used to normalize the output of the GRU layer, wherein the output dimension of the LayerNorm layer is the same as the output dimension of the GRU layer.

6. The ghost imaging reconstruction and classification method based on hybrid networks according to claim 1, characterized in that, The light intensity sequence and speckle fusion data feature extraction module includes: Multi-scale gated fusion module: Through a series of convolutional layers and gating mechanisms, combined with features extracted by multi-scale deep convolutional modules, the final fused feature output is generated; BatchNorm2d layer: Normalizes the input for each batch in a convolutional neural network.

7. The ghost imaging reconstruction and classification method based on hybrid networks according to claim 1, characterized in that, The multi-scale feature extraction process includes: The input feature data undergoes a first-stage 1×1 convolution process to map the number of channels in the input data to a predetermined dimension. By using a multi-scale depth convolution module, depthwise separable convolution operations with different dilation factors are employed to extract multi-scale features from the input data and generate feature maps at multiple scales. The feature maps at each scale are weighted and fused. The weighting coefficients are based on the channel allocation ratio, and the fused feature map is obtained by weighted summation of each channel. The fused feature map is input into the gated layer, which uses a 1×1 convolution to generate a gated signal. The gated signal is processed using the SiLU activation function to obtain the gate weight for each channel; The gated signal is multiplied channel by channel with the fused feature map; The second stage involves 1×1 convolution to adjust the number of channels, and the result is used as the final output.

8. The ghost imaging reconstruction and classification method based on hybrid networks according to claim 1, characterized in that, The image classification and image reconstruction module includes an image classification submodule and an image reconstruction submodule. The image classification submodule includes: Several convolutional-batch normalization blocks and pooling layers are used to progressively extract image features and reduce spatial dimensionality; An adaptive average pooling layer is used to transform the input two-dimensional feature map into an output of a specified size using an adaptive average pooling method. Fully connected layers are used to flatten the feature maps after convolution and pooling and output the classification results; The image reconstruction submodule includes: Several convolutional-batch normalization blocks: used to progressively extract features and reduce the number of channels; Output convolutional layer: used to generate the final image output; The Sigmoid activation function is used to generate normalized image output.

9. A terminal device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements as claimed in claim 1.

8. The steps of any of the methods described.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements as described in claim 1.

8. The steps of any of the methods described.

Images