Lensless photoelectric intelligent imaging optimization method and imaging method

Abstract

The application discloses a lens-free photoelectric intelligent imaging optimization method, which is applied to a lens-free photoelectric intelligent imaging system, and the lens-free photoelectric intelligent imaging system comprises a learnable optical diffraction layer, an image sensor and an image processing neural network which are sequentially arranged. The method comprises the following steps: acquiring training set image information; training the learnable optical diffraction layer and the image processing neural network according to the training set image information to obtain a target optical diffraction layer and a target image processing neural network after training; and the image sensor serves as a nonlinear connection layer. The learnable optical diffraction layer can effectively extract intensity or phase information of a light signal, so that the phase information can be converted into an intensity form and output through the image sensor, and multi-modal computational imaging is realized. Meanwhile, during the training, a deep optical concept is utilized to realize joint optimization in an end-to-end mode, and high-precision image feature extraction and reconstruction are realized.

Description

Technical Field

[0001] This invention belongs to the field of computer vision technology, specifically relating to a lensless photoelectric intelligent imaging optimization method and imaging method. Background Technology

[0002] Optical signals possess multi-dimensional information, such as amplitude and phase. However, existing image sensors only output light intensity information, resulting in the loss of phase information. How to convert phase information into intensity form for output by image sensors to achieve multimodal computational imaging remains a challenge. Optical diffraction neural networks utilize the diffraction effect of light to modulate the phase of incident light. The learnable phase diffraction layer allows most light to pass through, providing a high signal-to-noise ratio, making it suitable for low-light scenarios and photon-constrained imaging. Furthermore, optical diffraction neural networks offer advantages such as light speed, multimodality, and low power consumption. However, the signal feature extraction of optical convolutional neural networks relies on linear operations, making it difficult to match the feature extraction capabilities of convolutional operations. Therefore, achieving high-precision and multimodal feature extraction from optical signals remains a problem to be solved.

[0003] In current technologies, emerging applications such as implantable and wearable devices, autonomous vehicles, robotics, the Internet of Things (IoT), virtual reality / augmented reality, and human-computer interaction are driving the miniaturization of imaging devices. Traditional imaging devices include lenses, increasing weight, cost, and size, while also having strict requirements on focal length, i.e., the focal length is proportional to the aperture. However, lensless imaging devices abandon traditional optical lenses, i.e., optical focusing elements, retaining only the image sensor. Existing technologies use diffractive optical elements combined with computer image reconstruction algorithms. However, optimizing these two separately will lead to increased information entropy, which will severely affect the image quality of lensless imaging. Summary of the Invention

[0004] To address the aforementioned problems in the existing technology, this invention provides a lensless photoelectric intelligent imaging optimization method and imaging method. The technical problem to be solved by this invention is achieved through the following technical solution:

[0005] The first aspect of this invention provides a lensless photoelectric intelligent imaging optimization method, applied to a lensless photoelectric intelligent imaging system, the lensless photoelectric intelligent imaging system comprising: a learnable optical diffraction layer, an image sensor, and an image processing neural network arranged sequentially;

[0006] Includes the following steps:

[0007] Acquire training set image information; wherein, the training set image information includes: speckle image information or image information to be scaled; the speckle image information includes: speckle image intensity information or speckle image phase information; the image information to be scaled includes: image intensity information or image phase information to be scaled.

[0008] The learnable optical diffraction layer and the image processing neural network are trained based on the image information in the training set to obtain the trained target optical diffraction layer and the target image processing neural network.

[0009] During training, the image sensor serves as a non-linear connection layer between the learnable optical diffraction layer and the image processing neural network.

[0010] In one embodiment of the present invention, the transmission coefficient of each artificial neuron on the learnable optical diffraction layer is:

[0011]

[0012] Where i represents the i-th neuron, a i The amplitude of the i-th neuron is a constant. This represents the phase of the i-th neuron.

[0013] In one embodiment of the present invention, the transfer function between the learnable optical diffraction layer and the image sensor is:

[0014]

[0015] Where z represents the axial distance between the learnable optical diffraction layer and the image sensor, λ represents the illumination wavelength, and f x and f y These represent the frequencies in the x and y directions, respectively.

[0016] In one embodiment of the present invention, the intensity distribution of the i-th neuron on the learnable optical diffraction layer is represented as follows:

[0017]

[0018] in, This represents the intensity distribution of the k-th pixel in the input image.

[0019] In one embodiment of the present invention, the image processing neural network includes a U-Net neural network or a generative adversarial network.

[0020] In one embodiment of the present invention, the generative adversarial network includes: a generator and a decision maker;

[0021] The generator includes: a feature embedding module, a first downsampling module, a second downsampling module, a latent space feature aggregation bottleneck module, a first upsampling module, a second upsampling module, a nearest neighbor interpolation module, a Transformer encoder, and a nonlinear activation function layer;

[0022] The first downsampling module, the second downsampling module, the first upsampling module, and the second upsampling module are all Transformer coding modules that include a multi-head attention mechanism;

[0023] The output of the feature embedding module is connected to the input of the first downsampling module;

[0024] The output of the first downsampling module is connected to the input of the second downsampling module;

[0025] The output of the second downsampling module is connected to the input of the latent space feature aggregation bottleneck module;

[0026] The output of the latent space feature aggregation bottleneck module is connected to the input of the second upsampling module;

[0027] The output of the second upsampling module is connected to the input of the first upsampling module;

[0028] The output of the first upsampling module is connected to the input of the nearest neighbor interpolation module;

[0029] The nearest neighbor interpolation module, the Transformer encoder, and the nonlinear activation function layer are connected in sequence.

[0030] The first downsampling module and the first upsampling module are connected by a jump connection structure;

[0031] The second downsampling module and the second upsampling module are connected by a jump connection structure;

[0032] The output of the nonlinear activation function layer is connected to the input of the decision maker.

[0033] In one embodiment of the present invention, the Transformer encoder is a Transformer encoder that includes a linear space reduction attention mechanism.

[0034] In one embodiment of the invention, the decision maker is a pyramid-shaped visual Transformer.

[0035] A second aspect of this invention provides a lensless photoelectric intelligent imaging method, comprising the following steps:

[0036] Acquire image information to be processed; wherein, the image information to be processed includes: speckle image information to be processed or image information to be scaled; the speckle image information to be processed includes: speckle image intensity information or speckle image phase information; the image information to be scaled includes: image intensity information or image phase information.

[0037] The image information to be processed is input into the corresponding target optical diffraction layer provided in the first aspect of the present invention for processing and outputting the light field to be processed. Then, after being processed by the image sensor, it is input into the corresponding target image processing neural network provided in the first aspect of the present invention to output the reconstructed image or the scaled image.

[0038] The beneficial effects of this invention are:

[0039] This invention utilizes a learnable optical diffraction layer to effectively extract intensity or phase information of light signals. The phase information can be output to an image sensor in intensity form, thus converting the phase information into intensity for output through the image sensor, enabling multimodal computational imaging. Simultaneously, the image sensor, being a nonlinear function, allows for end-to-end joint optimization of the learnable optical diffraction layer and the image processing neural network during training using depth optics principles. This ensures that both the trained target optical diffraction layer and the target image processing neural network reach their optimal state, achieving high-precision image feature extraction and reconstruction.

[0040] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0041] Figure 1a This is a schematic flowchart of a lensless photoelectric intelligent imaging optimization method provided in an embodiment of the present invention;

[0042] Figure 1b This is an architectural diagram of a lensless photoelectric intelligent imaging system provided in an embodiment of the present invention;

[0043] Figure 2 A schematic flowchart of another lensless photoelectric intelligent imaging optimization method provided in an embodiment of the present invention;

[0044] Figure 3a This is a diagram of the U-Net neural network architecture provided in an embodiment of the present invention;

[0045] Figure 3b A diagram of a generative adversarial network architecture provided in an embodiment of the present invention;

[0046] Figure 4 This is a comparison diagram of the results of speckle imaging achieved by the present invention and traditional optical diffraction neural network phase information input;

[0047] Figure 5 This is a comparison chart showing the results of speckle imaging achieved by the present invention and traditional optical diffraction neural network intensity information input;

[0048] Figure 6 This is a comparison of the lens imaging results achieved by the present invention and the traditional optical diffraction neural network phase information input to achieve 2x magnification and 2x reduction;

[0049] Figure 7 This is a comparison of the lens imaging results achieved by the present invention and the traditional optical diffraction neural network using intensity information input to achieve 2x magnification and 2x reduction;

[0050] Figure 8 This is a schematic diagram of the Transformer encoding module for the multi-head attention mechanism provided in an embodiment of the present invention. Detailed Implementation

[0051] The present invention will be further described in detail below with reference to specific embodiments, but the implementation of the present invention is not limited thereto.

[0052] Example 1

[0053] like Figure 1a and Figure 1b As shown, this embodiment provides a lensless optoelectronic intelligent imaging optimization method applied to a lensless optoelectronic intelligent imaging system. The lensless optoelectronic intelligent imaging system includes: a learnable optical diffraction layer, an image sensor, and an image processing neural network arranged sequentially. Specifically, starting from the image input plane, image information is input through the image input plane and sequentially passes through the learnable optical diffraction layer, the image sensor, and the image processing neural network. The learnable optical diffraction layer and the image sensor serve as optical receiving modules for low-dimensional phase / intensity feature extraction, where the image sensor acts as a nonlinear function, and the image processing neural network, based on a computer neural network, serves as the image processing module.

[0054] Specifically, a lensless photoelectric intelligent imaging optimization method includes the following steps:

[0055] Step 11, obtain training set image information; wherein, the training set image information includes: speckle image information or image information to be scaled; the speckle image information includes: speckle image intensity information or speckle image phase information; the image information to be scaled includes: image intensity information or image phase information to be scaled.

[0056] Step 12: Train the learnable optical diffraction layer and image processing neural network based on the image information in the training set to obtain the trained target optical diffraction layer and target image processing neural network.

[0057] During training, the image sensor serves as a nonlinear connection layer between the learnable optical diffraction layer and the image processing neural network.

[0058] In this embodiment, the training of the learnable optical diffraction layer employs an optical diffraction neural network model. The image processing neural network is a computer neural network model used for the extraction and reconstruction of high-dimensional features. After processing the light field at the front end of the learnable optical diffraction layer, the training set image information is output and input into the learnable optical diffraction layer.

[0059] When the training set image information is speckle image information, the trained lensless photoelectric intelligent imaging system can be used for speckle image reconstruction. During training, it can be based on speckle image intensity information or speckle image phase information.

[0060] When the training set image information is the image information to be scaled, the trained lensless photoelectric intelligent imaging system can be used for scaling and reconstructing the input image, similar to lens imaging. During training, it can be based on the intensity information of the image to be scaled, or it can be based on the phase information of the image to be scaled.

[0061] In this embodiment, the intensity or phase information of the light signal can be effectively extracted through a learnable optical diffraction layer. The image sensor outputs the phase information as intensity, thus converting the phase information into intensity form and outputting it through the image sensor to achieve multimodal computational imaging. Simultaneously, the image sensor, acting as a nonlinear function between the learnable optical diffraction layer and the image processing neural network, enables end-to-end joint optimization of the two during training. This improves the matching degree between the target optical diffraction layer and the target image processing neural network, allowing both to reach their optimal state after training, thereby achieving high-precision image feature extraction and reconstruction.

[0062] Furthermore, the optical diffraction neural network is based on an all-optical machine learning framework. Each point on the learnable optical diffraction layer represents an artificial neuron, whose phase or amplitude can be independently modulated. The transmission coefficient of each artificial neuron on the learnable optical diffraction layer is:

[0063]

[0064] Where i represents the i-th neuron, a i The amplitude of the i-th neuron is a constant. This represents the phase of the i-th neuron. (Phase term) It can be viewed as a "bias" term of a learnable parameter, and the phase value of each neuron is restricted to 0 to 2π using the Sigmoid nonlinear activation function.

[0065] The transfer function between the learnable optical diffraction layer and the image sensor is expressed using the angular spectrum method as follows:

[0066]

[0067] Where z represents the axial distance between the learnable optical diffraction layer and the image sensor, λ represents the illumination wavelength, and f x and f y Let represent the frequencies in the x and y directions, respectively. Here, the transfer function between the image input plane and the learnable optical diffraction layer is also expressed using the above formula, where z represents the axial distance between the learnable optical diffraction layer and the image input plane.

[0068] The intensity distribution of the i-th neuron on the learnable optical diffraction layer is represented as:

[0069]

[0070] in, This represents the intensity distribution of the k-th pixel in the input image. The field strength directly in front of the learnable optical diffraction layer can be calculated based on the frequency domain representation of the intensity distribution of the k-th pixel, as well as its transfer function and transfer coefficient.

[0071] Furthermore, the image processing neural network can be a U-Net neural network or a generative adversarial network, which can solve the following three problems: First, the design optimization of the optical receiving module can introduce underoptimized reconstruction in the image processing module. Second, the reconstruction algorithm may greatly amplify noise in the image. Third, the reconstruction algorithm is ineffective for image enhancement with speckle patterns and lens imaging effects.

[0072] When the image processing neural network is a U-Net neural network, such as Figure 3a As shown. The encoder is used to extract features from the input image. The decoder aims to recover the encoded features. Convolutional layers are designed to fuse high-dimensional and low-dimensional features. Compared to fully connected layers, convolutional layers have higher feature extraction capabilities and robustness. Furthermore, the convolution operator can alleviate the well-known overfitting problem caused by limited training data. In a convolutional neural network, the data flow of each convolutional module consists of a 7x7 kernel convolutional layer, a dropout layer, a ReLU non-linear activation layer, and a batch normalization layer. The convolutional neural network uses max pooling and nearest neighbor interpolation (NNI) for downsampling and upsampling, and the output layer is a Softmax non-linear activation layer.

[0073] When the image processing neural network is a generative adversarial network, it can extract global features better and has a certain generalization ability.

[0074] Specifically, generative adversarial networks include: generators and decision-makers;

[0075] The generator includes: a feature embedding module 100, a first downsampling module 200, a second downsampling module 300, a latent space feature aggregation bottleneck module 400, a first upsampling module 500, a second upsampling module 600, a nearest neighbor interpolation module 700, a Transformer encoder 800, and a nonlinear activation function layer 900; the generator package is a generator with a global feature extraction mechanism, Transformer.

[0076] The first downsampling module 200, the second downsampling module 300, the first upsampling module 500, and the second upsampling module 600 are all Transformer coding modules that include a multi-head attention mechanism;

[0077] The output of the feature embedding module 100 is connected to the input of the first downsampling module 200; the output of the first downsampling module 200 is connected to the input of the second downsampling module 300; the output of the second downsampling module 300 is connected to the input of the latent space feature aggregation bottleneck module 400; the output of the latent space feature aggregation bottleneck module 400 is connected to the input of the second upsampling module 600; the output of the second upsampling module 600 is connected to the input of the first upsampling module 500; the output of the first upsampling module 500 is connected to the input of the nearest neighbor interpolation module 700; the nearest neighbor interpolation module 700, the Transformer encoder 800, and the nonlinear activation function layer 900 are connected in sequence.

[0078] The first downsampling module 200 and the first upsampling module 500 are connected via a skip connection structure; the second downsampling module 300 and the second upsampling module 600 are also connected via a skip connection structure; the output of the nonlinear activation function layer 900 is connected to the input of the decision unit. The nonlinear activation function layer 900 uses the Sigmoid nonlinear activation function.

[0079] like Figure 3bAs shown, the generator is designed as a U-shaped global feature extraction network, incorporating a Transformer encoding module with a multi-head attention mechanism as a sampling module (first downsampling module 200, second downsampling module 300, first upsampling module 500, and second upsampling module 600). First, the Transformer encoding modules (first downsampling module 200 and second downsampling module 300) perform attention calculations on the input image information and downsample layer by layer to extract features, outputting a feature map. Then, through skip connections, the feature image output by the Transformer encoding module in the encoder is fused into the corresponding layer's Transformer encoding module in the decoder (first upsampling module 500 and second upsampling module 600) to obtain a fused feature map. Finally, the decoding module performs corresponding attention calculations and upsampling layer by layer to achieve image enhancement, completing the training. The decision maker of the generative adversarial network uses a pyramid-shaped visual Transformer (PVT).

[0080] The structure of the Transformer encoding module in the multi-head attention mechanism is as follows: Figure 8 As shown (using the corresponding technology disclosed in patent 202211386549.X), the image information processing process of this module is as follows: The image information first passes through a batch normalization layer (BatchNorm), and then through a multi-head attention mechanism (MHA) module composed of self-attention mechanisms to update the attention of the feature map; after the feature map is concatenated, it is transformed by a linear layer and then passed through a dropout layer to accumulate features with the input information to obtain an intermediate feature map; next, it passes through a batch normalization layer, and then through a fully connected layer (Position-wise feed-forward networks) to perform a nonlinear transformation on the feature map, and then through a dropout layer, and finally through a feature accumulation operation with the intermediate feature map and then through a batch normalization layer for output.

[0081] In one feasible implementation, such as Figure 2 As shown, the lensless photoelectric intelligent imaging optimization method of the present invention can also specifically adopt the following process:

[0082] Step S1 involves pre-training the learnable optical diffraction layer and image processing neural network using a large amount of data, enabling them to extract image features. During training, the image sensor acts as a non-linear function between the learnable optical diffraction layer and the image processing neural network.

[0083] Step S2: Create training set image information and train the network obtained through pre-training in step S1; during the training process, the image sensor acts as a nonlinear function between the learnable optical diffraction layer and the image processing neural network.

[0084] Step S3: Create a test dataset A that is independent of the training set image information, and test the network trained in step S2.

[0085] Step S4: Verify whether the test results meet the various indicators, including whether they meet the performance indicators and verification indicators related to neural networks.

[0086] If the test results do not meet the performance criteria, adjust the network parameters and repeat steps S2 and S3 until the test results meet the expected verification criteria. Then, end the training and testing to obtain the target optical diffraction layer and the target image processing neural network.

[0087] If the test results meet all the performance indicators, then the training and testing will end, and the target optical diffraction layer and the target image processing neural network will be obtained.

[0088] Step S5: Create a validation dataset B to validate the network after training and testing, and realize image reconstruction or scaling.

[0089] The network training parameters include: learning rate, optimizer, loss function, batch size, and training epochs. Validation metrics include Pearson coherence coefficient (PCC), Jaccard index, similarity index (SSIM), and peak signal-to-noise ratio (PSNR). The parameter settings differ depending on whether the image information is speckle or to be scaled. During training with images to be scaled, the training objective can be set to either reduced or enlarged imaging as needed.

[0090] The training effect described above will be tested below:

[0091] First, in imaging, refractive index mismatch in the sample can lead to scattering. Using 220-mesh frosted glass as the scattering medium, speckle image information of the target image was acquired to train the learnable optical diffraction layer and image processing neural network in the aforementioned "end-to-end" manner.

[0092] 1. Phase information input enables speckle imaging:

[0093] The speckle image phase information is input into a lensless optoelectronic intelligent imaging system for training. During training, the optimal configuration of the parameters of the optical receiving module and the image processing module is optimized. After training, the speckle image phase information test set is input for testing, and the speckle image phase information test set is also input into a traditional optical diffraction neural network (obtained by a traditional independent training method) for testing. The test comparison results are as follows: Figure 4 As shown.

[0094] 2. Intensity information input enables speckle imaging:

[0095] The speckle image intensity information is input into the lensless optoelectronic intelligent imaging system for training. During training, the optimal configuration of the parameters of the optical receiving module and the image processing module is optimized. After training, the speckle image intensity information test set is input for testing, and the speckle image intensity information test set is also input into a traditional optical diffraction neural network for testing. The test comparison results are as follows: Figure 5 As shown.

[0096] Therefore, it can be seen that the reconstructed image of the lensless optoelectronic intelligent imaging system trained in this application has better clarity.

[0097] II. Lens-less image magnification and reduction imaging test:

[0098] 1. Image scaling is achieved by inputting phase information:

[0099] The phase information of the image to be scaled is input into a lensless optoelectronic intelligent imaging system for training. The training performs end-to-end training to achieve 2x magnification and 2x reduction of the input phase information. After training, a test set of the phase information of the image to be scaled is input for testing, and then input into a traditional optical diffraction neural network for testing. The test comparison results are as follows: Figure 6 As shown.

[0100] 2. Image scaling is achieved by inputting intensity information:

[0101] The intensity information of the image to be scaled is input into the lensless optoelectronic intelligent imaging system for training. The training performs end-to-end training, achieving 2x magnification and 2x reduction of the input intensity information. After training, the intensity information test set of the image to be scaled is input for testing, and then input into a traditional optical diffraction neural network for testing. The test comparison results are as follows: Figure 7 As shown.

[0102] Therefore, it can be seen that the reconstructed image of the lensless optoelectronic intelligent imaging system trained in this application has better clarity.

[0103] Example 2

[0104] This embodiment provides a lensless photoelectric intelligent imaging method, applied to a lensless photoelectric intelligent imaging system. The lensless photoelectric intelligent imaging system includes: a target optical diffraction layer trained when the training set image information is speckle image phase information, an image sensor, and a target image processing neural network trained when the training set image information is speckle image phase information, arranged sequentially in Embodiment 1. The method includes the following steps:

[0105] Step 21: Obtain the phase information of the speckle image to be processed; the phase information of the speckle image to be processed is a phase image from 0 to 2π.

[0106] Step 22: Input the image information to be processed into the target optical diffraction layer trained when the training set image information is speckle image phase information in Example 1, process it to output the light field to be processed, and then input it into the target image processing neural network trained when the training set image information is speckle image phase information in Example 1 after processing by the image sensor, and output the reconstructed image.

[0107] In step 22, the image information processing involves a nonlinear function between the image sensor as a learnable optical diffraction layer and the target image processing neural network.

[0108] Example 3

[0109] This embodiment provides a lensless photoelectric intelligent imaging method, applied to a lensless photoelectric intelligent imaging system. The lensless photoelectric intelligent imaging system includes: a target optical diffraction layer trained when the training set image information in Embodiment 1 is speckle image intensity information, an image sensor, and a target image processing neural network trained when the training set image information in Embodiment 1 is speckle image intensity information, arranged sequentially, including the following steps:

[0110] Step 31: Obtain the intensity information of the speckle image to be processed; the phase information of the speckle image to be processed is an intensity image normalized to 0 to 1.

[0111] Step 32: Input the image information to be processed into the target optical diffraction layer trained when the training set image information is speckle image intensity information in Example 1, process it to output the light field to be processed, and then input it into the target image processing neural network trained when the training set image information is speckle image intensity information in Example 1 after processing by the image sensor, and output the reconstructed image.

[0112] In step 32, the image sensor acts as a nonlinear function between the target learnable optical diffraction layer and the target image processing neural network.

[0113] Example 4

[0114] This embodiment provides a lensless photoelectric intelligent imaging method, applied to a lensless photoelectric intelligent imaging system. The lensless photoelectric intelligent imaging system includes: a target optical diffraction layer trained when the training set image information is the phase information of the image to be scaled, an image sensor, and a target image processing neural network trained when the training set image information is the phase information of the image to be scaled, arranged sequentially in Embodiment 1. The method includes the following steps:

[0115] Step 41: Obtain the phase information of the image to be processed and scaled; the phase information of the image to be processed and scaled is a phase image from 0 to 2π.

[0116] Step 42: Input the image information to be processed into the target optical diffraction layer trained when the training set image information is the phase information of the image to be scaled in Example 1, process it to output the light field to be processed, and then input it into the target image processing neural network trained when the training set image information is the phase information of the image to be scaled in Example 1 after processing by the image sensor, and output the scaled or enlarged image.

[0117] In step 42, the image sensor acts as a nonlinear function between the target learnable optical diffraction layer and the target image processing neural network.

[0118] Example 5

[0119] This embodiment provides a lensless photoelectric intelligent imaging method, applied to a lensless photoelectric intelligent imaging system. The lensless photoelectric intelligent imaging system includes: a target optical diffraction layer trained when the training set image information is the intensity information of the image to be scaled, an image sensor, and a target image processing neural network trained when the training set image information is the intensity information of the image to be scaled, arranged sequentially in Embodiment 1. The method includes the following steps:

[0120] Step 51: Obtain the intensity information of the image to be processed and scaled; the intensity information of the image to be processed and scaled is an intensity image normalized to 0 to 1.

[0121] Step 52: Input the image information to be processed into the target optical diffraction layer trained when the training set image information is the intensity information of the image to be scaled in Example 1, process it to output the light field to be processed, and then input it into the target image processing neural network trained when the training set image information is the intensity information of the image to be scaled in Example 1 after processing by the image sensor, and output the scaled or enlarged image.

[0122] In step 52, the image sensor acts as a nonlinear function between the target learnable optical diffraction layer and the target image processing neural network.

[0123] Example 6

[0124] This embodiment, as a feasible implementation, provides a lensless photoelectric intelligent imaging method, including the following steps:

[0125] Step 61: Obtain speckle image information containing intensity and phase. The speckle image information includes two main categories: intensity images normalized to 0 to 1 and phase images from 0 to 2π. The dataset consists of a training dataset, a test dataset, and a validation dataset.

[0126] Step 62: Train the optical receiving module and image processing module in the lensless photoelectric intelligent imaging model based on the image information;

[0127] Step 63: Input the image information to be processed into the trained model for feature extraction to achieve image enhancement.

[0128] Furthermore, the model training process is as follows:

[0129] First, the network is pre-trained using a large amount of data to enable it to extract image features;

[0130] Secondly, create a training dataset to train the model;

[0131] Next, create a validation dataset A that is independent of the training dataset and test the trained network. If the validation results do not meet the performance metrics, repeat the training of the network until the validation results meet the expected performance metrics.

[0132] Finally, a validation dataset B is created, and the trained and validated network is used to reconstruct images.

[0133] The network training parameters include: learning rate, optimizer, loss function, batch size, and training epochs. Validation metrics include Pearson coherence coefficient (PCC), Jaccard index, similarity index (SSIM), and signal-to-noise ratio (PSNR). The parameter settings during training differ depending on whether the image information is speckle or the image to be scaled.

[0134] Example 7

[0135] This embodiment, as a feasible implementation, provides a lensless photoelectric intelligent imaging method, including the following steps:

[0136] Step 71: Obtain scaled image information containing intensity and phase. The image information includes two main categories: intensity images normalized to 0 to 1 and phase images from 0 to 2π. The dataset consists of a training dataset and a validation dataset.

[0137] Step 72: Using the concept of "depth optics", train the optical receiving module and image processing module in the lensless optoelectronic intelligent imaging model according to the phase or intensity information of the image;

[0138] Step 73: Input the image information to be processed into the trained model for feature extraction to achieve image enhancement.

[0139] Furthermore, the model training process is as follows:

[0140] First, the network is pre-trained using a large amount of data to enable it to extract image features;

[0141] Secondly, a training dataset is created to train the network;

[0142] Next, create a validation dataset A that is independent of the training dataset and test the trained network. If the validation results do not meet the performance metrics, repeat the training of the network until the validation results meet the expected performance metrics.

[0143] Finally, a validation dataset B is created, and the trained and validated network is used to scale images.

[0144] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0145] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. In addition, those skilled in the art can combine and integrate the different embodiments or examples described in this specification.

[0146] The above description, in conjunction with specific preferred embodiments, provides a further detailed explanation of the present invention. It should not be construed that the specific implementation of the present invention is limited to these descriptions. For those skilled in the art, various simple deductions or substitutions can be made without departing from the concept of the present invention, and all such modifications and substitutions should be considered within the scope of protection of the present invention.

Claims

1. A lensless photoelectric intelligent imaging optimization method, characterized in that, An application is made in a lensless optoelectronic intelligent imaging system, which includes: a learnable optical diffraction layer, an image sensor, and an image processing neural network arranged sequentially; wherein, the learnable optical diffraction layer is used to perform phase modulation on the incident light, and the image processing neural network is used to reconstruct or scale the image; Includes the following steps: Acquire training set image information; wherein, the training set image information includes: speckle image information or image information to be scaled; the speckle image information includes: speckle image intensity information or speckle image phase information; the image information to be scaled includes: image intensity information or image phase information to be scaled. The learnable optical diffraction layer and the image processing neural network are trained based on the image information in the training set to obtain the trained target optical diffraction layer and the target image processing neural network. During training, the image sensor serves as a non-linear connection layer between the learnable optical diffraction layer and the image processing neural network to achieve end-to-end joint optimization of the learnable optical diffraction layer and the image processing neural network.

2. The lensless photoelectric intelligent imaging optimization method according to claim 1, characterized in that, The transmission coefficient of each artificial neuron on the learnable optical diffraction layer is: ； in, Indicates the first One neuron, Indicates the first The amplitude of each neuron is a constant. Indicates the first The phase of each neuron.

3. The lensless photoelectric intelligent imaging optimization method according to claim 1, characterized in that, The transfer function between the learnable optical diffraction layer and the image sensor is: ； in, This represents the axial distance between the learnable optical diffraction layer and the image sensor. Indicates the illumination wavelength. and They represent and Frequency in the direction.

4. The lensless photoelectric intelligent imaging optimization method according to claim 3, characterized in that, The learnable optical diffraction layer on the first The intensity distribution of each neuron is represented as follows: ； in, Represents the first input image Intensity distribution of each pixel.

5. The lensless photoelectric intelligent imaging optimization method according to claim 1, characterized in that, The image processing neural network includes a U-Net neural network or a generative adversarial network.

6. The lensless photoelectric intelligent imaging optimization method according to claim 5, characterized in that, The generative adversarial network includes: a generator and a decision maker; The generator includes: a feature embedding module, a first downsampling module, a second downsampling module, a latent space feature aggregation bottleneck module, a first upsampling module, a second upsampling module, a nearest neighbor interpolation module, a Transformer encoder, and a nonlinear activation function layer; The first downsampling module, the second downsampling module, the first upsampling module, and the second upsampling module are all Transformer coding modules that include a multi-head attention mechanism; The output of the feature embedding module is connected to the input of the first downsampling module; The output of the first downsampling module is connected to the input of the second downsampling module; The output of the second downsampling module is connected to the input of the latent space feature aggregation bottleneck module; The output of the latent space feature aggregation bottleneck module is connected to the input of the second upsampling module; The output of the second upsampling module is connected to the input of the first upsampling module; The output of the first upsampling module is connected to the input of the nearest neighbor interpolation module; The nearest neighbor interpolation module, the Transformer encoder, and the nonlinear activation function layer are connected in sequence. The first downsampling module and the first upsampling module are connected by a jump connection structure; The second downsampling module and the second upsampling module are connected by a jump connection structure; The output of the nonlinear activation function layer is connected to the input of the decision maker.

7. The lensless photoelectric intelligent imaging optimization method according to claim 6, characterized in that, The Transformer encoder is a Transformer encoder that includes a linear space reduction attention mechanism.

8. The lensless photoelectric intelligent imaging optimization method according to claim 6, characterized in that, The decision maker is a pyramid-shaped visual Transformer.

9. A lensless photoelectric intelligent imaging method, used to implement any one of claims 1 to 8, characterized in that, Includes the following steps: Acquire image information to be processed; wherein, the image information to be processed includes: speckle image information to be processed or image information to be scaled; the speckle image information to be processed includes: speckle image intensity information or speckle image phase information; the image information to be scaled includes: image intensity information or image phase information. The image information to be processed is input into the corresponding target optical diffraction layer as described in any one of claims 1-8 for processing to output the light field to be processed. Then, after being processed by the image sensor, it is input into the corresponding target image processing neural network as described in any one of claims 1-8 to output the reconstructed image or the scaled image.

Images