Image reconstruction methods, devices, and electronic equipment based on metasurface imaging

Abstract

An image reconstruction method, apparatus, and electronic device based on metasurface imaging are disclosed, relating to the fields of optical imaging and image processing; capable of improving image quality in complex environments. The image reconstruction method based on metasurface imaging includes: acquiring a multispectral polarization image using a metasurface imaging device; inputting the multispectral polarization image into a spectral reconstruction module to output a reconstructed hyperspectral image; inputting the hyperspectral image into a polarization reconstruction module, whereby the polarization reconstruction module employs a polarization total variation reconstruction sensing algorithm to obtain a polarization image of the target polarization direction.

Description

Technical Field

[0001] This application relates to the field of optical imaging and image processing, and in particular to an image reconstruction method, apparatus and electronic device based on metasurface imaging. Background Technology

[0002] Traditional visible light optical imaging systems rely on complex lens groups to achieve imaging functions, which have inherent drawbacks such as bulky size, complex structure, and difficulty in chromatic aberration compensation. Under complex environments such as strong light illumination, surface reflection, backlighting, and low illumination, existing imaging systems are susceptible to stray light interference, resulting in problems such as image glare, blurring, reduced resolution, and edge chromatic aberration, which makes it impossible to effectively extract target information.

[0003] While existing metasurface technologies have shown potential in optical field manipulation, most designs focus only on single-dimensional optical field control, failing to achieve coordinated control of phase, polarization, and spectrum, and lacking systematic solutions for complex environmental interference. Furthermore, images manipulated by metasurfaces are susceptible to aberrations and dispersion, making it difficult for traditional image restoration methods to simultaneously preserve polarization and spectral information while correcting distortion, thus limiting their application in practical scenarios.

[0004] Therefore, developing an imaging technology that combines lightweight design, multi-dimensional light field manipulation capabilities, and efficient image restoration is key to solving the problem of high-quality imaging in complex environments. Summary of the Invention

[0005] This application provides an image reconstruction method, apparatus, and electronic device based on metasurface imaging to solve the technical problems of existing imaging systems, such as bulky size, poor adaptability to complex environments, and poor image quality.

[0006] In a first aspect, this application provides an image reconstruction method based on metasurface imaging, comprising: Acquiring multispectral polarization images using a metasurface imaging device; The multispectral polarization image is input into the spectral reconstruction module, which outputs the reconstructed hyperspectral image. The hyperspectral image is input into the polarization image reconstruction module, which uses a polarization total variation reconstruction sensing algorithm to obtain a polarization image of the target polarization direction.

[0007] The method provided in this embodiment employs a metasurface imaging device to synergistically control the phase, polarization, and spectrum of the light field. By using the subwavelength nanostructure of the metasurface to synergistically manipulate the phase, polarization, and spectral dimensions of the light field, it disrupts the spatial coherence of the incident light, suppresses speckle noise and parasitic interference fringes, and improves imaging quality. Combined with a deep learning-based intelligent image restoration algorithm, it achieves simultaneous optimization of system lightweighting, aberration correction, and environmental interference suppression, enabling efficient acquisition of clear polarization spectral images under complex lighting conditions.

[0008] Secondly, this application provides an image reconstruction apparatus based on metasurface imaging, comprising: Metasurface image acquisition module, used to acquire multispectral polarization images using a metasurface imaging device; The image restoration and reconstruction module is used to input the multispectral polarization image into the spectral reconstruction module and output the reconstructed hyperspectral image; the hyperspectral image is input into the polarization reconstruction module, and the polarization reconstruction module uses a polarization total variation reconstruction sensing algorithm to obtain a polarization image of the target polarization direction.

[0009] Thirdly, this application provides an electronic device including a memory and one or more processors. The memory stores one or more computer programs, each including instructions that, when executed by the processor, cause the electronic device to perform the image reconstruction method based on metasurface imaging as described in the first aspect.

[0010] Fourthly, this application provides a computer-readable storage medium storing instructions that, when executed on an electronic device, cause the electronic device to perform the image reconstruction method based on metasurface imaging as described in the first aspect.

[0011] Fifthly, this application provides a computer program product that, when run on an electronic device, causes the electronic device to perform the image reconstruction method based on metasurface imaging as described in the first aspect.

[0012] Understandably, the beneficial effects achieved by the image reconstruction apparatus, electronic device, computer-readable storage medium, and computer program product based on metasurface imaging provided above can be referred to the beneficial effects in the first aspect, and will not be repeated here. Attached Figure Description

[0013] Figure 1 A schematic flowchart illustrating the image reconstruction method based on metasurface imaging provided in this application embodiment; Figure 2 Another flowchart of the image reconstruction method based on metasurface imaging provided in the embodiments of this application; Figure 3 This is another schematic diagram of the image reconstruction method based on metasurface imaging provided in the embodiments of this application; Figure 4 A schematic flowchart of the spectral reconstruction module in the image reconstruction method based on metasurface imaging provided in this application embodiment; Figure 5 A schematic flowchart of the polarization reconstruction module in the image reconstruction method based on metasurface imaging provided in the embodiments of this application; Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0014] To facilitate a clear description of the technical solutions in the embodiments of this application, the terms "first" and "second" are used in the embodiments of this application to distinguish identical or similar items with substantially the same function and effect. For example, "first chip" and "second chip" are only used to distinguish different chips and do not limit their order. Those skilled in the art will understand that the terms "first" and "second" do not limit the quantity or execution order, and the terms "first" and "second" do not necessarily imply that they are different. It should be noted that in the embodiments of this application, the words "exemplary" or "for example" are used to indicate that they are examples, illustrations, or descriptions. Any embodiment or design scheme described as "exemplary" or "for example" in this application should not be construed as being better or more advantageous than other embodiments or design schemes. Specifically, the use of the words "exemplary" or "for example" is intended to present the relevant concepts in a specific manner. In the embodiments of this application, "at least one" means one or more, and "more than one" means two or more.

[0015] It should be noted that "at the time of..." in the embodiments of this application can be either at the instant when a certain situation occurs, or for a period of time after the occurrence of a certain situation. The embodiments of this application do not make specific limitations on this.

[0016] The implementation of this embodiment will now be described in detail with reference to the accompanying drawings.

[0017] This embodiment provides an image reconstruction method based on metasurface imaging. For example, this image reconstruction method based on metasurface imaging can be applied to various electronic devices such as computers (PCs), tablets, virtual reality / augmented reality devices, wearable devices, industrial computers, and vehicle-mounted systems; it can also be applied to servers, cloud computing, server clusters, etc. This embodiment does not impose any special limitations on it.

[0018] Figure 1 A schematic flowchart of the image reconstruction method based on metasurface imaging provided in this application embodiment is shown.

[0019] like Figure 1 As shown, the image reconstruction method based on metasurface imaging may include the following steps: Step 101: Acquire multispectral polarization images using a metasurface imaging device.

[0020] Step 102: Input the multispectral polarization image into the spectral reconstruction module and output the reconstructed hyperspectral image.

[0021] Step 103: Input the hyperspectral image into the polarization reconstruction module. The polarization reconstruction module uses the polarization total variation reconstruction sensing algorithm to obtain the polarization image of the target polarization direction.

[0022] In step 101, the metasurface imaging device is obtained through the selection of wafer structure and the design of surface source. Specifically, different wafer structures are traversed and their transmittance response curves are obtained to screen out wafer structures that meet the requirements of multispectral polarization modulation; the surface sources of the wafer structure are randomly arranged to obtain spectral response data of different surface source structures under multiple polarizations, and the surface source structure is optimized through deep learning algorithms; based on the optimized surface source structure, processing and detector pixel patching are performed to form a metasurface imaging device, and multispectral polarization image data is obtained using the metasurface imaging device.

[0023] The process of traversing different wafer structures and obtaining their transmittance response curves to screen out wafer structures that meet the requirements of multispectral polarization modulation specifically includes: traversing wafer structures, which include wafer units with different geometries, sizes, and materials; obtaining the transmittance response curves of each wafer structure in the target band; and determining the target wafer structure that meets the requirements of multispectral polarization modulation based on the bandwidth, peak transmittance, and polarization discrimination of the transmittance curves.

[0024] The process involves randomly arranging the surface sources of a crystal structure to obtain spectral response data for different surface source structures under multiple polarizations. Optimizing the surface source structure using a deep learning algorithm specifically includes: randomly arranging the surface sources of the target crystal structure to generate multiple sets of surface source structure schemes; building an optical testing platform to obtain spectral response datasets of different surface source structures under polarizations of 0°, 45°, 90°, and 135°; and using a deep learning algorithm to train the dataset and optimize the spectral polarization modulation performance of the surface source structure.

[0025] The nanostructure units of the surface source structure are arranged with superpixels as the basic units; the superpixel is composed of a 3×3 array of 9 preferred structural units selected by residual isotropic power. The superpixel is periodically replicated and expanded within the imaging target area to destroy the spatial coherence of the incident light and suppress speckle noise and parasitic interference fringes.

[0026] In step 102, the spectral reconstruction module specifically includes an initial feature extraction module, a feature iteration optimization module, and a first reconstruction module. The initial feature extraction module includes two branches. The first branch upsamples the input multispectral polarization image and uses bicubic interpolation to enlarge the image spatial size to the target size, extracting shallow features. The second branch uses three parallel convolutional layers to extract features respectively, and adds the features from the two branches element-wise to obtain the initial feature map. The feature iteration optimization module consists of multiple dense residual channel affinity (DRCA) modules cascaded with a spatial information guided propagation (SGP) module. The DRCA module includes an encoder / decoder and a channel affinity submodule. Multi-scale features are extracted through the encoder / decoder, and channel weights are learned using the channel affinity submodule. The features are then weighted to obtain depth-optimized features. The SGP module simulates the imaging degradation process to generate residual information, and adaptively fuses the residual information with the main features through a deformable adaptive fusion submodule to refine the spatial features. The first reconstruction module adds and fuses the initial feature map with the depth-optimized features through a global skip connection, and then uses a convolutional layer to adjust the number of channels of the fused features, outputting the final high-resolution hyperspectral image.

[0027] In step 103, the polarization reconstruction module uses the Polarization Total Variation Reconstruction Sensing (PTCS) algorithm to reconstruct the high-resolution hyperspectral image, obtaining a polarization image of the target polarization direction; the PTCS algorithm is used to optimize the following formula:

[0028] Where y is the input image, i.e., the hyperspectral image reconstructed in the previous step, x is the reconstructed image, and A is the measurement matrix. The weights are the total variation constraint coefficients. For the weights of the interspectral constraint coefficients, Let x be the nuclear norm. This algorithm is a stepwise optimization process, where x is the image obtained after the final optimization by the PTCS algorithm.

[0029] The hyperspectral image optimized by the PTCS algorithm is input into the trained Pz-Unet network to obtain the polarization image of the target polarization direction. The Pz-Unet network model is a U-Net encoder-decoder network based on Res2Net-SE and context self-attention.

[0030] After obtaining the polarization image of the target polarization direction, the polarization image can be transformed into the Fourier domain. The polarization image can then be optimized using an adversarial network and a peak signal-to-noise ratio loss function constraint, resulting in an optimized polarization image.

[0031] In this embodiment, by systematically traversing wafer units of different geometries, sizes, and materials, the transmittance response curves of these wafers in the target wavelength range of 300nm-1000nm are tested and analyzed. Based on the bandwidth, peak transmittance, and polarization discrimination of the transmittance curves, wafer structures suitable for multispectral polarization modulation are selected, providing the basic units for subsequent surface source design. The surface source design module first randomly arranges the surface sources of the selected wafer structures to generate multiple sets of surface source structure schemes. Then, an optical testing platform is built to collect spectral response datasets of different surface source structures under polarizations of 0°, 45°, 90°, and 135°. Next, a deep learning algorithm integrating convolutional neural networks and reinforcement learning is used to train the dataset end-to-end with spectral mean square error loss and polarization contrast loss as objective functions to optimize the spectral polarization modulation performance of the surface source structure. Finally, based on the optimization results, the micro-nano fabrication design of the surface source is completed, and the detector pixels are patched to ensure the spatial matching between the surface source and the detector. The spectral polarization data processing module uses an imaging detector to acquire multispectral polarization image data after being modulated by the wafer and the surface source. It then uses polarization restoration algorithms and spectral restoration algorithms to accurately restore the image data in the polarization and spectral dimensions, and finally fuses and outputs multispectral polarization data.

[0032] In one exemplary embodiment, the image reconstruction method based on metasurface imaging may specifically include three parts: crystal structure selection, surface source design, and spectral polarization data, such as... Figure 2 As shown. The wafer structure selection includes: traversing wafer structures, obtaining transmittance response curves for different wafer structures, and thus selecting a suitable wafer structure. The surface source design includes: randomly arranging surface sources, obtaining spectral response datasets of different surface source structures under different polarizations, using a surface source optimization deep learning algorithm for surface source optimization, selecting a suitable surface source structure design, then fabricating the design, and performing radial detector pixel patching to obtain the metasurface imaging device. For the spectral polarization data part, firstly, multispectral polarization image data is obtained through the designed metasurface imaging device, and then polarization and spectral restoration are performed to obtain multispectral polarization data.

[0033] Figure 3 This diagram illustrates the process of optimizing the traversal of the wafer structure and the random arrangement of surface sources. (Refer to...) Figure 3 Chip structure selection module and surface source design module; The wafer structure selection module is used to traverse different wafer structures and obtain their transmittance response curves, and to screen out wafer structures that meet the requirements of multispectral polarization modulation. The surface source design module is used to randomly arrange the surface sources of the wafer structure, obtain the spectral response dataset of different surface source structures under multiple polarizations, optimize the surface source structure through deep learning algorithms, and complete the fabrication design and detector pixel patching. The spectral polarization data processing module is used to acquire multispectral polarization image data, perform polarization restoration and spectral restoration, and output multispectral polarization data.

[0034] The workflow of the wafer structure selection module includes: Step 1: Traverse the wafer structure, covering wafer units of different geometries, sizes and materials; Step 2: Test and obtain the transmittance response curves of each crystal structure in the target wavelength band; Step 3: Select a suitable wafer structure based on the bandwidth, peak transmittance, and polarization resolution of the transmittance curve.

[0035] For example, the selected wafer structure needs to satisfy: Bandwidth: Covers the target wavelength band, and the transmittance changes gradually within this band or has the spectral response characteristics required by the design. Peak transmittance: Has high transmittance at the target wavelength to improve the signal-to-noise ratio. Polarization discrimination: The transmittance curves for different polarization directions (e.g., 0°, 45°, 90°, 135°) show significant differences to support the effective extraction of polarization information.

[0036] The workflow of the surface source design module includes: Step 1: Randomly arrange the surface sources of the selected wafer structure to generate multiple sets of surface source structure schemes; Step 2: Build an optical testing platform and obtain spectral response datasets of different surface source structures under polarization at 0°, 45°, 90°, and 135°. Step 3: Use deep learning algorithms to train the dataset and optimize the spectral polarization modulation performance of the surface source structure; Correlation coefficient:

[0037]

[0038]

[0039] in, Represents the transmission spectrum response of the i-th and j-th metasurface units. express Covariance between the two transmission spectrum responses , express The mean value, σ represents the standard deviation of the transmission spectrum. This represents the number of spectral bands.

[0040] By obtaining low correlation coefficients in the spectral response curves of different surface source units, an approximately orthogonal sensing matrix is ​​constructed, thereby transforming the original underdetermined reconstruction problem into a stable optimization problem, which enables high-quality reconstruction of spectral data.

[0041] Based on the optimization results, a suitable surface source structure is selected, micro-nano fabrication design is carried out, and the detector pixels are patched to achieve precise matching between the surface source and the detector.

[0042] Metasurface imaging devices can produce different responses to light with different polarizations through structural anisotropy. Wavelength selection and spectral modulation are achieved through structural resonance effects. These modulations are simultaneously realized on the same metasurface structure, enabling multi-dimensional coordinated manipulation of incident light.

[0043] By using a metasurface imaging device to coordinate the phase, polarization, and spectrum of the light field, multispectral polarization image data is obtained. Then, a polarization spectrum reconstruction module is used to restore the polarization and spectrum, and output multispectral polarization data.

[0044] The polarization spectral reconstruction module includes a spectral reconstruction module and a polarization reconstruction module. The spectral reconstruction module is composed of multiple cascaded Dense Residual Channel Affinity (DRCA) modules and a Spatial Information Guided Propagation (SGP) module. It achieves hyperspectral image reconstruction through bicubic interpolation upsampling, shallow feature extraction, iterative refinement, and global skip connections. The polarization reconstruction module adopts the PTCS algorithm combined with the Pz-Unet network. The PTCS algorithm achieves preliminary reconstruction through spatial total variation constraints and effective singular values ​​and constraints between spectra. The Pz-Unet network embeds the Res2Net-SE-Conv module and the context self-attention module to improve the image reconstruction quality at low sampling rates.

[0045] The spectral reconstruction module uses a model consisting of a cascaded DRCA module and an SGP module. The specific processing procedure is as follows: Figure 4 As shown, it includes the following: Step 1: The model upsamples the low-resolution hyperspectral image (i.e., the input multispectral polarization image) using bicubic interpolation to obtain preliminary hyperspectral features. Then, it extracts shallow features from the upsampled image and the panchromatic image using two 1×1 convolutions and expands the number of channels. The sum of these features yields the initial features.

[0046] Step 2: Iteratively optimize features through multiple cascaded DRCA and SGP modules: The DRCA module extracts features through convolution operations of the encoder and decoder and cross-layer splicing, and the CAP module models channel correlation; the SGP module uses degradation simulation to generate residual information and refines features in combination with deformable adaptive fusion sub-modules.

[0047] Step 3: The initial features and optimized features are fused through global skip connections, and the number of channels is adjusted by 1×1 convolution to output a hyperspectral image.

[0048] The polarization reconstruction module uses the PTCS algorithm combined with the Pz-Unet network, such as... Figure 5 As shown, the specific processing procedure is as follows: Step 1: Using the PTCS algorithm, combined with spatial total variation reduction and inter-spectral low-rank constraints, the image is initially reconstructed by minimizing the gradients of neighboring pixels and maximizing the sum of effective singular values, resulting in multispectral images with four polarization directions. The joint reconstruction model of the PTCS algorithm can be expressed as the following optimization problem:

[0049] Where y: input image, x: image to be reconstructed, and A: measurement matrix (related to polarization / spectral response). Total variation constraint coefficient weights : Interspectral constraint coefficient weights Nuclear norm; The first term in the formula is the data fidelity term, which ensures that the reconstructed data is consistent with the input data; the second term is the spatial total variation constraint, which enhances the smoothness of the data by minimizing the gradient; the third term is the interspectral low-rank constraint, which maximizes the effective sum of singular values ​​and enhances the interspectral correlation.

[0050] The Polarization Full-Range Compressive Sensing Algorithm (PTCS) primarily addresses the reconstruction optimization problem by reconstructing a high-quality multispectral image x from input data y. Specifically, it's an iterative approximation process that alternately updates image and auxiliary variables using a total variation method, gradually finding the optimal solution that satisfies all constraints. Applying the total variation method to polarization determination improves the efficiency of polarization reconstruction.

[0051] Step 2: Use an image with a certain polarization direction as training data and input it into the Pz-Unet network for training. The Pz-Unet network model is a U-Net encoder-decoder network based on Res2Net-SE and context self-attention. Step 3: By inputting the spectral reconstruction image (output of the spectral reconstruction module) into the model trained by the decoding network, a high-quality polarization image corresponding to the spectrum can be output.

[0052] In an exemplary embodiment, the method may further include the following steps: Step 1: Phase, polarization, and spectral modulation of incident light are performed through a metasurface modulation module to suppress ambient light interference.

[0053] Step 2: Use the image acquisition module to capture the adjusted polarization spectrum image and obtain the original image data containing degradation features such as aberrations, dispersion, blurring, and low resolution.

[0054] Step 3: Mark the coordinate information of each position on the metasurface with random patches, and input the original image data into the image reconstruction network.

[0055] The marked metasurface location coordinates are used to establish a mapping relationship between the original image data and the metasurface structure, helping the reconstruction network understand the source of image degradation and thus perform restoration more accurately.

[0056] Step 4: Reconstruct the hyperspectral image using a model cascaded with the Dense Residual Channel Affinity Module (DRCA) and the Spatial Information Guided Propagation Module (SGP).

[0057] Step 5: Reconstruct hyperspectral polarization images of four polarization directions (0°, 45°, 90°, 135°) using the PTCS algorithm combined with the Pz-Unet network.

[0058] Step 6: Transform the reconstructed image to the Fourier domain, and output a high-quality restored image by optimizing the adversarial network and constraining the peak signal-to-noise ratio (PSNR) loss function.

[0059] Furthermore, this embodiment also provides an image reconstruction device based on metasurface imaging, which can be used to execute the above-described image reconstruction method based on metasurface imaging. Specifically, the image reconstruction device based on metasurface imaging includes: a metasurface image acquisition module, used to acquire a multispectral polarization image using a metasurface imaging device; an image restoration and reconstruction module, used to input the multispectral polarization image and the metasurface position information into a spectral reconstruction module, and output a reconstructed hyperspectral image; and inputting the hyperspectral image into a polarization reconstruction module, whereby the polarization reconstruction module uses a polarization total variation reconstruction sensing algorithm to obtain a polarization image of the target polarization direction.

[0060] The specific details of each module or unit in the above-mentioned image reconstruction device based on metasurface imaging have been described in detail in the corresponding image reconstruction method based on metasurface imaging, so they will not be repeated here.

[0061] This application also provides an electronic device. Figure 6 A schematic diagram of the structure of an electronic device suitable for implementing embodiments of the present disclosure is shown. Figure 6 The electronic device 600 shown is merely an example and should not be construed as limiting the functionality and scope of use of the embodiments disclosed herein.

[0062] like Figure 6As shown, the electronic device 600 includes a central processing unit (CPU) 601, which can perform various appropriate actions and processes based on a program stored in a read-only memory (ROM) 602 or a program loaded from a storage section 608 into a random access memory (RAM) 603. The RAM 603 also stores various programs and data required for system operation. The CPU 601, ROM 602, and RAM 603 are interconnected via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0063] The following components are connected to I / O interface 605: an input section 606 including a keyboard, mouse, etc.; an output section 607 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 608 including a hard disk, etc.; and a communication section 609 including a network interface card such as a LAN card, modem, etc. The communication section 609 performs communication processing via a network such as the Internet. A drive 610 is also connected to I / O interface 605 as needed. A removable medium 611, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on drive 610 as needed so that computer programs read from it can be installed into storage section 608 as needed.

[0064] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable storage medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 609, and / or installed from removable medium 611. When the computer program is executed by central processing unit (CPU) 601, it performs the functions defined in the embodiments of this application.

[0065] For example, when the computer program is executed by the central processing unit (CPU) 601, it can perform the following: acquire a multispectral polarization image using a metasurface imaging device; input the multispectral polarization image into a spectral reconstruction module and output a reconstructed hyperspectral image; input the hyperspectral image into a polarization image reconstruction module, which uses a polarization total variation reconstruction sensing algorithm to obtain a polarization image of the target polarization direction.

[0066] It should be noted that the computer-readable medium disclosed herein may be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium may be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media can also be any computer-readable medium other than computer-readable storage media, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.

[0067] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0068] The units described in the embodiments of this disclosure can be implemented in software or hardware, and the described units can also be located in a processor. The names of these units do not necessarily limit the unit itself.

[0069] In another aspect, this application also provides a computer-readable medium, which may be included in the electronic device described in the above embodiments; or it may exist independently and not assembled into the electronic device. The computer-readable medium carries one or more programs, which include instructions that, when executed by the electronic device, cause the electronic device to perform the methods described in the above embodiments.

[0070] It should be noted that although several modules or units for the device used to perform actions have been mentioned in the detailed description above, this division is not mandatory. In fact, according to the embodiments of this application, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided and embodied by multiple modules or units.

[0071] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. An image reconstruction method based on metasurface imaging, characterized in that, include: Acquiring multispectral polarization images using a metasurface imaging device; The multispectral polarization image is input into the spectral reconstruction module, which outputs the reconstructed hyperspectral image. The hyperspectral image is input into the polarization reconstruction module, which uses a polarization total variation reconstruction sensing algorithm to obtain a polarization image of the target polarization direction.

2. The image reconstruction method based on metasurface imaging according to claim 1, characterized in that, The method of acquiring multispectral polarization images using a metasurface imaging device includes: By traversing different crystal structures and obtaining their transmittance response curves, crystal structures suitable for multispectral polarization modulation requirements are selected. Randomly arrange the surface sources of the crystal structure to obtain the spectral response data of different surface source structures under multiple polarizations, and optimize the surface source structure through deep learning algorithms; Based on the optimized surface source structure, a metasurface imaging device is formed through processing and detector pixel patching. The metasurface imaging device is then used to acquire multispectral polarization images.

3. The image reconstruction method based on metasurface imaging according to claim 2, characterized in that, The process of traversing different crystal structures and obtaining their transmittance response curves to screen out crystal structures suitable for multispectral polarization modulation includes: Traversing the wafer structure, which includes wafer units of different geometries, sizes, and materials; Obtain the transmittance response curves of each crystal structure in the target wavelength band; Based on the bandwidth, peak transmittance, and polarization discrimination of the transmittance curve, the target wafer structure adapted to the multispectral polarization modulation requirements is determined.

4. The image reconstruction method based on metasurface imaging according to claim 3, characterized in that, The process of randomly arranging surface sources in a wafer structure to obtain spectral response data of different surface source structures under multiple polarizations, and optimizing the surface source structure using a deep learning algorithm, includes: Randomly arrange the surface sources of the target wafer structure to generate multiple sets of surface source structure schemes; An optical testing platform was built to obtain spectral response datasets of different surface source structures under polarization at 0°, 45°, 90°, and 135°. Deep learning algorithms are used to train the dataset to optimize the spectral polarization modulation performance of the surface source structure.

5. The image reconstruction method based on metasurface imaging according to claim 1, characterized in that, The step of inputting the multispectral polarization image into the spectral reconstruction module and outputting the reconstructed hyperspectral image includes: The spectral reconstruction module includes an initial feature extraction module, a feature iteration optimization module, and a first reconstruction module; The initial feature extraction module includes two branches. The first branch upsamples the input multispectral polarization image and uses bicubic interpolation to enlarge the image spatial size to the target size to extract shallow features. The second branch uses three parallel convolutional layers to extract features respectively, and adds the features from the two branches element by element to obtain the initial feature map; The feature iteration optimization module consists of multiple dense residual channel affinity (DRCA) modules cascaded with the spatial information guided propagation (SGP) module; the DRCA module includes an encoder / decoder and a channel affinity submodule. Multi-scale features are extracted by encoder and decoder, and channel weights are learned by channel affinity submodule. The features are then weighted to obtain deeply optimized features. The SGP module simulates the imaging degradation process to generate residual information, and then uses a deformable adaptive fusion submodule to adaptively fuse the residual information with the main features to refine the spatial features. The first reconstruction module adds and fuses the initial feature map with the depth-optimized features through global skip connections, and then uses a convolutional layer to adjust the number of channels of the fused features, outputting the final high-resolution hyperspectral image.

6. The image reconstruction method based on metasurface imaging according to claim 1, characterized in that, The polarization reconstruction module uses a polarization total variation reconstruction sensing algorithm to obtain a polarization image of the target polarization direction, including: The polarization reconstruction module employs the Polarization Total Variation Reconstruction Sensing (PTCS) algorithm to reconstruct the hyperspectral image; the PTCS algorithm is used to optimize the following formula: Where y is the hyperspectral image, x is the reconstructed image, and A is the measurement matrix. The weights are the total variation constraint coefficients. For the weights of the interspectral constraint coefficients, For nuclear norm; The hyperspectral image optimized by the PTCS algorithm is input into the trained Pz-Unet network to obtain the polarization image of the target polarization direction. The Pz-Unet network model is a U-Net encoder-decoder network based on Res2Net-SE and context self-attention.

7. The image reconstruction method based on metasurface imaging according to claim 1, characterized in that, After obtaining the polarization image of the target polarization direction, the following steps are also included: The polarization image is converted to the Fourier domain, and then optimized using an adversarial network and a peak signal-to-noise ratio loss function constraint, resulting in an optimized polarization image.

8. The image reconstruction method based on metasurface imaging according to claim 4, characterized in that, The nanostructure units of the surface source structure are arranged with superpixels as the basic units; wherein the superpixel is composed of a 3×3 array of 9 preferred structural units selected by residual isotropic power. The superpixel is periodically replicated and expanded within the imaging target area to destroy the spatial coherence of the incident light and suppress speckle noise and parasitic interference fringes.

9. An image reconstruction device based on metasurface imaging, characterized in that, include: Metasurface image acquisition module, used to acquire multispectral polarization images using a metasurface imaging device; The image restoration and reconstruction module is used to input the multispectral polarization image into the spectral reconstruction module and output the reconstructed hyperspectral image; the hyperspectral image is input into the polarization reconstruction module, and the polarization reconstruction module uses a polarization total variation reconstruction sensing algorithm to obtain a polarization image of the target polarization direction.

10. An electronic device, characterized in that, The device includes a processor and a memory, the memory storing one or more computer programs, the one or more computer programs including instructions that, when executed by the electronic device, cause the electronic device to perform the image reconstruction method based on metasurface imaging as described in any one of claims 1-8.

Images