A meta-imaging system and method based on a complementary perception brain-like chip

Abstract

The present disclosure relates to the technical field of computational imaging, and proposes a meta-imaging system and method based on a complementary perception brain-like chip. The system comprises a complementary optical acquisition module that collects signal light of a target sample to form a common intermediate image containing at least two spectral components; a complementary light field encoding module that performs spatial and angular joint encoding of the common intermediate image under multiple joint encoding states to form a complementary encoded light field; a complementary spectral distribution module that distributes the complementary encoded light field to at least two spectral channels with different spectral responses, and each channel of the complementary encoded light signals shares a corresponding joint encoding state; a brain-like complementary perception module that performs photoelectric acquisition of each channel of the complementary encoded light signals through a cognitive-oriented channel and a motion-oriented channel; and a multi-dimensional complementary information expression and reconstruction module that establishes a joint information expression relationship of space, angle, time and spectrum, and performs dynamic compensation and meta-imaging joint reconstruction. The system can improve the stability and imaging quality of meta-imaging reconstruction in a dynamic scene.

Description

Technical Field

[0001] This disclosure relates to the field of computational imaging technology, and in particular to a meta-imaging system and method based on a complementary sensing neuromorphic chip. Background Technology

[0002] In scenarios such as live-cell observation, deep tissue imaging, and high-speed biological process imaging, imaging systems not only need to possess high spatial resolution and a large field of view, but also need to meet performance requirements such as high-speed dynamic sensing, high sensitivity, low phototoxicity, and dynamic aberration correction. However, existing meta-imaging systems typically perform sampling and computational reconstruction based on a single spectral channel and single intensity information. Although they can recover the static three-dimensional morphology of the target scene, they are difficult to fully acquire the dynamic change information of the target scene. In scenarios such as rapid cell movement, rapid changes in fluorescence intensity, enhanced scattering in deep tissues, and long-term live-cell observation, problems such as motion artifacts, dynamic blurring, local overexposure, low signal-to-noise ratio, and unstable reconstruction are easily generated.

[0003] Furthermore, existing computational imaging methods typically design each stage—information acquisition, coding and modulation, image perception, and computational reconstruction—independently, lacking an integrated architecture for the collaborative acquisition and unified representation of multi-source information, and failing to fully utilize the synergistic relationships between different information dimensions. Therefore, there is an urgent need to propose a novel meta-imaging system that achieves the collaborative acquisition and unified representation of multi-source information during information acquisition, coding and modulation, image perception, and computational reconstruction, thereby improving the stability and imaging quality of meta-imaging reconstruction in dynamic scenes. Summary of the Invention

[0004] In view of this, this disclosure proposes a meta-imaging system and method based on a complementary sensing neuromorphic chip, which at least partially solves the problems of temporal misalignment and spatial mismatch of sampling data corresponding to different coding states during dynamic imaging, lack of unified correspondence between multispectral information and data from different sensing pathways, and difficulty in adapting the coding process to the dynamic changes of the target sample, thereby improving the stability and imaging quality of meta-imaging reconstruction in dynamic scenes.

[0005] According to one aspect of this disclosure, a meta-imaging system based on a complementary sensing neuromorphic chip is provided, the system comprising:

[0006] A complementary optical acquisition module is used to collect the signal light of the target sample and form a common intermediate image containing at least two spectral components;

[0007] The complementary light field coding module is used to perform spatial and angular joint coding of the common intermediate image under multiple joint coding states based on a unified time reference, so as to form a complementary coded light field.

[0008] A complementary spectral allocation module is used to allocate the complementary coded light field to at least two spectral channels with different spectral responses to obtain at least two complementary coded light signals, wherein the complementary coded light signals allocated under the same joint coding state share the joint coding state;

[0009] The brain-like complementary sensing module includes a complementary sensing brain-like chip with a cognitive guidance pathway and a motion guidance pathway, used to perform photoelectric acquisition on each of the complementary coded optical signals to obtain complementary optical field sampling data, wherein the cognitive guidance pathway outputs spatial structure data and the motion guidance pathway outputs spatiotemporal change data;

[0010] The multidimensional complementary information expression and reconstruction module is used to associate the spatial structure data and the spatiotemporal variation data with the joint encoding state and the spectral channel at the time of acquisition, respectively, to establish the joint information expression relationship of space, angle, time and spectrum, to dynamically compensate the complementary light field sampling data according to the spatiotemporal variation data, and to perform meta-imaging joint reconstruction with the spatial structure data as a constraint to obtain the meta-imaging result.

[0011] The complementary optical field coding module adaptively adjusts the subsequent joint coding state based on the spatiotemporal change data.

[0012] In one possible implementation, the complementary optical acquisition module includes a light-collecting objective, an imaging lens group, and a multi-band filter assembly;

[0013] The light-collecting objective lens is used to collect signal light from the same observation area of ​​the target sample through the same acquisition aperture, and the signal light contains the at least two spectral components;

[0014] The multi-band filter assembly is disposed between the light-collecting objective and the imaging lens group, and has at least two transmission bands corresponding to the at least two spectral components, for simultaneously filtering out the spectral components from the signal light.

[0015] The imaging lens group is used to make the signal light of each spectral component passing through the multi-band filter component propagate along the same imaging optical path and form the common intermediate image on the same intermediate image plane, so that the complementary optical field encoding module can perform spatial and angular joint encoding on the common intermediate image.

[0016] In one possible implementation, the complementary optical field encoding module includes a scanning mirror, a movable encoding microlens array, an encoding controller, and a driving component, wherein the driving component is connected to the scanning mirror and the movable encoding microlens array, respectively.

[0017] The joint encoding state includes the angular deflection state of the scanning mirror and the spatial displacement state of the movable encoding microlens array;

[0018] The encoding controller is used to control the scanning mirror and the movable encoding microlens array to form corresponding joint encoding states based on the unified time reference, and to determine the duration of each joint encoding state.

[0019] The encoding controller is also configured to receive the spatiotemporal change data during the period when the sampling time interval is within the same joint encoding state, determine the motion amplitude of the target sample based on the spatiotemporal change data, and adjust the combination, execution order, or holding duration of subsequent joint encoding states based on the motion amplitude.

[0020] In one possible implementation, the complementary spectral allocation module includes a spectral splitting component and a spectral filtering component and a relay imaging component respectively disposed in each of the spectral channels;

[0021] The beam splitter is used to distribute the complementary coded light field to each of the spectral channels, and at least two of the spectral filtering components have different transmission band ranges or relative transmittance between bands.

[0022] Each complementary coded optical signal corresponds to the same observation area of ​​the target sample, and is imaged onto a non-overlapping photosensitive area on the complementary sensing neuromorphic chip via the corresponding relay imaging component. Each photosensitive area is associated with a corresponding spectral channel.

[0023] For each photosensitive area, the cognitive guidance pathway outputs spatial structure data of the corresponding spectral channel, and the motion guidance pathway outputs spatiotemporal variation data of the corresponding spectral channel.

[0024] In one possible implementation, the cognitive guidance pathway and the motion guidance pathway are synchronously acquired based on the unified time reference;

[0025] The cognitive guidance pathway outputs the spatial structure data at a first sampling frequency, and the spatial structure data is intensity frame data characterizing the incident light intensity and its spatial distribution.

[0026] The motion guidance path outputs the spatiotemporal variation data at a second sampling frequency, which is higher than the first sampling frequency. The spatiotemporal variation data includes spatial difference data and temporal difference data.

[0027] The brain-like complementary perception module is also used to record the exposure time interval for each frame of spatial structure data based on the unified time reference, record the sampling time interval for each group of spatiotemporal change data, and associate multiple groups of spatiotemporal change data whose sampling time intervals are located in the same exposure time interval with the corresponding spatial structure data.

[0028] In one possible implementation, associating the spatial structure data and the spatiotemporal variation data with the joint encoding state and corresponding spectral channels at the time of acquisition, respectively, to establish a joint information representation relationship of space, angle, time, and spectrum, includes:

[0029] Based on the unified time reference, the corresponding sampling time, joint coding status identifier, spectral channel identifier, and sensing path identifier are associated with the complementary light field sampling data.

[0030] Based on the angular deflection state of the scanning mirror, the spatial displacement state of the movable coded microlens array, and the pre-calibrated mapping relationship between the microlens unit and the imaging pixel, the spatial sampling position and observation angle corresponding to the complementary light field sampling data are determined.

[0031] Based on the pre-defined coordinate transformation relationship between different spectral channels and the spatial mapping relationship between the cognitive guidance path and the motion guidance path, the spatial sampling position is transformed to a unified object coordinate system;

[0032] Based on the sampling time and the converted spatial sampling location, the spatial structure data and the spatiotemporal change data are uniformly associated with the corresponding observation angle, joint coding state, spectral channel and sensing path to establish the joint information expression relationship.

[0033] In one possible implementation, the dynamic compensation includes:

[0034] Based on the spatiotemporal change data collected during each joint coding state, and combined with the joint information expression relationship, the local motion field and dynamic perturbation parameters of the imaging field of view of the target sample are estimated, wherein the local motion field characterizes the motion difference between different local regions, and the dynamic perturbation parameters characterize the overall offset of the imaging field of view.

[0035] Based on the local motion field, the dynamic disturbance parameters, and the time interval between each sampling time and the predetermined reference time, determine the motion transformation relationship of the complementary optical field sampling data corresponding to each joint coding state to the predetermined reference time;

[0036] By combining the spatial sampling position and observation angle corresponding to each joint coding state, and while retaining the observation angle identifier, spectral channel identifier and joint coding state identifier corresponding to each sampling data, the complementary light field sampling data corresponding to different joint coding states are time-aligned and spatially registered according to the motion transformation relationship to obtain the compensated light field sampling data.

[0037] In one possible implementation, the meta-imaging joint reconstruction includes:

[0038] Based on the spatial structure data in the compensated light field sampling data and the corresponding joint coding state and spectral channel, a spatial and angular joint reconstruction model is established. The spatial and angular joint reconstruction model includes the spatial and angular sampling relationship corresponding to each joint coding state and the spectral response of each spectral channel.

[0039] The fusion weights are determined based on the signal-to-noise ratio, exposure saturation, and registration residuals after dynamic compensation in each local region of data corresponding to different joint coding states and different spectral channels.

[0040] Using the fusion weights, the differences between the intensity data predicted by the joint spatial and angular reconstruction model and the spatial structure data in the compensated light field sampling data are weighted to form a weighted data consistency term;

[0041] Based on the edge or texture features provided by the spatial structure data, a structural constraint term is constructed. The objective function composed of the weighted data consistency term and the structural constraint term is jointly solved to recover the light field information of the target sample and generate the meta-imaging result.

[0042] In one possible implementation, the multidimensional complementary information representation and reconstruction module is further configured to perform digital adaptive optics correction on the recovered optical field information, wherein the digital adaptive optics correction includes:

[0043] Based on the data corresponding to different observation angles in the light field information, the wavefront information at different field of view positions is estimated;

[0044] Based on the wavefront information and the spectral response of the corresponding spectral channel, the spatial variation point spread function at different field positions is determined, and the corresponding local optical model is established.

[0045] Based on the spatial structure data, the spatiotemporal variation data, and the local signal-to-noise ratio at each field of view, the corresponding local correction parameters are determined, including regularization coefficients for local deconvolution.

[0046] Based on the local optical model and the local correction parameters, local deconvolution is performed at each field of view to compensate for optical aberrations, resulting in a corrected meta-imaging result.

[0047] According to another aspect of this disclosure, a meta-imaging method based on a complementary sensing neuromorphic chip is provided, the method comprising:

[0048] The signal light from the target sample is collected to form a common intermediate image containing at least two spectral components;

[0049] Based on a unified time reference, the common intermediate image is jointly encoded in space and angle under multiple joint encoding states to form a complementary encoded light field.

[0050] The complementary coded optical field is assigned to at least two spectral channels with different spectral responses to obtain at least two complementary coded optical signals, wherein the complementary coded optical signals assigned under the same joint coding state share the joint coding state;

[0051] Using a complementary sensory neuromorphic chip with cognitive and motor guidance pathways, photoelectric acquisition is performed on the complementary coded optical signals of each pathway to obtain complementary optical field sampling data. The cognitive guidance pathway outputs spatial structure data, and the motor guidance pathway outputs spatiotemporal variation data.

[0052] The spatial structure data and the spatiotemporal variation data are respectively associated with the joint encoding state and the corresponding spectral channel at the time of acquisition, and a joint information expression relationship of space, angle, time and spectrum is established. The complementary light field sampling data is dynamically compensated according to the spatiotemporal variation data, and the meta-imaging joint reconstruction is performed with the spatial structure data as a constraint to obtain the meta-imaging result.

[0053] The joint coding state is adaptively adjusted based on the spatiotemporal change data.

[0054] In one possible implementation, the device for collecting the signal light of the target sample includes a light-collecting objective, an imaging lens group, and a multi-band filter assembly.

[0055] The light-collecting objective lens is used to collect signal light from the same observation area of ​​the target sample through the same acquisition aperture, and the signal light contains the at least two spectral components;

[0056] The multi-band filter assembly is disposed between the light-collecting objective and the imaging lens group, and has at least two transmission bands corresponding to the at least two spectral components, for simultaneously filtering out the spectral components from the signal light.

[0057] The imaging lens group is used to make the signal light of each spectral component passing through the multi-band filter component propagate along the same imaging optical path and form the common intermediate image on the same intermediate image plane, so that the complementary optical field encoding module can perform spatial and angular joint encoding on the common intermediate image.

[0058] In one possible implementation, the device for joint spatial and angular encoding of the common intermediate image under multiple joint encoding states based on a unified time reference includes a scanning mirror, a movable encoding microlens array, an encoding controller, and a driving component, wherein the driving component is connected to the scanning mirror and the movable encoding microlens array respectively.

[0059] The joint encoding state includes the angular deflection state of the scanning mirror and the spatial displacement state of the movable encoding microlens array;

[0060] The encoding controller is used to control the scanning mirror and the movable encoding microlens array to form corresponding joint encoding states based on the unified time reference, and to determine the duration of each joint encoding state.

[0061] The encoding controller is also configured to receive the spatiotemporal change data during the period when the sampling time interval is within the same joint encoding state, determine the motion amplitude of the target sample based on the spatiotemporal change data, and adjust the combination, execution order, or holding duration of subsequent joint encoding states based on the motion amplitude.

[0062] In one possible implementation, the device for distributing the complementary coded light field to at least two spectral channels with different spectral responses includes a beam-splitting component and a spectral filtering component and a relay imaging component respectively disposed in each of the spectral channels.

[0063] The beam splitter is used to distribute the complementary coded light field to each of the spectral channels, and at least two of the spectral filtering components have different transmission band ranges or relative transmittance between bands.

[0064] Each complementary coded optical signal corresponds to the same observation area of ​​the target sample, and is imaged onto a non-overlapping photosensitive area on the complementary sensing neuromorphic chip via the corresponding relay imaging component. Each photosensitive area is associated with a corresponding spectral channel.

[0065] For each photosensitive area, the cognitive guidance pathway outputs spatial structure data of the corresponding spectral channel, and the motion guidance pathway outputs spatiotemporal variation data of the corresponding spectral channel.

[0066] In one possible implementation, the cognitive guidance pathway and the motion guidance pathway are synchronously acquired based on the unified time reference;

[0067] The cognitive guidance pathway outputs the spatial structure data at a first sampling frequency, and the spatial structure data is intensity frame data characterizing the incident light intensity and its spatial distribution.

[0068] The motion guidance path outputs the spatiotemporal variation data at a second sampling frequency, which is higher than the first sampling frequency. The spatiotemporal variation data includes spatial difference data and temporal difference data.

[0069] The brain-like complementary perception module is also used to record the exposure time interval for each frame of spatial structure data based on the unified time reference, record the sampling time interval for each group of spatiotemporal change data, and associate multiple groups of spatiotemporal change data whose sampling time intervals are located in the same exposure time interval with the corresponding spatial structure data.

[0070] In one possible implementation, associating the spatial structure data and the spatiotemporal variation data with the joint encoding state and corresponding spectral channels at the time of acquisition, respectively, to establish a joint information representation relationship of space, angle, time, and spectrum, includes:

[0071] Based on the unified time reference, the corresponding sampling time, joint coding status identifier, spectral channel identifier, and sensing path identifier are associated with the complementary light field sampling data.

[0072] Based on the angular deflection state of the scanning mirror, the spatial displacement state of the movable coded microlens array, and the pre-calibrated mapping relationship between the microlens unit and the imaging pixel, the spatial sampling position and observation angle corresponding to the complementary light field sampling data are determined.

[0073] Based on the pre-defined coordinate transformation relationship between different spectral channels and the spatial mapping relationship between the cognitive guidance path and the motion guidance path, the spatial sampling position is transformed to a unified object coordinate system;

[0074] Based on the sampling time and the converted spatial sampling location, the spatial structure data and the spatiotemporal change data are uniformly associated with the corresponding observation angle, joint coding state, spectral channel and sensing path to establish the joint information expression relationship.

[0075] In one possible implementation, the dynamic compensation includes:

[0076] Based on the spatiotemporal change data collected during each joint coding state, and combined with the joint information expression relationship, the local motion field and dynamic perturbation parameters of the imaging field of view of the target sample are estimated, wherein the local motion field characterizes the motion difference between different local regions, and the dynamic perturbation parameters characterize the overall offset of the imaging field of view.

[0077] Based on the local motion field, the dynamic disturbance parameters, and the time interval between each sampling time and the predetermined reference time, determine the motion transformation relationship of the complementary optical field sampling data corresponding to each joint coding state to the predetermined reference time;

[0078] By combining the spatial sampling position and observation angle corresponding to each joint coding state, and while retaining the observation angle identifier, spectral channel identifier and joint coding state identifier corresponding to each sampling data, the complementary light field sampling data corresponding to different joint coding states are time-aligned and spatially registered according to the motion transformation relationship to obtain the compensated light field sampling data.

[0079] In one possible implementation, the meta-imaging joint reconstruction includes:

[0080] Based on the spatial structure data in the compensated light field sampling data and the corresponding joint coding state and spectral channel, a spatial and angular joint reconstruction model is established. The spatial and angular joint reconstruction model includes the spatial and angular sampling relationship corresponding to each joint coding state and the spectral response of each spectral channel.

[0081] The fusion weights are determined based on the signal-to-noise ratio, exposure saturation, and registration residuals after dynamic compensation in each local region of data corresponding to different joint coding states and different spectral channels.

[0082] Using the fusion weights, the differences between the intensity data predicted by the spatial and angular joint reconstruction model and the spatial structure data in the compensated light field sampling data are weighted to form a weighted data consistency term;

[0083] Based on the edge or texture features provided by the spatial structure data, a structural constraint term is constructed. The objective function composed of the weighted data consistency term and the structural constraint term is jointly solved to recover the light field information of the target sample and generate the meta-imaging result.

[0084] In one possible implementation, the method further includes performing digital adaptive optics correction on the recovered light field information, the digital adaptive optics correction comprising:

[0085] Based on the data corresponding to different observation angles in the light field information, the wavefront information at different field of view positions is estimated;

[0086] Based on the wavefront information and the spectral response of the corresponding spectral channel, the spatial variation point spread function at different field positions is determined, and the corresponding local optical model is established.

[0087] Based on the spatial structure data, the spatiotemporal variation data, and the local signal-to-noise ratio at each field of view, the corresponding local correction parameters are determined, including regularization coefficients for local deconvolution.

[0088] Based on the local optical model and the local correction parameters, local deconvolution is performed at each field of view to compensate for optical aberrations, resulting in a corrected meta-imaging result.

[0089] According to another aspect of this disclosure, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the above-described method.

[0090] According to another aspect of this disclosure, a non-volatile computer-readable storage medium is provided, on which a computer program is stored, which, when executed by a processor, implements the steps of the above-described method.

[0091] According to another aspect of this disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps of the above-described method.

[0092] In this embodiment, by collecting the signal light of the target sample, a common intermediate image containing at least two spectral components is formed, allowing the signals of at least two spectral components to converge and synchronously enter the subsequent encoding and acquisition process in the same intermediate image. By performing spatial and angular joint encoding on the common intermediate image under multiple joint encoding states based on a unified time reference, the encoded data obtained at different times maintain a unified correspondence in the time dimension. By allocating the complementary encoded light field to at least two spectral channels with different spectral responses, and allowing each complementary encoded light signal allocated under the same joint encoding state to share the joint encoding state, the multispectral signals are allocated to different spectral channels without destroying the correspondence between the spectrum and the encoding state. The spatial structure data is output through the cognitive guidance pathway, and the spatiotemporal change data is output through the motion guidance pathway, allowing the two types of data to be obtained in parallel by the same complementary sensory neuromorphic chip. By establishing a joint information expression relationship of space, angle, time, and spectrum, the multidimensional information maintains a correspondence under the same data organization framework. Based on this, since the spatiotemporal variation data is output by the motion-guided pathway and the spatial structure data is output by the cognitive-guided pathway, the inputs required for dynamic compensation and meta-imaging joint reconstruction both come from the same acquisition process. Because dynamic compensation is performed on the complementary light field sampling data based on the spatiotemporal variation data, the sampling data obtained under different joint encoding states can be corrected based on the dynamic information obtained during acquisition before meta-imaging joint reconstruction, reducing the deviation between data under different joint encoding states caused by different acquisition times. Meta-imaging joint reconstruction is constrained by the spatial structure data and shares the data associated with the joint information expression relationship with dynamic compensation. Furthermore, because the complementary light field encoding module adaptively adjusts the subsequent joint encoding state based on the spatiotemporal variation data, the subsequent joint encoding state can be updated based on the acquired spatiotemporal variation data. This improves the stability and imaging quality of meta-imaging reconstruction in dynamic scenes.

[0093] Other features and aspects of this disclosure will become clear from the following detailed description of exemplary embodiments with reference to the accompanying drawings. Attached Figure Description

[0094] The accompanying drawings, which are included in and form part of this specification, illustrate exemplary embodiments, features, and aspects of this disclosure together with the specification and serve to explain the principles of this disclosure.

[0095] Figure 1 A block diagram of a meta-imaging system based on a complementary sensing neuromorphic chip according to an embodiment of the present disclosure is shown.

[0096] Figure 2 A schematic diagram of the optical path system of a meta-imaging system based on a complementary sensing neuromorphic chip according to an embodiment of the present disclosure is shown.

[0097] Figure 3 A schematic diagram of the data processing flow of a meta-imaging system based on a complementary sensing neuromorphic chip according to an embodiment of the present disclosure is shown.

[0098] Figure 4 A flowchart illustrating a meta-imaging method based on a complementary sensing neuromorphic chip according to an embodiment of the present disclosure is shown.

[0099] Figure 5 This is a block diagram illustrating a meta-imaging device based on a complementary sensing neuromorphic chip according to an exemplary embodiment. Detailed Implementation

[0100] Various exemplary embodiments, features, and aspects of this disclosure will now be described in detail with reference to the accompanying drawings. The same reference numerals in the drawings denote elements that have the same or similar functions. Although various aspects of the embodiments are shown in the drawings, they are not necessarily drawn to scale unless specifically indicated otherwise.

[0101] As used herein, the terms “comprising,” “including,” “having,” or variations thereof are open-ended and include one or more of the stated features, integrals, elements, steps, components, or functions, but do not exclude the presence or addition of one or more other features, integrals, elements, steps, components, functions, or groups thereof.

[0102] When an element is referred to as “connected,” “coupled,” “responding,” or a variation thereof relative to another element, it may be directly connected, coupled, or responding to another element, or there may be an intermediate element present.

[0103] Although the terms first, second, third, etc., may be used herein to describe various elements / operations, these elements / operations should not be limited by these terms. These terms are only used to distinguish one element / operation from another. Therefore, without departing from the teachings of the inventive concept, a first element / operation in some embodiments may be referred to as a second element / operation in other embodiments.

[0104] The term “exemplary” as used herein means “serving as an example, embodiment, or illustration.” Any embodiment illustrated herein as “exemplary” is not necessarily to be construed as superior to or better than other embodiments.

[0105] Furthermore, to better illustrate this disclosure, numerous specific details are set forth in the following detailed description. Those skilled in the art will understand that this disclosure can be practiced without certain specific details. In some instances, methods, means, components, and circuits well known to those skilled in the art have not been described in detail in order to highlight the main points of this disclosure.

[0106] It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, data stored, data displayed, etc.) and signals involved in this application are all authorized by the user or fully authorized by all parties, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant regions.

[0107] When meta-imaging systems are applied in scenarios such as high-speed live-cell observation, dynamic fluorescence microscopy, deep tissue imaging, and long-term live-cell observation, the imaging objects are typically in constant flux. Cells migrate and deform at micrometer-scale speeds, the intensity of fluorescent markers fluctuates rapidly with physiological processes, the scattering characteristics of deep tissues change with imaging depth and observation time, and stage vibration and fluid disturbances can also cause overall field-of-view shifts. Correlation meta-imaging systems typically require time-division acquisition of data under different scanning states during the acquisition process. A single complete acquisition consists of data obtained at multiple times, with the acquisition cycle corresponding to the aforementioned dynamic timescales. This results in information belonging to the same location within the target scene being acquired at different times, easily leading to temporal and spatial misalignments between data from different moments.

[0108] Furthermore, related meta-imaging systems typically perform sampling and computational reconstruction based on a single spectral channel and single intensity information. On the one hand, a single spectral channel can only obtain signals from a single spectral range, and the response differences of the target scene in different spectral ranges are not utilized, resulting in a lack of spectral dimension information during the acquisition stage. On the other hand, relying solely on a single intensity image for reconstruction lacks specialized means of perceiving dynamic information such as motion, deformation, and brightness fluctuations in the scene. The acquisition of dynamic information lags behind the imaging process itself, making it difficult to match the time scale of high-speed dynamic processes. Simultaneously, the independent design of each stage—information acquisition, encoding and modulation, image perception, and computational reconstruction—leads to a lack of unified correspondence between data from different stages, inconsistent input parameters for each processing step, and a tendency to introduce biases during reconstruction. In addition, inhomogeneous refractive index and enhanced scattering within the sample can produce varying degrees of aberrations in different imaging regions, making it difficult to take into account the actual aberration states of each region when using uniform parameters for overall correction.

[0109] The combination of these factors results in inconsistent imaging data in dynamic scenes, spatial misalignment, and difficulty in achieving spatial resolution, completeness of the dynamic process, and quality balance between different imaging regions. This makes it impossible to meet the requirements of high-speed live cell observation, dynamic fluorescence microscopy, and deep tissue imaging for imaging quality and observation continuity.

[0110] This disclosure provides a meta-imaging system and method based on a complementary sensing neuromorphic chip. The system collects signal light from a target sample using a complementary optical acquisition module to form a common intermediate image. A complementary light field encoding module performs spatial and angular joint encoding on the common intermediate image under multiple joint encoding states. A complementary spectral allocation module distributes the encoded light field to at least two spectral channels with different spectral responses. A dual-path acquisition module of the neuromorphic complementary sensing module acquires spatial structure data and spatiotemporal variation data. Finally, a multi-dimensional complementary information expression and reconstruction module completes dynamic compensation and meta-imaging joint reconstruction.

[0111] Figure 1 A block diagram illustrating the modular structure of a meta-imaging system based on a complementary sensing neuromorphic chip according to an embodiment of the present disclosure is shown. Figure 1 As shown, the system includes:

[0112] The complementary optical acquisition module 101 is used to collect the signal light of the target sample and form a common intermediate image containing at least two spectral components.

[0113] Signal light can represent the light generated by a target sample under illumination and carrying its optical information; for example, it can include fluorescence, reflected light, or scattered light.

[0114] Spectral components can represent the light components in different wavelength ranges of signal light, such as fluorescence, reflected light, or scattered light in different wavelength ranges.

[0115] A common intermediate image can represent an optical image formed by each spectral component along a common imaging optical path on the same intermediate image plane; a common intermediate image is an optical image during the optical propagation stage, and does not represent multiple digital images that have been separated.

[0116] As an example, in fluorescence microscopy, the target sample emits two types of fluorescence in different wavelength ranges at the same time. The collected signal light contains two spectral components, and the signals of the two spectral components merge in the same image to form a common intermediate image.

[0117] The complementary light field coding module 102 is used to perform spatial and angular joint coding on the common intermediate image under multiple joint coding states based on a unified time reference, so as to form a complementary coded light field.

[0118] A unified time base can represent the clock reference shared by the system's encoding and acquisition processes.

[0119] The joint encoding state represents the optical state of the complementary optical field encoding module during the encoding process. Under different joint encoding states, the complementary optical field encoding module has different spatial and angular sampling relationships for the common intermediate image, thus obtaining different spatial and angular sampling data in different joint encoding states.

[0120] Spatial and angular joint encoding can represent the process of encoding an optical signal simultaneously according to two dimensions: spatial position and propagation direction. Each data obtained after encoding corresponds to a spatial position of the target sample and a propagation direction of the optical signal.

[0121] A complementary coded light field can represent the light field formed after the joint encoding of space and angle, and can carry both spatial position information and propagation direction information.

[0122] For example, under the control of a unified time reference, the complementary optical field coding module switches between multiple joint coding states over time; in different joint coding states, the spatial and angular sampling relationships used for the signal at the same spatial position in the common intermediate image are different, thereby obtaining the corresponding complementary coded optical fields at different times.

[0123] For specific methods of forming multiple joint coding states and performing spatial and angular joint coding, please refer to the possible implementation methods provided in this disclosure, which will not be elaborated here.

[0124] The complementary spectral allocation module 103 is used to allocate the complementary coded light field to at least two spectral channels with different spectral responses to obtain at least two complementary coded light signals, wherein the complementary coded light signals allocated under the same joint coding state share the joint coding state.

[0125] A spectral channel can represent a detection branch with a corresponding spectral response. Different spectral channels have different responses to different spectral components of incident light.

[0126] Complementary coded optical signals can represent the optical signals obtained after the complementary coded optical field is allocated to each of the spectral channels. Since the allocation occurs after spatial and angular joint coding, the complementary coded optical signals allocated under the same joint coding state share that joint coding state. That is, multispectral detection does not need to repeatedly establish coding and acquisition timing for each spectral channel, and the spectral dimension and the coding state dimension remain in correspondence.

[0127] For example, the complementary coded light field is allocated to two spectral channels with different spectral responses, one spectral channel responding to one spectral component and the other spectral channel responding to another spectral component, and the two complementary coded light signals are formed in their respective spectral channels; since the two complementary coded light signals are allocated from the complementary coded light field under the same joint coding state, they share the joint coding state.

[0128] The brain-like complementary sensing module 104 includes a complementary sensing brain-like chip with a cognitive guidance pathway and a motion guidance pathway, used to perform photoelectric acquisition on each of the complementary coded optical signals to obtain complementary light field sampling data, wherein the cognitive guidance pathway outputs spatial structure data and the motion guidance pathway outputs spatiotemporal change data.

[0129] Complementary sensory neuromorphic chips can represent neuromorphic visual sensing chips that integrate cognitive-guided pathways and motor-guided pathways.

[0130] The cognitive-guided pathway can refer to the pathway in a complementary sensory neuromorphic chip that performs frame-based intensity sensing of incident light; the motion-guided pathway can refer to the pathway in a complementary sensory neuromorphic chip that senses the spatiotemporal changes of incident light in a differential manner; the cognitive-guided pathway and the motion-guided pathway can also be collectively referred to as the sensory pathway.

[0131] Spatial structure data can represent data output by the cognitive guidance pathway that characterizes the spatial structure of the target sample.

[0132] Spatiotemporal variation data can represent data output by the motion-guided pathway that characterizes the dynamic changes of the target sample.

[0133] Complementary light field sampling data can represent sampling data obtained by the complementary sensing neuromorphic chip through photoelectric acquisition of each complementary coded light signal, including the spatial structure data and the spatiotemporal variation data.

[0134] For example, the complementary sensing neuromorphic chip performs photoelectric acquisition of the complementary coded light signals of each spectral channel. The cognitive guidance pathway performs frame-by-frame intensity sensing of the incident light and outputs spatial structure data. The motion guidance pathway senses the spatiotemporal changes of the incident light in a differential manner and outputs spatiotemporal change data. The data output by the two pathways together constitute the complementary light field sampling data of each spectral channel.

[0135] Specifically, the methods for photoelectric acquisition of the complementary coded optical signals through cognitive guidance and motion guidance can be found in the possible implementations provided in this disclosure, and will not be elaborated here.

[0136] The multidimensional complementary information expression and reconstruction module 105 is used to associate the spatial structure data and the spatiotemporal variation data with the joint encoding state and the spectral channel at the time of acquisition, respectively, to establish the joint information expression relationship of space, angle, time and spectrum, to dynamically compensate the complementary light field sampling data according to the spatiotemporal variation data, and to perform meta-imaging joint reconstruction with the spatial structure data as a constraint to obtain the meta-imaging result.

[0137] The complementary optical field coding module adaptively adjusts the subsequent joint coding state based on the spatiotemporal change data.

[0138] Joint information expression relationships can represent a multidimensional information organization structure that establishes corresponding relationships between spatial, angular, temporal, and spectral dimensions. Each set of joint information corresponds to a multidimensional information description of the target sample at the same acquisition time, the same joint encoding state, the same spectral channel, and the same spatial location and propagation direction.

[0139] Angle can represent the direction of propagation of light signals at a certain spatial location in the target sample relative to the imaging system.

[0140] Dynamic compensation can refer to the process of correcting complementary light field sampling data at different acquisition times based on the dynamic changes of the target sample represented by the spatiotemporal change data, so as to reduce the impact of dynamic changes on the consistency between data at different acquisition times.

[0141] Meta-imaging joint reconstruction can be described as a process of jointly solving complementary light field sampling data under different joint coding states, using the spatial structure data as constraints, to obtain the meta-imaging result.

[0142] Meta-imaging results can represent the imaging results obtained after performing the dynamic compensation and meta-imaging joint reconstruction on the target sample.

[0143] For example, each set of joint information in the joint information expression relationship corresponds to the same acquisition time and the same spectral channel. The spatial structure data is stored in correspondence with the joint coding state and the spectral channel it belongs to when it is acquired. Dynamic compensation corrects the complementary light field sampling data in the corresponding joint coding state according to the spatiotemporal change data. Meta-imaging joint reconstruction uses the spatial structure data in the joint information expression relationship as constraints to jointly solve the complementary light field sampling data in different joint coding states to obtain the meta-imaging result.

[0144] For specific methods of establishing joint information expression relationships and performing dynamic compensation and meta-imaging joint reconstruction, please refer to the possible implementation methods provided in this disclosure, which will not be elaborated here.

[0145] Furthermore, the complementary optical field encoding module adaptively adjusts the subsequent joint encoding state based on the spatiotemporal variation data, thereby forming a closed-loop relationship between optical encoding, complementary sensing, dynamic feedback, and computational reconstruction. Feedback control can be executed in parallel with the reconstruction processing of the acquired data, but the subsequent joint encoding state is determined only based on the acquired data, without using data that has not yet been acquired.

[0146] For example, after obtaining the spatiotemporal variation data that has been collected, the complementary optical field coding module adaptively adjusts the subsequent joint coding state according to the data, such as increasing or decreasing the degree of difference between subsequent adjacent joint coding states, so that the complementary optical field sampling data obtained under different joint coding states maintain an appropriate degree of distinction.

[0147] For specific methods of adaptively adjusting the subsequent joint coding state based on the spatiotemporal change data, please refer to the possible implementation methods provided in this disclosure, which will not be elaborated here.

[0148] In this embodiment, by collecting the signal light of the target sample, a common intermediate image containing at least two spectral components is formed, allowing the signals of at least two spectral components to converge and synchronously enter the subsequent encoding and acquisition process in the same intermediate image. By performing spatial and angular joint encoding on the common intermediate image under multiple joint encoding states based on a unified time reference, the encoded data obtained at different times maintains a unified correspondence in the time dimension, and the data in each joint encoding state simultaneously carries spatial location information and propagation direction information. By allocating the complementary encoded light field to at least two spectral channels with different spectral responses, and allowing each complementary encoded light signal allocated under the same joint encoding state to share the joint encoding state, the multispectral signals are allocated to different spectral channels without destroying the correspondence between the spectrum and the encoding state. The spatial structure data is output through the cognitive guidance pathway, and the spatiotemporal change data is output through the motion guidance pathway, allowing the two types of data to be obtained in parallel by the same complementary sensory neuromorphic chip. By establishing a joint information expression relationship of space, angle, time, and spectrum, the multidimensional information maintains correspondence under the same data organization framework. Based on this, since the spatiotemporal variation data is output by the motion-guided pathway and the spatial structure data is output by the cognitive-guided pathway, the inputs required for dynamic compensation and meta-imaging joint reconstruction both come from the same acquisition process. Because dynamic compensation is performed on the complementary light field sampling data based on the spatiotemporal variation data, the sampling data obtained under different joint encoding states can be corrected based on the dynamic information obtained during acquisition before meta-imaging joint reconstruction, reducing the deviation between data under different joint encoding states caused by different acquisition times. Meta-imaging joint reconstruction is constrained by the spatial structure data and shares the data associated with the joint information expression relationship with dynamic compensation. Furthermore, because the complementary light field encoding module adaptively adjusts the subsequent joint encoding state based on the spatiotemporal variation data, the subsequent joint encoding state can be updated based on the acquired spatiotemporal variation data. This improves the stability and imaging quality of meta-imaging reconstruction in dynamic scenes.

[0149] In one possible implementation, the complementary optical acquisition module includes a light-collecting objective, an imaging lens group, and a multi-band filter assembly;

[0150] The light-collecting objective lens is used to collect signal light from the same observation area of ​​the target sample through the same acquisition aperture, and the signal light contains the at least two spectral components;

[0151] The multi-band filter assembly is disposed between the light-collecting objective and the imaging lens group, and has at least two transmission bands corresponding to the at least two spectral components, for simultaneously filtering out the spectral components from the signal light.

[0152] The imaging lens group is used to make the signal light of each spectral component passing through the multi-band filter component propagate along the same imaging optical path and form the common intermediate image on the same intermediate image plane, so that the complementary optical field encoding module can perform spatial and angular joint encoding on the common intermediate image.

[0153] For example, the light-collecting objective can collect signal light from the same observation area of ​​the target sample through the same acquisition aperture; a multi-band filter assembly can be disposed between the light-collecting objective and the imaging lens group; the imaging lens group includes a tube lens for forming an intermediate image and a relay lens disposed according to the optical path layout.

[0154] Multi-band filtering components have at least two transmission bands, allowing signal light within each transmission band to pass through simultaneously during the same acquisition period while suppressing optical signals outside the transmission bands. Multi-band filtering components can employ filters or equivalent filtering components with multiple transmission windows, enabling the selection of the band to be observed within a shared optical path, without directing different bands to independent pre-coding optical paths.

[0155] For example, signal light passing through a multi-band filter assembly is imaged by an imaging lens group to form a common intermediate image on the same intermediate image plane. The imaging lens group is configured according to the selected band range so that the effective imaging area of ​​each band covers the common intermediate image.

[0156] As an example, in fluorescence microscopy, the illumination system includes an excitation source, an excitation filter assembly, and a dichroic mirror for guiding the excitation light. The illumination system provides excitation light to the target sample, and a condenser objective collects the signal light generated by the sample. The dichroic mirror, used to separate the excitation light from the sample signal light, and the spectrometer, located downstream of the encoding module and used to divide the detection spectral channels, are respectively set and labeled according to their actual functions.

[0157] The selected transmission band should be within both the effective transmission range of the optical device used and the effective response range of the complementary sensory neuromorphic chip. Different spectral components can characterize different marker structures or different optical responses. It is not required that their brightness distribution be the same, nor is it necessary for one band to specifically correspond to the cognitive guidance pathway and another band to specifically correspond to the motor guidance pathway.

[0158] Figure 2 A schematic diagram of the optical path system of a meta-imaging system based on a complementary sensing neuromorphic chip according to an embodiment of the present disclosure is shown. Figure 2As shown, the illumination light emitted by the light source is incident on the target sample through the illumination system and the dichroic mirror. The signal light generated by the excitation or scattering of the target sample is collected by the microscope, passes through the tube mirror, and reaches the relay lens after the beam aperture is constrained by the aperture stop, forming a common intermediate image carrying the target sample information. The common intermediate image is incident on the scanning mirror and completes spatial and angular joint encoding through the movable coded microlens array. The scanning mirror and the movable coded microlens array move in coordination based on a unified time reference to form multiple joint encoding states. Corresponding complementary coded light fields are obtained in different joint encoding states. The complementary coded light fields are assigned to at least two spectral channels with different spectral responses to obtain at least two complementary coded light signals. Each complementary coded light signal is transmitted to the detection surface through the dichroic mirror. The complementary coded light signal of each spectral channel is collected by the cognitive guidance path and the motion guidance path respectively. The spatial structure data output by the cognitive guidance path and the spatiotemporal change data output by the motion guidance path constitute the complementary light field sampling data.

[0159] In this embodiment, the light-collecting objective lens collects signal light from the same observation area of ​​the target sample through the same acquisition aperture, allowing signals of different spectral components to enter the same optical path from the same acquisition aperture. The multi-band filter assembly has at least two transmission bands corresponding to the at least two spectral components and simultaneously filters out each spectral component from the signal light, allowing signals of different spectral components to pass through in parallel during the same acquisition period, rather than changing the filter device time-divisionally. The imaging lens group allows the signal light of each spectral component to propagate along the same imaging optical path and form the common intermediate image on the same intermediate image plane. Since each spectral component propagates along the same optical path and converges on the same intermediate image plane during the same acquisition period, the complementary optical field encoding module can simultaneously cover each spectral component by performing a single spatial and angular joint encoding on the signal of each spectral component. This avoids the misalignment of acquisition time of different spectral components caused by time-division acquisition, provides a premise for each spectral channel to share the same joint encoding state, improves the temporal consistency of multispectral acquisition, and thus improves the stability and imaging quality of meta-imaging reconstruction in dynamic scenes.

[0160] In one possible implementation, the complementary optical field encoding module includes a scanning mirror, a movable encoding microlens array, an encoding controller, and a driving component, wherein the driving component is connected to the scanning mirror and the movable encoding microlens array, respectively.

[0161] The joint encoding state includes the angular deflection state of the scanning mirror and the spatial displacement state of the movable encoding microlens array;

[0162] The encoding controller is used to control the scanning mirror and the movable encoding microlens array to form corresponding joint encoding states based on the unified time reference, and to determine the duration of each joint encoding state.

[0163] The encoding controller is also used to receive the spatiotemporal change data during the period when the sampling time interval is within the same joint encoding state, determine the motion amplitude of the target sample based on the spatiotemporal change data, and adjust the combination, execution order or holding duration of subsequent joint encoding states based on the motion amplitude.

[0164] For example, the scanning mirror can be driven by an angle driving component, and the movable coded microlens array can be fixed on a translation driving component and capable of displacement in a direction parallel to the array plane. Both types of driving components are controlled by an encoding controller.

[0165] For example, the scanning mirror can be placed in the relay optical path that propagates the common intermediate image to the microlens array.

[0166] As an example, a folding scanning mirror can be placed at the common intermediate image plane or its conjugate plane, and the common intermediate image can be transmitted to the image plane where the microlens array is located through a subsequent relay lens. The scanning mirror deflects at a small angle to change the direction distribution of the beam incident on the array, and the array is translated to change the sampling position of the microlens unit relative to the common intermediate image.

[0167] For the kth joint coding state It can be described as follows:

[0168]

[0169] in, and It can represent the angular deflection components of the scanning mirror in the x-axis and y-axis directions in the k-th joint coding state; and The k-th joint encoding state can represent the spatial displacement components of the movable coded microlens array in the x and y directions; k can represent the sequence number of the joint encoding state. For implementations where movement occurs only in one direction, the corresponding other components can remain fixed. The encoding controller stores candidate states and their corresponding spatial and angular sampling maps, and selects state combinations and execution order according to the current encoding plan.

[0170] Candidate states can be filtered through static calibration, ensuring that different states provide usable complementary sampling positions and maintaining the effective beam within the array and detection area. Static calibration refers to the process of pre-determining and filtering the sampling positions and beam coverage corresponding to each candidate state by sequentially holding the scanning mirror and the movable coded microlens array in each candidate state before the system's formal acquisition. If the movements of two actuators cancel each other out in a specific combination, resulting in essentially identical actual sampling relationships, this combination will not be repeatedly counted as a new effective sampling state. Occlusion, overlap, and vignetting areas between microlens sub-images are marked as invalid.

[0171] The encoding can adopt a "state transition—stabilization wait—state hold—enter the next state" approach. For the k-th state, record the state start time, the actuator stabilization time, and the state end time; the effective hold interval after stabilization is denoted as... ,exist Within this range, the scanning mirror angle and array position are maintained within a preset tolerance range. It can represent the effective holding interval of the k-th joint encoded state.

[0172] The drive stabilization time can be determined based on position feedback or a pre-measured drive response. When no position sensor is configured, a calibrated correspondence between drive commands and steady-state positions is used, with sufficient stabilization waiting time allowed. This correspondence between drive commands and steady-state positions can represent the pre-measured correspondence between the drive command input to the drive component and the actual steady-state position reached by the actuator after executing the command.

[0173] The duration of the encoded state retention needs to meet the requirements of exposure, sampling, and safety intervals for complementary sensing neuromorphic chips. Priority should be given to ensuring that the complete exposure process of the single-frame spatial structure data involved in the reconstruction is within an effective retention interval, so as to avoid incorrectly corresponding the spatial structure data of a single state after integrating samples from multiple different encoded states into the same frame of spatial structure data.

[0174] The coding controller receives the spatiotemporal change data obtained during the effective maintenance of the same joint coding state and uses the motion estimation results to determine the subsequent coding arrangement. Motion estimation can be performed by the coding controller or by the multidimensional complementary information representation and reconstruction module, which then transmits the motion amplitude and motion confidence, characterizing the reliability of the motion estimation results, to the coding controller.

[0175] As an example, the spatiotemporal variation data is first organized into corresponding sub-view data based on the spectral channel, microlens unit, and angle. Each sub-view corresponds to the variation response obtained at the same angle within the same spectral channel at continuous sampling times. The spatial and angular correspondence of the sampling positions involved in each variation data is preserved during organization. Variation data that crosses microlens boundaries, crosses spectral regions, or corresponds to different angles and does not satisfy the current spatial gradient model are not directly regarded as spatial gradients within the same image.

[0176] For example, a normalized response distribution map can be constructed on the effective sampling grid of the sensor based on the response differences between each sampling location, and then a sub-view at the same angle can be extracted based on this map. This construction can be accomplished by minimizing the weighted difference between the predicted response difference and the measured response difference, and by setting a mean constraint to eliminate constant uncertainty in the reconstruction. The response changes between each sampling time are used to provide the change constraints and candidate dynamic regions for the corresponding time period.

[0177] For the same spectral channel and the same angle, local motion estimation is performed on subviews obtained at consecutive sampling times. Local optical flow or structure block matching can be used: matching regions are determined by the edges and textures in the subview, the response changes between adjacent sampling times and their sampling intervals are used to limit the time range of motion occurrence, and the displacement direction and magnitude are determined by the structural correspondence at consecutive times.

[0178] Under the condition that the data is transformed to the same response scale and the local structure remains consistent within adjacent sampling intervals, a local estimation model can be expressed as:

[0179]

[0180] in, and It can represent the amount of change of the same subview in two spatial directions. It can represent the time variation normalized over the sampling interval. and It can represent the velocity components in the two directions to be estimated. This can represent local brightness variations. A weighted fit is performed on multiple valid sampling points within the local neighborhood, and constrained by the estimation results from the previous time step, to obtain the local velocity. Each variation is calculated from the data after response and spatial mapping correction, rather than directly using the chip's original code values.

[0181] Motion confidence, a parameter that evaluates the reliability of local motion estimation results, can be determined based on the number of effective gradients, local fitting residuals, and consistency of forward and backward matching. When a region lacks texture, exhibits saturation, is occluded, or its brightness changes are indistinguishable from displacement, the motion confidence of that region is reduced; not all response changes are interpreted as sample displacement, nor are fluorescence enhancements or attenuations directly compensated for geometric motion.

[0182] The velocities of each effective local region are converted to a unified spatial reference scale or a unified reference pixel scale. The maximum velocity modulus can be taken from the region where the confidence level reaches a threshold. and multiplied by a fixed prediction duration The predicted motion amplitude used for control is obtained. If different spectral channels exhibit different motion structures, they are estimated separately, and then the global encoding arrangement is determined based on the upper limit of motion in the effective region. Different spectral channels are not forced to share the same set of motion estimation results. It can represent the predicted motion amplitude determined based on local velocity and used to control the coding arrangement.

[0183] As an example, let the allowable predicted displacement be... The allowable range for the duration of state retention is to The duration of the target state to be maintained in the next adjustable state is determined by the following formula:

[0184]

[0185] in, It can indicate the duration for which the target is maintained in the next adjustable state; It can represent small positive numbers used to avoid division by zero; It can represent the lower limit of the allowable range of state retention time, which is determined by full exposure, multiple sets of effective sampling, and necessary safety intervals; It can represent the upper limit of the allowable range of the state retention time, which is determined by the preset acquisition cycle and the effective optical signal requirements; This can represent the predicted motion amplitude. The formula gives the target value, but the actual holding time is adjusted according to the device's allowed exposure sequence and state switching times.

[0186] When the motion is small, a longer holding time or a combination of states with sufficient information coverage is used. When the motion increases, the subsequent holding time is shortened within the above constraints, or a shorter combination of states that still meets the required spatial and angular coverage is selected. If the allowable displacement cannot be met even after the minimum available time has been reached, the corresponding acquisition period is marked as a high dynamic or low confidence period. Within this sampling period, the same resolution as static dense sampling is not guaranteed.

[0187] When motion confidence is insufficient, the controller maintains the preset safety coding arrangement or repeats a calibrated reference state to re-acquire the structural correspondence. The adjusted state combination and execution order must remain calibrated, and new states lacking sampling mappings should not be arbitrarily selected.

[0188] Feedback adjustments take effect after the current effective exposure ends and the actuator is allowed to switch. The time required for data transmission, motion estimation, command delivery, and drive stabilization is included in the scheduling; if the calculation result fails to catch up with the immediately following state, it takes effect in the subsequent adjustable state. Therefore, the "subsequent joint encoding state" is not limited to the immediately following state, nor is it contingent upon unconfirmed zero-latency feedback.

[0189] In this embodiment, the joint encoding state is jointly determined by the angular deflection state of the scanning mirror and the spatial displacement state of the movable encoding microlens array. The combination of these two degrees of freedom allows the system to form diverse joint encoding states with a limited number of components, expanding the coverage of spatial and angular sampling. The encoding controller, based on the unified time reference, controls the two to form corresponding joint encoding states and determines the holding duration through the driving component, ensuring strict alignment between the encoding process and the acquisition process in the time dimension. The encoding controller determines the motion amplitude of the target sample based on the spatiotemporal change data obtained during the holding period of the same joint encoding state, and adjusts the combination, execution order, or holding duration of subsequent joint encoding states accordingly. Since the motion amplitude is determined by the spatiotemporal change data and subsequent joint encoding states are adjusted by the motion amplitude, the more intense the motion, the more compact the encoding sampling arrangement; the gentler the motion, the more sufficient the sampling arrangement. This matches the switching rhythm of the encoding state with the sample motion state reflected by the motion amplitude, reducing sampling redundancy in high-speed dynamic scenes and insufficient sampling in dynamic scenes, improving the adaptability and data utilization efficiency of encoding acquisition in dynamic scenes, thereby improving the stability and imaging quality of meta-imaging reconstruction in dynamic scenes.

[0190] In one possible implementation, the complementary spectral allocation module includes a spectral splitting component and a spectral filtering component and a relay imaging component respectively disposed in each of the spectral channels;

[0191] The beam splitter is used to distribute the complementary coded light field to each of the spectral channels, and at least two of the spectral filtering components have different transmission band ranges or relative transmittance between bands.

[0192] Each complementary coded optical signal corresponds to the same observation area of ​​the target sample, and is imaged onto a non-overlapping photosensitive area on the complementary sensing neuromorphic chip via the corresponding relay imaging component. Each photosensitive area is associated with a corresponding spectral channel.

[0193] For each photosensitive area, the cognitive guidance pathway outputs spatial structure data of the corresponding spectral channel, and the motion guidance pathway outputs spatiotemporal variation data of the corresponding spectral channel.

[0194] The beam splitter distributes the spatially and angularly encoded light field to each spectral channel, and the spectral filter further determines the effective transmission band or the relative transmittance between bands for each channel.

[0195] The beam splitter can employ a dichroic beam splitter or a combination of a beam splitter and channel filters. Each spectral channel can have a primary transmission band that does not overlap, or it can have partially overlapping but different spectral responses. The number of channels and bandwidth are determined based on the required spectral information, the available photosensitive area of ​​the chip, and the effective light signal obtainable by each channel.

[0196] Taking two spectral channels as an example, after spectral separation, a first relay imaging optical path and a second relay imaging optical path are set up to transmit two spectral versions of the same coded pattern to the first and second photosensitive areas of the complementary sensing neuromorphic chip, respectively. The relay imaging component images the coded pattern formed by the microlens array, rather than simply reprojecting the ordinary intensity image of the sample onto the chip.

[0197] As an example, the two relay optical paths can be positioned by adjusting the folding mirrors, and the imaging magnification can be configured to ensure that the two coded patterns fall into non-overlapping photosensitive areas. Light-shielding or protective intervals are set at the area boundaries to avoid crosstalk between adjacent optical paths; each area covers an effective sampling unit capable of providing cognitive guidance and motion guidance data.

[0198] When using the same chip for regional reception, the complete output of the chip can be read first, and then data from different spectral channels can be extracted based on the calibrated regional mask. This embodiment does not require the chip to provide an independent readout interface for each region, nor does it require modification of the internal pixel arrangement of the chip.

[0199] The pixel range, microlens unit position, imaging magnification, distortion, and the correspondence between the cognitive guidance path and the motion guidance path in the first and second photosensitive regions are respectively calibrated, so that the data of both regions can be mapped to the same object-side reference coordinate system. Coordinate calibration represents geometric correspondence and does not imply that all structures under different spectra have the same brightness or motion.

[0200] Since all spectral channels share the same chip photosensitive area, the configuration of the repeater magnification takes into account the effective field of view, the number of microlens sub-image samples, and the size of the photosensitive area. Sampling points that are cropped or fall within invalid boundaries are marked in the data log and are not considered valid measurements.

[0201] In this embodiment, the beam splitting component distributes the complementary coded light field, which has already undergone spatial and angular joint encoding, to each of the spectral channels. Since the distribution occurs after encoding, the complementary coded light signals obtained by each spectral channel naturally carry the spatial and angular information assigned during the encoding process. The complementary coded light signals distributed under the same joint encoding state share this joint encoding state, so that multispectral detection does not need to repeatedly establish encoding and acquisition timing for each spectral channel, thus maintaining the correspondence between the spectral dimension and the encoding state dimension. Each complementary coded light signal is imaged onto a non-overlapping photosensitive area on the complementary sensing neuromorphic chip via the corresponding relay imaging component, avoiding the aliasing of different spectral channels on the same photosensitive area. For each photosensitive area, the cognitive guidance pathway and the motion guidance pathway respectively output the spatial structure data and spatiotemporal change data of the corresponding spectral channel, so that each spectral channel can simultaneously obtain spatial structure data and spatiotemporal change data. Thus, the system achieves parallel complementary sensing of multispectral channels using a single complementary sensing neuromorphic chip, improving the accuracy of the correspondence between multispectral data and encoded state, cognitive guidance pathways and motion guidance pathways, as well as the compactness of the system structure, thereby improving the stability and imaging quality of meta-imaging reconstruction in dynamic scenes.

[0202] In one possible implementation, the cognitive guidance pathway and the motion guidance pathway are synchronously acquired based on the unified time reference;

[0203] The cognitive guidance pathway outputs the spatial structure data at a first sampling frequency, and the spatial structure data is intensity frame data characterizing the incident light intensity and its spatial distribution.

[0204] The motion guidance path outputs the spatiotemporal variation data at a second sampling frequency, which is higher than the first sampling frequency. The spatiotemporal variation data includes spatial difference data and temporal difference data.

[0205] The brain-like complementary perception module is also used to record the exposure time interval for each frame of spatial structure data based on the unified time reference, record the sampling time interval for each group of spatiotemporal change data, and associate multiple groups of spatiotemporal change data whose sampling time intervals are located in the same exposure time interval with the corresponding spatial structure data.

[0206] As an example, the complementary sensing neuromorphic chip can employ the Tianmouc chip, which is a complementary sensing neuromorphic visual chip that integrates both frame-based intensity sensing and differential sensing functions. The cognitive guidance pathway provides frame-based intensity or color data, while the motion guidance pathway provides spatial and temporal differential data. The complementary sensing neuromorphic chip and its associated acquisition circuitry handle photoelectric acquisition and readout functions. This system utilizes the aforementioned outputs and the optical encoding process to establish a collaborative relationship.

[0207] Spatial structure data is denoted as This can represent intensity frame data obtained within a given exposure range. When quantitative reconstruction is required, the original data or linearized data that preserves the response relationship is preferred, and corrected according to black level, gain, and exposure time; the image values ​​used for display and non-linearly enhanced are not directly used as linear light intensity measurements.

[0208] Spatial difference data is denoted as Time difference data is denoted as . It can characterize the spatial response differences between corresponding sampling locations. It can characterize the response changes within the corresponding time interval. Both are decoded, debiased, and calibrated according to the output format of the chip used.

[0209] The first sampling frequency can represent the frequency at which the cognitive guidance path outputs spatial structure data; the second sampling frequency can represent the frequency at which the motion guidance path outputs spatiotemporal change data. The specific frame rate, bit width, and exposure configuration are determined by the chip's operating mode. This embodiment does not use a fixed maximum frame rate as the operating parameter that all data modes can simultaneously achieve.

[0210] The unified time reference is a clock reference shared by the encoding control process and the acquisition process. For acquisition circuits with independent counters, their timestamps can be converted to time coordinates under the unified time reference through synchronization markers and clock deviation calibration. The unified time reference is used to establish the timing relationship of each event and does not indicate that two paths have the same sampling frequency or the same exposure duration.

[0211] Record the start and end times of exposure for each frame of spatial structure data to form an exposure time interval. Record the corresponding sampling time interval for each set of spatiotemporal variation data. For satisfying lie in Multiple sets of spatiotemporal variation data within the frame are associated with the spatial structure data of the nth frame. It can represent the exposure time interval of the spatial structure data of the nth frame. It can represent the frame number of spatial structure data; This can represent the sampling time interval of the j-th group of spatiotemporal variation data. It can represent the group number of spatiotemporal change data.

[0212] Effective status acquisition further satisfies: lie in Inside, used for dynamic estimation in this state Also located The validity of the time difference is determined jointly by the two sampling times involved in the difference; if If a state transition boundary is crossed, or if its internal reference still contains the response before the transition, it is marked as transitional data and is not directly used for sample motion estimation.

[0213] If the devices used have line-by-line exposure differences, the exposure time offset should be retained for the corresponding pixel line, and the state retention range should cover all valid lines involved in the frame processing. The arrival time of the received data packet does not directly replace the chip sampling time. The exposure time marker, frame count, and their relationship with the system clock are jointly determined by the actual acquisition interface and synchronization calibration.

[0214] For example, the encoding controller arranges an effective hold interval containing at least one frame of complete spatial structure data and multiple sets of spatiotemporal variation data within that exposure interval. For free-running acquisition modes, the encoding state switching can be scheduled outside the predetermined effective exposure interval, and frames affected by the state switching can be discarded.

[0215] In this embodiment, the cognitive guidance pathway and the motion guidance pathway are synchronously acquired based on the unified time reference, giving the two types of data a unified time coordinate. The cognitive guidance pathway outputs the spatial structure data as intensity frame data at a first sampling frequency, enabling the spatial structure data to be obtained completely frame by frame. The motion guidance pathway outputs the spatiotemporal change data at a second sampling frequency higher than the first sampling frequency, allowing dynamic information to be captured with higher temporal density. Since the spatiotemporal change data includes spatial difference data and temporal difference data, dynamic change information can be recorded separately in two differential forms. The brain-like complementary perception module associates multiple sets of spatiotemporal change data with corresponding spatial structure data within the same exposure time interval, enabling accurate pairing of the spatiotemporal change data and corresponding spatial structure data in the time dimension. Thus, while ensuring the imaging quality of the spatial structure data, the system obtains spatiotemporal change data with higher temporal density, improving the accuracy of the temporal correspondence between the spatiotemporal change data and the spatial structure data and the completeness of dynamic perception, thereby improving the stability and imaging quality of meta-imaging reconstruction in dynamic scenes.

[0216] In one possible implementation, associating the spatial structure data and the spatiotemporal variation data with the joint encoding state and corresponding spectral channels at the time of acquisition, respectively, to establish a joint information representation relationship of space, angle, time, and spectrum, includes:

[0217] Based on the unified time reference, the corresponding sampling time, joint coding status identifier, spectral channel identifier, and sensing path identifier are associated with the complementary light field sampling data.

[0218] Based on the angular deflection state of the scanning mirror, the spatial displacement state of the movable coded microlens array, and the pre-calibrated mapping relationship between the microlens unit and the imaging pixel, the spatial sampling position and observation angle corresponding to the complementary light field sampling data are determined.

[0219] Based on the pre-defined coordinate transformation relationship between different spectral channels and the spatial mapping relationship between the cognitive guidance path and the motion guidance path, the spatial sampling position is transformed to a unified object coordinate system;

[0220] Based on the sampling time and the converted spatial sampling location, the spatial structure data and the spatiotemporal change data are uniformly associated with the corresponding observation angle, joint coding state, spectral channel and sensing path to establish the joint information expression relationship.

[0221] Perceptual pathway identifiers can represent identifier information output from cognitive or motor pathways; cognitive and motor pathways can be collectively referred to as perceptual pathways.

[0222] Sampling time can represent the point in time when the sampled data was collected.

[0223] The joint coding state identifier can represent the marking information of the joint coding state corresponding to the sampled data.

[0224] Spectral channel identifiers can represent the marking information that identifies the spectral channel to which the sampled data belongs.

[0225] The object-space coordinate system can represent a coordinate system that unifies data from different spectral channels and different pathways under the same spatial reference.

[0226] Spatial sampling location can represent the spatial location of the sampled data in the object coordinate system.

[0227] The observation angle can represent the angle of the direction of light signal propagation corresponding to the sampled data, and together with the spatial sampling position, it identifies the spatial and angular source of the sampled data.

[0228] For example, the system performs spatial, temporal, and response calibrations before formal data acquisition. The calibration results are saved separately according to joint coding status, spectral channels, and sensing pathways, and associated with version identifiers of the data processed using these results.

[0229] As an example, both shading and uniform illumination data can be collected to determine the black level, effective response range, dead pixels, and gain differences for each photosensitive area. For motion-guided paths, the bias, sign, range, and effective response interval of the response are recorded using static patterns and controlled brightness variations. Saturated or abnormal data that cannot be reliably converted are marked as invalid.

[0230] Using point-based calibration targets or rule-based calibration patterns at known locations, data is collected under various candidate joint encoding states to determine the boundaries of microlens sub-images, the centers of microlens units, and their mapping relationships with chip pixels. Combining object-side position changes and incident direction calibration, a mapping is established from the sampling pixels to the object-side reference plane position and the observation angle.

[0231] For spectral channels Perceptual pathways and joint coding status The mapping can be represented as:

[0232]

[0233] in, It can represent spectral channels Perceptual pathways and joint coding status The corresponding mapping; and It can represent the pixel coordinates of the corresponding data grid; and It can represent the sampling position on the object reference plane; and This can represent the observation angle coordinates. This mapping can be stored using a parametric model or a lookup table. A valid pixel actually corresponds to a finite spatial and angular sampling range; the corresponding sampling kernel is retained in the reconstruction model, without requiring it to be an ideal single ray.

[0234] By using the same calibrated target observable in each target band, we determine the coordinate transformation between different spectral channels and the correspondence between the cognitive guidance path and the motion guidance path data grid.

[0235] Subsequently, using light intensity modulation or synchronization markers that can record time points, the correspondence between the exposure process of spatial structure data, the sampling time interval of spatiotemporal variation data, and the encoding control clock is determined, and the transition time that needs to be excluded after actuator switching is identified. The effective response of each spectral channel is saved for subsequent data processing.

[0236] Based on the above calibration, each sampling record is associated with at least: sampled value and data type, sampling time or time interval, joint encoding status identifier, spectral channel identifier, sensing pathway identifier, microlens unit identifier, spatial sampling location, observation angle, and validity mark. The spatial structure data of each frame and multiple sets of spatiotemporal variation data within the corresponding sampling time interval are organized through an association index, and are not counted as independent intensity measurements.

[0237] For data falling within different state retention intervals, the state to which it belongs is determined based on its time interval and state record. For frames that span state transitions, they are excluded from single-state reconstruction data; when such data needs to be used, an integral observation relationship containing exposure contributions from multiple states should be established separately, and the most recently acquired data should not be used to replace this relationship.

[0238] The joint information representation relationship only establishes a correspondence basis for geometry, temporal sequence, and data type, and does not automatically eliminate sample motion. Dynamic compensation is still performed based on the actual measured spatiotemporal changes; when different spectral channels have different label structures, their structures or brightness are not forced to be the same.

[0239] In this embodiment, the complementary light field sampling data is associated with sampling time, joint encoding status identifier, spectral channel identifier, and sensing path identifier, so that each sampling data carries complete source and timing information. The spatial sampling position and observation angle are determined according to the angle deflection state of the scanning mirror, the spatial displacement state of the movable coded microlens array, and the pre-calibrated mapping relationship, so that the spatial and angular information of the light signal given by the encoding process can be quantitatively analyzed. The spatial sampling position is transformed to a unified object coordinate system according to the coordinate transformation relationship between different spectral channels and the spatial mapping relationship between the two paths, so that the data of different spectral channels and different sensing paths can be compared under the same spatial reference. The sampling time and the transformed spatial sampling position are uniformly associated to establish a strictly corresponding joint information expression relationship between spatial, angular, temporal, and spectral multidimensional information. Since the above correspondence has been established in the unified association process, the subsequent dynamic compensation and meta-imaging joint reconstruction can directly use the associated data as input, without having to perform cross-modal and cross-channel registration again. This avoids information fragmentation caused by processing data from different sources separately, improves the accuracy of the joint information expression relationship and the efficiency of subsequent processing, thereby improving the stability and imaging quality of meta-imaging reconstruction in dynamic scenes.

[0240] In one possible implementation, the dynamic compensation includes:

[0241] Based on the spatiotemporal change data collected during each joint coding state, and combined with the joint information expression relationship, the local motion field and dynamic perturbation parameters of the imaging field of view of the target sample are estimated, wherein the local motion field characterizes the motion difference between different local regions, and the dynamic perturbation parameters characterize the overall offset of the imaging field of view.

[0242] Based on the local motion field, the dynamic disturbance parameters, and the time interval between each sampling time and the predetermined reference time, determine the motion transformation relationship of the complementary optical field sampling data corresponding to each joint coding state to the predetermined reference time;

[0243] By combining the spatial sampling position and observation angle corresponding to each joint coding state, and while retaining the observation angle identifier, spectral channel identifier and joint coding state identifier corresponding to each sampling data, the complementary light field sampling data corresponding to different joint coding states are time-aligned and spatially registered according to the motion transformation relationship to obtain the compensated light field sampling data.

[0244] For example, the global translation component is obtained from the reliable motion vector through robust fitting, and the remaining motion of each region relative to the global component is then used as the local motion. When there is a fixed background or reference marker, it can be preferentially used to estimate the overall offset of the system.

[0245] Based on the local and global components at each moment, the motion transformation is accumulated segment by segment, and the sampling time is... The corresponding object position is transformed to For local regions with approximately uniform velocity over a short period, the product of velocity and time interval can be used to approximate the displacement; for regions with significant deformation or velocity changes, a piecewise composite local transformation can be used. It can represent the predetermined reference time to which all sampled data are uniformly aligned.

[0246] When reliable motion data is lacking during state transitions, short-term predictions can be made using motion models of adjacent valid intervals, and verified using similar features obtained after re-entering the valid state. If the prediction span exceeds the set range or verification fails, the compensation reliability of that interval can be reduced or relevant data can be excluded.

[0247] Dynamic compensation is performed separately for spectral channels and viewing angles within a unified object coordinate system. After correction, the original viewing angle, spectral channels, and joint encoding status identifiers are retained. All features under different viewing angles are not forcibly registered to the same image position to avoid eliminating parallax used for 3D reconstruction as motion error.

[0248] Different spectral channels share corresponding motion parameters only when they have a credible common structure or a common overall perturbation basis; local motions of different marker structures are handled separately. Thus, geometric co-registration is not misused as the assumption that cross-spectral motions are completely identical.

[0249] For spatial structure data with sufficiently small displacement during exposure, spatial transformation can be performed using the midpoint of the exposure as the effective sampling time. If there is non-negligible motion during exposure, a single translation cannot eliminate the blurring caused by exposure integration. The motion trajectory during exposure should be retained in the observation relationship on which subsequent reconstruction is based. If the motion trajectory cannot be reliably estimated, the corresponding data should be downweighted or marked as invalid.

[0250] Spatiotemporal variation data are updated in association with their corresponding sampling locations and sampling time intervals, and are not directly used as ordinary intensity images for numerical interpolation and added to spatial structure data. The compensated light field sampling data can include intensity measurements, spatiotemporal variation observations, motion transformations, and their reliability, all associated with a reference time.

[0251] For example, suppose a local structure in the same spectral channel and at the same viewing angle has coordinates at (100,100), (103,102), (106,104), and (109,106) at four time points after coded geometric correction. Using the first time point as a reference, the subsequent positions can be shifted back to (3,2), (6,4), and (9,6) respectively according to the estimated motion transformation. This example only illustrates coordinate transformations under known translation conditions and does not imply that four images at different viewing angles can be translated indiscriminately, nor does it represent measured motion data.

[0252] In this embodiment, the local motion field and the dynamic perturbation parameter are estimated based on the spatiotemporal change data collected during each joint coding state. Since the local motion field represents the motion differences between different local regions and the dynamic perturbation parameter represents the overall shift of the imaging field of view, the local motion of the sample and the overall perturbation of the system are characterized and compensated separately, avoiding the aliasing of the two types of motion by a single compensation model. The motion transformation relationship is determined based on the local motion field, the dynamic perturbation parameter, and the time interval between each sampling time and a predetermined reference time, so that the complementary light field sampling data obtained under different joint coding states are uniformly transformed to the same reference time. Temporal alignment and spatial registration are performed while preserving the observation angle identifier, spectral channel identifier, and joint coding state identifier, so that the compensation process does not destroy the dimensional identifier carried by the data, thus preserving the complete information dimension for subsequent reconstruction. Therefore, the data obtained under different joint coding states are aligned in time and registered in space, which reduces the impact of target motion and environmental disturbances on the data consistency between different joint coding states, improves the temporal and spatial consistency of the compensated light field sampling data, provides a reliable data foundation for meta-imaging joint reconstruction, and thus improves the stability and imaging quality of meta-imaging reconstruction in dynamic scenes.

[0253] In one possible implementation, the meta-imaging joint reconstruction includes:

[0254] Based on the spatial structure data in the compensated light field sampling data and the corresponding joint coding state and spectral channel, a spatial and angular joint reconstruction model is established. The spatial and angular joint reconstruction model includes the spatial and angular sampling relationship corresponding to each joint coding state and the spectral response of each spectral channel.

[0255] The fusion weights are determined based on the signal-to-noise ratio, exposure saturation, and registration residuals after dynamic compensation in each local region of data corresponding to different joint coding states and different spectral channels.

[0256] Using the fusion weights, the differences between the intensity data predicted by the joint spatial and angular reconstruction model and the spatial structure data in the compensated light field sampling data are weighted to form a weighted data consistency term;

[0257] Based on the edge or texture features provided by the spatial structure data, a structural constraint term is constructed. The objective function composed of the weighted data consistency term and the structural constraint term is jointly solved to recover the light field information of the target sample and generate the meta-imaging result.

[0258] The multidimensional complementary information representation and reconstruction module can establish an observation model based on the spatial structure data and its joint encoding state, spectral channels, and observation angle in the compensated light field sampling data. The spatial structure data provides intensity measurements; the spatiotemporal variation data is used to determine motion transformations, temporal relationships, and corresponding reliability. The spatiotemporal variation data and spatial structure data are not directly added as observations of the same dimension.

[0259] For example, it can be (x,y,u,v) represents the optical field to be recovered corresponding to the response of the c-th spectral channel at the reference time. x and y can represent the sampling positions on the object-side reference plane, and u and v can represent the observation angle coordinates. The effective spectral responses of each channel are stored in the calibration parameters. This can represent the light field under the weighted response.

[0260] Based on the joint encoding state and corresponding calibration results, a sampling operator is established from candidate light fields to chip intensity measurements. This operator can include spatial and angular sampling ranges, relay imaging magnification, effective pixel range, and channel response. After applying motion compensation, the same compensation transformation is simultaneously applied to the observation model's predicted values ​​to obtain predicted values ​​in the same coordinate system as the compensated measured values.

[0261] As an example, an observation relationship that includes exposure integral can be established:

[0262]

[0263] in, It can represent the intensity data of the spatial structure data of the nth frame after exposure normalization on the cth spectral channel; It can represent the sampling operator corresponding to the k-th joint coding state and the c-th spectral channel; It can represent the value obtained from the reference time by motion estimation. When it's exposed Transformation; It can represent the exposure time interval of the spatial structure data of the nth frame; This can represent measurement error. The exposure integral can be discretely summed over the effective sampling times. This model is suitable for situations where the selected local area can maintain structural correspondence during exposure, and the brightness change can be modeled or is sufficiently small; if these conditions are not met, the reliability of the corresponding data is reduced.

[0264] The fusion weights are determined based on the signal-to-noise ratio, exposure saturation, and registration residual after compensation for each local region r. One possible calculation method is:

[0265]

[0266] in, It can represent the fusion weight; It can represent an effectiveness mask; It can represent the local saturation ratio; A signal-to-noise ratio metric that can represent a linear scale; It can represent the registration residual of the corresponding structure within the same spectral channel; the aforementioned subscripts n, c, and r can respectively represent the spatial structure data of the nth frame, the cth spectral channel, and the rth local region; and This can represent a positive normalization parameter. The noise level is estimated by response calibration and observation conditions, and local texture fluctuations are not directly treated as noise. The weights of sampling points that are fully saturated or have no effective mapping are set to zero.

[0267] The weights are used to adjust the residual contribution between the measured data and the model prediction data, without summing pixel values ​​from different spectra or viewpoints indiscriminately. When all candidate data are of low confidence, the low confidence markers are retained, and regions lacking effective observations are not converted into high confidence results through normalization.

[0268] The weighted data consistency term can be represented by the term obtained by weighting the differences between the predicted intensity data and the spatial structure data in the compensated light field sampling data using the fusion weights.

[0269] Structural constraint terms can represent terms constructed based on edge or texture features provided by the spatial structure data and used to constrain the reconstruction results.

[0270] Using compensated spatial structure data, a reference structure map is constructed within the corresponding spectral channels, and edge or texture intensity is extracted. Structural constraints can be implemented using edge-guided spatial smoothing: increasing smoothing constraints in regions lacking reliable edges and reducing cross-edge smoothing in regions with reliable edges. The smoothing constraint intensity for each local region can also be determined based on the aforementioned signal-to-noise ratio index. To mitigate the impact of noise in low signal-to-noise ratio regions, stronger smoothing constraints are applied, for example, to regions with lower signal-to-noise ratios. Structural constraints are established separately for each spectral channel; the labeling structures of two channels are not assumed to be identical.

[0271] The objective function can be represented as a function composed of the weighted data consistency term and the structural constraint term, used to solve for the light field of the target sample to be reconstructed.

[0272] As an example, nonnegativity constraints can be used to solve the problem:

[0273]

[0274] in, It can represent the light field of the target sample to be reconstructed; It can represent the... The light field obtained by joint solution, This can represent the extraction operator for the r-th local region; The prediction operator can represent the joint coding state and the spectral channel corresponding to it; A signal-to-noise ratio metric that can represent a linear scale; This can represent the intensity measurement obtained after dynamic compensation of the exposure-normalized intensity data; This can represent the structural constraint term; This can represent constraint coefficients. For example, a spatial gradient penalty, weighted by edge intensity, can be applied to light field slices at each viewing angle, causing structural constraints to act in the spatial direction without achieving surface sharpness enhancement by eliminating angular differences.

[0275] Projected gradients or other iterative solution methods can be used to sequentially perform model prediction, weighted residual calculation, structural constraint gradient calculation, and nonnegative projection. The process stops when the relative change of the objective function falls below a set threshold or when the preset maximum number of iterations is reached. The output restores the light field while preserving effective coverage and local confidence.

[0276] Based on the recovered light field, two-dimensional structural images, digital refocusing images, and multi-focal-plane images can be generated; when depth maps or three-dimensional volume data are needed, they can be calculated in conjunction with the corresponding parallax or three-dimensional optical forward model. The output resolution is adapted to the actual space, angular sampling coverage, and signal-to-noise ratio.

[0277] In this embodiment, a joint spatial and angular reconstruction model is established based on the spatial structure data in the compensated light field sampling data, as well as the corresponding joint encoding state and spectral channels, ensuring that the reconstruction model strictly corresponds to the actual optical encoding and spectral detection processes. Fusion weights are determined based on the signal-to-noise ratio, exposure saturation, and registration residuals of each local region, allowing regions with high data quality to contribute more to the reconstruction while suppressing noise and residual effects in regions with low data quality, thus avoiding saturation distortion in overexposed regions and noise amplification in low signal-to-noise ratio regions. The fusion weights are used to weight the differences between the intensity data predicted by the model and the measured spatial structure data, forming a weighted data consistency term, ensuring that the joint solution is dominated by reliable data. Structural constraint terms are constructed based on the edge or texture features provided by the spatial structure data, ensuring that the structural edges and texture distribution of the reconstruction results conform to the guidance of the measured data. Since the weighted data consistency term and the structural constraint term are solved jointly in the same objective function, the spatial, angular and spectral multidimensional information are jointly constrained in the same reconstruction process, which restores the light field information of the target sample and generates the meta-imaging result, thereby improving the stability, noise resistance and structural fidelity of meta-imaging reconstruction in dynamic scenes.

[0278] In one possible implementation, the multidimensional complementary information representation and reconstruction module is further configured to perform digital adaptive optics correction on the recovered optical field information, wherein the digital adaptive optics correction includes:

[0279] Based on the data corresponding to different observation angles in the light field information, the wavefront information at different field of view positions is estimated;

[0280] Based on the wavefront information and the spectral response of the corresponding spectral channel, the spatial variation point spread function at different field positions is determined, and the corresponding local optical model is established.

[0281] Based on the spatial structure data, the spatiotemporal variation data, and the local signal-to-noise ratio at each field of view, the corresponding local correction parameters are determined, including regularization coefficients for local deconvolution.

[0282] Based on the local optical model and the local correction parameters, local deconvolution is performed at each field of view to compensate for optical aberrations, resulting in a corrected meta-imaging result.

[0283] For example, performing digital adaptive optics correction on the recovered light field information can divide the imaging field of view into local regions with overlapping boundaries and establish a suitable local optical model in each region.

[0284] As an example, regions with reliable structures can be selected from multi-view data obtained after motion compensation. Combining the calibrated angular geometry with the geometrical parallax determined by digital refocusing or depth estimation, local sub-aperture displacements that cannot be explained by the target geometry are calculated. The local wavefront is then fitted using the calibration relationship between the sub-aperture position and the wavefront slope.

[0285] The optical path difference of the local wavefront can be expressed as a linear combination of a finite number of Zernike polynomials:

[0286]

[0287] in, The optical path difference of the local wavefront in the r-th local region can be represented by this expression. It can represent normalized aperture coordinates; It can represent basis functions selected according to explicitly chosen numbers and normalization rules; It can represent the m-th coefficient to be fitted in the r-th local region; It can represent the term number of a Zernike polynomial; This can represent the sequence number of a local region. Weighted least-squares fitting is performed based on the residual displacement and confidence level of each sub-aperture. The global phase constant, which cannot be determined by intensity observations, is fixed at zero; components already treated as global geometric shifts are not repeatedly compensated for in local aberration correction.

[0288] As an example, for a deep tissue region at a depth of approximately 50 micrometers in the field of view, the estimated local wavefront can be represented as a linear combination of Zernike polynomials, fitted to obtain the defocus coefficient Z4 = +0.65λ, astigmatism coefficient Z5 = -0.72λ, coma coefficient Z7 = +0.41λ, and spherical aberration coefficient Z9 = +0.28λ, where λ can represent the wavelength of light. In the spatially varying point spread function generated based on the above coefficients, the defocus component causes the point spread function to have a circular diffused halo, the astigmatism component stretches the point spread function by approximately 1.2 times in the horizontal direction, the coma component causes the point spread function to have an asymmetrical tail shape, and the spherical aberration component enhances the side lobes of the point spread function by approximately 15%.

[0289] The wavefront estimation requires sufficient effective observation angles and spatial structure data. For regions where texture is insufficient, occlusion is significant, or strong scattering causes the model to fail, validated parameters from the neighborhood are used, the previous effective parameters are retained, or no new corrections are performed for that region. At the same time, low confidence markers are set to avoid forcibly fitting higher-order aberrations from observations lacking information.

[0290] The point spread function is established based on the local wavefront, aperture transmittance, and effective spectral weights of the corresponding spectral channels. In incoherent intensity imaging and locally approximating isohalos implementations, the squares of the aperture complex amplitude Fourier transform moduli at different wavelengths can be spectrally weighted, and the resulting point spread function is normalized so that its integral is one. The phase term uses 2π / λ, where λ can represent the wavelength corresponding to the spectral channel.

[0291] Effective spectral weights are determined by the spectral channel responses and the reference emission spectrum of the selected sample signal. When the bandwidth is narrow, an equivalent model at the center wavelength can be used after evaluating the approximation error; when the bandwidth is wide or the response differences within the band are significant, multi-wavelength discrete weighting is used. Different spectral channels are established with corresponding models, and the point spread function at the same wavelength is not directly shared due to the shared joint coding state.

[0292] The local optical model describes the optical degradation that remains after joint reconstruction. If the aforementioned joint reconstruction already includes inverse compensation for some aberrations, then the equivalent point spread function of the remaining degradation is used in this step to avoid repeated deconvolution of the same degradation.

[0293] Let the local image to be corrected, g, the local point spread function, h, and the image to be restored, o, represent this relationship as follows:

[0294]

[0295] in, It can represent a local image to be corrected. It can represent the local point spread function. The symbol can represent the restored image obtained by solving the image to be restored; the asterisk can represent a convolution operation; It can represent a local regularization term. This can represent the local regularization coefficient. The local deconvolution is used to compensate for the optical aberrations characterized by the local optical model.

[0296] The local regularization coefficient is determined based on spatial structure data, spatiotemporal variation data, and local signal-to-noise ratio (SNR). For example, the reliability edge of the spatial structure data is used to control the spatial smoothing intensity, while spatiotemporal variation data and compensation residuals are used to measure the degree of dynamic inconsistency. When the SNR decreases or the dynamic residual increases, the stabilization constraint is increased. The parameters are adjusted within preset upper and lower limits to avoid noise amplification or over-smoothing.

[0297] Each local region retains the boundary extension used for convolution calculations, and a normalized window function is used for fusion in overlapping areas to reduce block seams. Wavefront and correction parameters are updated only when supported by valid data. The above processing yields the corrected meta-imaging results and corresponding quality labels.

[0298] In this embodiment, wavefront information at different field-of-view positions is estimated based on data corresponding to different observation angles in the light field information, allowing aberration estimation to be based on the recovered multi-angle light field information without the need for additional wavefront measurement devices. The spatial variation point spread function is determined based on the wavefront information and the spectral response of the corresponding spectral channels, and a corresponding local optical model is established, enabling the aberration differences at different field-of-view positions to be characterized separately. Local correction parameters are determined based on the spatial structure data, the spatiotemporal variation data, and the local signal-to-noise ratio at each field-of-view position, allowing the correction intensity to adaptively adjust according to the structural reliability, dynamic state, and signal-to-noise ratio of each region. Local deconvolution is performed at each field-of-view position based on the local optical model and the local correction parameters, compensating for the optical aberrations at each field-of-view position. Thus, the system digitally achieves adaptive aberration correction that varies with the field-of-view position, further reducing local blurring and distortion caused by deep tissue scattering, and further improving imaging quality based on the original imaging results.

[0299] Figure 3 This diagram illustrates a data processing flow of a meta-imaging system based on a complementary sensing neuromorphic chip according to an embodiment of the present disclosure, as shown below. Figure 3 As shown, the system sequentially performs complementary optical acquisition, complementary optical field encoding, complementary spectral allocation, and complementary sensing acquisition, followed by joint information representation and meta-imaging reconstruction. Optionally, it then performs digital adaptive optics correction and outputs the imaging results.

[0300] Figure 4 A flowchart illustrating a meta-imaging method based on a complementary sensing neuromorphic chip according to an embodiment of the present disclosure is shown, as follows: Figure 4 As shown, the method includes:

[0301] S201. Collect the signal light of the target sample to form a common intermediate image containing at least two spectral components;

[0302] S202. Based on a unified time reference, perform spatial and angular joint coding on the common intermediate image under multiple joint coding states to form a complementary coded light field;

[0303] S203. The complementary coded optical field is distributed to at least two spectral channels with different spectral responses to obtain at least two complementary coded optical signals, wherein the complementary coded optical signals distributed under the same joint coding state share the joint coding state;

[0304] S204. Using a complementary sensing neuromorphic chip with cognitive and motor guidance pathways, photoelectric acquisition is performed on the complementary coded optical signals of each pathway to obtain complementary optical field sampling data, wherein the cognitive guidance pathway outputs spatial structure data and the motor guidance pathway outputs spatiotemporal change data.

[0305] S205. Associate the spatial structure data and the spatiotemporal variation data with the joint encoding state and the spectral channel at the time of acquisition, respectively, establish the joint information expression relationship of space, angle, time and spectrum, dynamically compensate the complementary light field sampling data according to the spatiotemporal variation data, and perform meta-imaging joint reconstruction with the spatial structure data as a constraint to obtain the meta-imaging result.

[0306] The joint coding state is adaptively adjusted based on the spatiotemporal change data.

[0307] In one possible implementation, the device for collecting the signal light of the target sample includes a light-collecting objective, an imaging lens group, and a multi-band filter assembly.

[0308] The light-collecting objective lens is used to collect signal light from the same observation area of ​​the target sample through the same acquisition aperture, and the signal light contains the at least two spectral components;

[0309] The multi-band filter assembly is disposed between the light-collecting objective and the imaging lens group, and has at least two transmission bands corresponding to the at least two spectral components, for simultaneously filtering out the spectral components from the signal light.

[0310] The imaging lens group is used to make the signal light of each spectral component passing through the multi-band filter component propagate along the same imaging optical path and form the common intermediate image on the same intermediate image plane, so that the complementary optical field encoding module can perform spatial and angular joint encoding on the common intermediate image.

[0311] In one possible implementation, the device for joint spatial and angular encoding of the common intermediate image under multiple joint encoding states based on a unified time reference includes a scanning mirror, a movable encoding microlens array, an encoding controller, and a driving component, wherein the driving component is connected to the scanning mirror and the movable encoding microlens array respectively.

[0312] The joint encoding state includes the angular deflection state of the scanning mirror and the spatial displacement state of the movable encoding microlens array;

[0313] The encoding controller is used to control the scanning mirror and the movable encoding microlens array to form corresponding joint encoding states based on the unified time reference, and to determine the duration of each joint encoding state.

[0314] The encoding controller is also used to receive the spatiotemporal change data during the period when the sampling time interval is within the same joint encoding state, determine the motion amplitude of the target sample based on the spatiotemporal change data, and adjust the combination, execution order or holding duration of subsequent joint encoding states based on the motion amplitude.

[0315] In one possible implementation, the device for distributing the complementary coded light field to at least two spectral channels with different spectral responses includes a beam-splitting component and a spectral filtering component and a relay imaging component respectively disposed in each of the spectral channels.

[0316] The beam splitter is used to distribute the complementary coded light field to each of the spectral channels, and at least two of the spectral filtering components have different transmission band ranges or relative transmittance between bands.

[0317] Each complementary coded optical signal corresponds to the same observation area of ​​the target sample, and is imaged onto a non-overlapping photosensitive area on the complementary sensing neuromorphic chip via the corresponding relay imaging component. Each photosensitive area is associated with a corresponding spectral channel.

[0318] For each photosensitive area, the cognitive guidance pathway outputs spatial structure data of the corresponding spectral channel, and the motion guidance pathway outputs spatiotemporal variation data of the corresponding spectral channel.

[0319] In one possible implementation, the cognitive guidance pathway and the motion guidance pathway are synchronously acquired based on the unified time reference;

[0320] The cognitive guidance pathway outputs the spatial structure data at a first sampling frequency, and the spatial structure data is intensity frame data characterizing the incident light intensity and its spatial distribution.

[0321] The motion guidance path outputs the spatiotemporal variation data at a second sampling frequency, which is higher than the first sampling frequency. The spatiotemporal variation data includes spatial difference data and temporal difference data.

[0322] The brain-like complementary perception module is also used to record the exposure time interval for each frame of spatial structure data based on the unified time reference, record the sampling time interval for each group of spatiotemporal change data, and associate multiple groups of spatiotemporal change data whose sampling time intervals are located in the same exposure time interval with the corresponding spatial structure data.

[0323] In one possible implementation, associating the spatial structure data and the spatiotemporal variation data with the joint encoding state and corresponding spectral channels at the time of acquisition, respectively, to establish a joint information representation relationship of space, angle, time, and spectrum, includes:

[0324] Based on the unified time reference, the corresponding sampling time, joint coding status identifier, spectral channel identifier, and sensing path identifier are associated with the complementary light field sampling data.

[0325] Based on the angular deflection state of the scanning mirror, the spatial displacement state of the movable coded microlens array, and the pre-calibrated mapping relationship between the microlens unit and the imaging pixel, the spatial sampling position and observation angle corresponding to the complementary light field sampling data are determined.

[0326] Based on the pre-defined coordinate transformation relationship between different spectral channels and the spatial mapping relationship between the cognitive guidance path and the motion guidance path, the spatial sampling position is transformed to a unified object coordinate system;

[0327] Based on the sampling time and the converted spatial sampling location, the spatial structure data and the spatiotemporal change data are uniformly associated with the corresponding observation angle, joint coding state, spectral channel and sensing pathway to establish the joint information expression relationship.

[0328] In one possible implementation, the dynamic compensation includes:

[0329] Based on the spatiotemporal change data collected during each joint coding state, and combined with the joint information expression relationship, the local motion field and dynamic perturbation parameters of the imaging field of view of the target sample are estimated, wherein the local motion field characterizes the motion differences between different local regions, and the dynamic perturbation parameters characterize the overall offset of the imaging field of view.

[0330] Based on the local motion field, the dynamic disturbance parameters, and the time interval between each sampling time and the predetermined reference time, determine the motion transformation relationship of the complementary optical field sampling data corresponding to each joint coding state to the predetermined reference time;

[0331] By combining the spatial sampling position and observation angle corresponding to each joint coding state, and while retaining the observation angle identifier, spectral channel identifier and joint coding state identifier corresponding to each sampling data, the complementary light field sampling data corresponding to different joint coding states are time-aligned and spatially registered according to the motion transformation relationship to obtain the compensated light field sampling data.

[0332] In one possible implementation, the meta-imaging joint reconstruction includes:

[0333] Based on the spatial structure data in the compensated light field sampling data and the corresponding joint coding state and spectral channel, a spatial and angular joint reconstruction model is established. The spatial and angular joint reconstruction model includes the spatial and angular sampling relationship corresponding to each joint coding state and the spectral response of each spectral channel.

[0334] The fusion weights are determined based on the signal-to-noise ratio, exposure saturation, and registration residuals after dynamic compensation in each local region of data corresponding to different joint coding states and different spectral channels.

[0335] Using the fusion weights, the differences between the intensity data predicted by the spatial and angular joint reconstruction model and the spatial structure data in the compensated light field sampling data are weighted to form a weighted data consistency term;

[0336] Based on the edge or texture features provided by the spatial structure data, a structural constraint term is constructed. The objective function composed of the weighted data consistency term and the structural constraint term is jointly solved to recover the light field information of the target sample and generate the meta-imaging result.

[0337] In one possible implementation, the method further includes performing digital adaptive optics correction on the recovered light field information, the digital adaptive optics correction comprising:

[0338] Based on the data corresponding to different observation angles in the light field information, the wavefront information at different field of view positions is estimated;

[0339] Based on the wavefront information and the spectral response of the corresponding spectral channel, the spatial variation point spread function at different field positions is determined, and the corresponding local optical model is established.

[0340] Based on the spatial structure data, the spatiotemporal variation data, and the local signal-to-noise ratio at each field of view, the corresponding local correction parameters are determined, including regularization coefficients for local deconvolution.

[0341] Based on the local optical model and the local correction parameters, local deconvolution is performed at each field of view to compensate for optical aberrations, resulting in a corrected meta-imaging result.

[0342] It should be noted that although the technical solution of this disclosure has been introduced above using two spectral channels and multiple joint coding states as an example, those skilled in the art will understand that this disclosure is not limited to this. In fact, the number of spectral channels and the transmission band of each spectral channel, the number and combination of joint coding states, the sampling frequency of each data, the movement mode of the scanning mirror and the movable coding microlens array, and the output type of the meta-imaging results (e.g., two-dimensional structural images, digital refocusing images, multi-focal plane images, depth images, or three-dimensional volume data) can all be flexibly set according to the actual application scenario, as long as each complementary coded optical signal shares the same joint coding state and data is output by the cognitive guidance path and the motion guidance path respectively in each photosensitive area.

[0343] In some embodiments, the functions or modules of the apparatus provided in this disclosure can be used to perform the methods described in the above method embodiments. The specific implementation can be referred to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0344] This disclosure also provides an electronic device, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the above method.

[0345] This disclosure also provides a non-volatile computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the above-described method.

[0346] This disclosure also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the above-described method.

[0347] Figure 5 This is a block diagram illustrating a meta-imaging device based on a complementary sensing neuromorphic chip, according to an exemplary embodiment. For example, device 1900 can be provided as a server or terminal device. (Refer to...) Figure 5The apparatus 1900 includes a processing component 1922, which further includes one or more processors, and memory resources represented by memory 1932 for storing instructions, such as application programs, that can be executed by the processing component 1922. The application programs stored in memory 1932 may include one or more modules, each corresponding to a set of instructions. Furthermore, the processing component 1922 is configured to execute instructions to perform the methods described above.

[0348] Device 1900 may also include a power supply component 1926 configured to perform power management of device 1900, a wired or wireless network interface 1950 configured to connect device 1900 to a network, and an input / output interface 1958 (I / O interface). Device 1900 can operate on an operating system, such as Windows Server, stored in memory 1932. TM macOS X TM Unix TM Linux TM FreeBSD TM Or similar.

[0349] In an exemplary embodiment, a non-volatile computer-readable storage medium is also provided, such as a memory 1932 including computer program instructions that can be executed by a processing component 1922 of the device 1900 to perform the above-described method.

[0350] Computer-readable storage media can be tangible devices capable of holding and storing programs / instructions used by instruction execution devices. Computer-readable storage media can be, for example—but not limited to—electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combination of the foregoing. The computer-readable storage media used herein are not to be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or electrical signals transmitted through wires.

[0351] The computer program (or computer-readable program instructions) described herein can be downloaded from a computer-readable storage medium to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage medium in the respective computing / processing device.

[0352] The computer program (or computer program instructions) used to perform the operations of this disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The computer-readable program instructions may execute entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry, such as programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), is personalized by utilizing state information from the computer-readable program instructions to implement various aspects of this disclosure.

[0353] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0354] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processor of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner; thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0355] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0356] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those shown in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0357] The various embodiments of this disclosure have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or technical improvements to the embodiments in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A meta-imaging system based on a complementary sensing neuromorphic chip, characterized in that, include: A complementary optical acquisition module is used to collect the signal light of the target sample and form a common intermediate image containing at least two spectral components; The complementary light field coding module is used to perform spatial and angular joint coding of the common intermediate image under multiple joint coding states based on a unified time reference, so as to form a complementary coded light field. A complementary spectral allocation module is used to allocate the complementary coded light field to at least two spectral channels with different spectral responses to obtain at least two complementary coded light signals, wherein the complementary coded light signals allocated under the same joint coding state share the joint coding state; The brain-like complementary sensing module includes a complementary sensing brain-like chip with a cognitive guidance pathway and a motion guidance pathway, used to perform photoelectric acquisition on each of the complementary coded optical signals to obtain complementary optical field sampling data, wherein the cognitive guidance pathway outputs spatial structure data and the motion guidance pathway outputs spatiotemporal change data; The multidimensional complementary information expression and reconstruction module is used to associate the spatial structure data and the spatiotemporal variation data with the joint encoding state and the spectral channel at the time of acquisition, respectively, to establish the joint information expression relationship of space, angle, time and spectrum, to dynamically compensate the complementary light field sampling data according to the spatiotemporal variation data, and to perform meta-imaging joint reconstruction with the spatial structure data as a constraint to obtain the meta-imaging result. The complementary optical field coding module adaptively adjusts the subsequent joint coding state based on the spatiotemporal change data.

2. The system according to claim 1, characterized in that, The complementary optical acquisition module includes a light-collecting objective lens, an imaging lens group, and a multi-band filter assembly. The light-collecting objective lens is used to collect signal light from the same observation area of ​​the target sample through the same acquisition aperture, and the signal light contains the at least two spectral components; The multi-band filter assembly is disposed between the light-collecting objective and the imaging lens group, and has at least two transmission bands corresponding to the at least two spectral components, for simultaneously filtering out the spectral components from the signal light. The imaging lens group is used to make the signal light of each spectral component passing through the multi-band filter component propagate along the same imaging optical path and form the common intermediate image on the same intermediate image plane, so that the complementary optical field encoding module can perform spatial and angular joint encoding on the common intermediate image.

3. The system according to claim 1, characterized in that, The complementary light field encoding module includes a scanning mirror, a movable encoding microlens array, an encoding controller, and a driving component, wherein the driving component is connected to the scanning mirror and the movable encoding microlens array, respectively. The joint encoding state includes the angular deflection state of the scanning mirror and the spatial displacement state of the movable encoding microlens array; The encoding controller is used to control the scanning mirror and the movable encoding microlens array to form corresponding joint encoding states based on the unified time reference, and to determine the duration of each joint encoding state. The encoding controller is also used to receive the spatiotemporal change data during the period when the sampling time interval is within the same joint encoding state, determine the motion amplitude of the target sample based on the spatiotemporal change data, and adjust the combination, execution order or holding duration of subsequent joint encoding states based on the motion amplitude.

4. The system according to claim 1, characterized in that, The complementary spectral allocation module includes a beam splitting component and a spectral filtering component and a relay imaging component respectively disposed in each of the spectral channels; The beam splitter is used to distribute the complementary coded light field to each of the spectral channels, and at least two of the spectral filtering components have different transmission band ranges or relative transmittance between bands. Each complementary coded optical signal corresponds to the same observation area of ​​the target sample, and is imaged onto a non-overlapping photosensitive area on the complementary sensing neuromorphic chip via the corresponding relay imaging component. Each photosensitive area is associated with a corresponding spectral channel. For each photosensitive area, the cognitive guidance pathway outputs spatial structure data of the corresponding spectral channel, and the motion guidance pathway outputs spatiotemporal variation data of the corresponding spectral channel.

5. The system according to claim 1, characterized in that, The cognitive guidance pathway and the motion guidance pathway are synchronously acquired based on the unified time reference; The cognitive guidance pathway outputs the spatial structure data at a first sampling frequency, and the spatial structure data is intensity frame data characterizing the incident light intensity and its spatial distribution. The motion guidance path outputs the spatiotemporal variation data at a second sampling frequency, which is higher than the first sampling frequency. The spatiotemporal variation data includes spatial difference data and temporal difference data. The brain-like complementary perception module is also used to record the exposure time interval for each frame of spatial structure data based on the unified time reference, record the sampling time interval for each group of spatiotemporal change data, and associate multiple groups of spatiotemporal change data whose sampling time intervals are located in the same exposure time interval with the corresponding spatial structure data.

6. The system according to claim 3, characterized in that, The step of associating the spatial structure data and the spatiotemporal variation data with the joint encoding state and spectral channel at the time of acquisition, respectively, to establish the joint information expression relationship of space, angle, time, and spectrum, includes: Based on the unified time reference, the corresponding sampling time, joint coding status identifier, spectral channel identifier, and sensing path identifier are associated with the complementary light field sampling data. Based on the angular deflection state of the scanning mirror, the spatial displacement state of the movable coded microlens array, and the pre-calibrated mapping relationship between the microlens unit and the imaging pixel, the spatial sampling position and observation angle corresponding to the complementary light field sampling data are determined. Based on the pre-defined coordinate transformation relationship between different spectral channels and the spatial mapping relationship between the cognitive guidance path and the motion guidance path, the spatial sampling position is transformed to a unified object coordinate system; Based on the sampling time and the converted spatial sampling location, the spatial structure data and the spatiotemporal change data are uniformly associated with the corresponding observation angle, joint coding state, spectral channel and sensing pathway to establish the joint information expression relationship.

7. The system according to claim 6, characterized in that, The dynamic compensation includes: Based on the spatiotemporal change data collected during each joint coding state, and combined with the joint information expression relationship, the local motion field and dynamic perturbation parameters of the imaging field of view of the target sample are estimated, wherein the local motion field characterizes the motion differences between different local regions, and the dynamic perturbation parameters characterize the overall offset of the imaging field of view. Based on the local motion field, the dynamic disturbance parameters, and the time interval between each sampling time and the predetermined reference time, determine the motion transformation relationship of the complementary optical field sampling data corresponding to each joint coding state to the predetermined reference time; By combining the spatial sampling position and observation angle corresponding to each joint coding state, and while retaining the observation angle identifier, spectral channel identifier and joint coding state identifier corresponding to each sampling data, the complementary light field sampling data corresponding to different joint coding states are time-aligned and spatially registered according to the motion transformation relationship to obtain the compensated light field sampling data.

8. The system according to claim 7, characterized in that, The meta-imaging joint reconstruction includes: Based on the spatial structure data in the compensated light field sampling data and the corresponding joint coding state and spectral channel, a spatial and angular joint reconstruction model is established. The spatial and angular joint reconstruction model includes the spatial and angular sampling relationship corresponding to each joint coding state and the spectral response of each spectral channel. The fusion weights are determined based on the signal-to-noise ratio, exposure saturation, and registration residuals after dynamic compensation in each local region of data corresponding to different joint coding states and different spectral channels. Using the fusion weights, the differences between the intensity data predicted by the spatial and angular joint reconstruction model and the spatial structure data in the compensated light field sampling data are weighted to form a weighted data consistency term; Based on the edge or texture features provided by the spatial structure data, a structural constraint term is constructed. The objective function composed of the weighted data consistency term and the structural constraint term is jointly solved to recover the light field information of the target sample and generate the meta-imaging result.

9. The system according to claim 8, characterized in that, The multidimensional complementary information representation and reconstruction module is further used to perform digital adaptive optics correction on the recovered light field information, wherein the digital adaptive optics correction includes: Based on the data corresponding to different observation angles in the light field information, the wavefront information at different field of view positions is estimated; Based on the wavefront information and the spectral response of the corresponding spectral channel, the spatial variation point spread function at different field positions is determined, and the corresponding local optical model is established. Based on the spatial structure data, the spatiotemporal variation data, and the local signal-to-noise ratio at each field of view, the corresponding local correction parameters are determined, including regularization coefficients for local deconvolution. Based on the local optical model and the local correction parameters, local deconvolution is performed at each field of view to compensate for optical aberrations, resulting in a corrected meta-imaging result.

10. A meta-imaging method based on a complementary sensing neuromorphic chip, characterized in that, The method includes: The signal light from the target sample is collected to form a common intermediate image containing at least two spectral components; Based on a unified time reference, the common intermediate image is jointly encoded in space and angle under multiple joint encoding states to form a complementary encoded light field. The complementary coded optical field is assigned to at least two spectral channels with different spectral responses to obtain at least two complementary coded optical signals, wherein the complementary coded optical signals assigned under the same joint coding state share the joint coding state; Using a complementary sensory neuromorphic chip with cognitive and motor guidance pathways, photoelectric acquisition is performed on the complementary coded optical signals of each pathway to obtain complementary optical field sampling data. The cognitive guidance pathway outputs spatial structure data, and the motor guidance pathway outputs spatiotemporal variation data. The spatial structure data and the spatiotemporal variation data are respectively associated with the joint encoding state and the corresponding spectral channel at the time of acquisition, and a joint information expression relationship of space, angle, time and spectrum is established. The complementary light field sampling data is dynamically compensated according to the spatiotemporal variation data, and the meta-imaging joint reconstruction is performed with the spatial structure data as a constraint to obtain the meta-imaging result. The joint coding state is adaptively adjusted based on the spatiotemporal change data.

Images