A method for image compression sensing signal processing of integrated metasurface CIS wafer bonding

By integrating metasurface CIS wafer bonding and photoelectric co-mapping, the problems of bandwidth occupation and computational delay in high-dimensional pixel data transmission in image processing systems are solved, enabling efficient image feature extraction and accurate deep neural network inference.

CN123123746APending Publication Date: 2026-11-17SHIFANG DINGFENG PACKAGING & PRINTING CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202611364675.3
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2026-09-04
Publication Date
2026-11-17

AI Technical Summary

Technical Problem

In existing image processing systems, the high-dimensional pixel data output by image sensors increases the data transmission bandwidth usage of the analog-to-digital conversion interface and the system power consumption. At the same time, the separation of the optical sensing link from the back-end neural network calculation leads to computational delay and feature extraction errors.

Method used

By integrating metasurface CIS wafer bonding, a photoelectric joint mapping equation is constructed to generate a set of non-negative analog mask tensors and an affine calibration parameter set. Combined with a programmable sample-and-hold circuit and a computing module, the pixel voltage signal is weighted and differentially accumulated to compress and reduce the dimension of the output to simulate a one-dimensional feature vector, and a one-dimensional feature affine reconstruction operation is performed in the backend.

Benefits of technology

This reduces the data transmission bandwidth usage of the system's analog-to-digital conversion interface, decreases the amount of data read and written by the digital backend and the overall power consumption of the system, while maintaining the effectiveness of image feature extraction and the inference accuracy of deep neural networks.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN123123746A_ABST
    Figure CN123123746A_ABST
Patent Text Reader

Abstract

This invention relates to the field of image signal processing technology and discloses an image compression sensing signal processing method integrating metasurface CIS wafer bonding. The method includes: constructing a photoelectric joint mapping equation, solving it using an alternating optimization algorithm and hardware-sensory fine-tuning to generate a set of non-negative analog mask tensors, a set of affine calibration parameters, and fine-tuned back-end neural network model weights, which are then deployed to an image sensor module and a computing module; a broadband light field passes through the metasurface to generate diffraction separation, which is converted into charge distribution and pixel voltage signals in the image sensor module, forming a pre-nonlinear mapping; the image sensor module controls a programmable sample-and-hold circuit to perform weighted allocation and differential accumulation operations on the pixel voltage signals, compressing and reducing the dimensionality to output an analog one-dimensional feature vector, which is then quantized to output a digital one-dimensional feature vector; and the computing module extracts the affine parameters, performs reconstruction operations, generates a correction feature tensor, and passes it to the back-end network for inference.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention relates to the field of image signal processing technology, specifically to an image compression sensing signal processing method integrating metasurface CIS wafer bonding. Background Technology

[0002] In image signal processing systems, conventional hardware architectures employ a data flow model that separates optical acquisition, image sensor photoelectric analog-to-digital conversion, and back-end processor digital computation. The image sensor receives the ambient light field and converts it into full-resolution, high-dimensional pixel data, then sends the complete image frame to the back-end computing unit via a data transmission interface. The back-end computing unit deploys a deep neural network, which receives the high-dimensional pixel data and performs multi-level feature extraction and task inference operations in the digital domain.

[0003] As machine vision tasks demand increasingly higher perceptual resolution, this hardware architecture that separates sensing and computation exposes corresponding physical bottlenecks. The high-dimensional data output from the image sensor consumes significant system power during transmission through a column-parallel analog-to-digital converter, and the continuous high-throughput data stream imposes high bandwidth requirements on the data transmission interfaces between hardware components. Furthermore, a large number of raw pixels containing background noise are unfiltered and transferred to the digital computing chip before network forward computation, increasing the memory read / write burden and overall system latency, making it difficult to meet the high-energy-efficiency processing requirements of edge devices in real-time environments.

[0004] To alleviate data transmission bandwidth pressure, some industry solutions attempt to move computational tasks forward to the image sensor. However, due to the objective limitation that the underlying analog circuits can only represent non-negative physical weights, directly mapping standard neural network operators at the sensor's analog front end will introduce physical truncation errors and zero-point shifts. Furthermore, the spatial dispersion modulation capabilities of the front-end optical components and the inherent physical nonlinear response characteristics of the photoelectric conversion process have not been effectively integrated into the system's feature extraction computation chain.

[0005] If the signal truncation error in the front-end physical domain cannot be effectively compensated by the back-end digital domain, it will directly cause the feature extraction feature space of the system to shift, making it difficult for the system to maintain the original inference accuracy of the target deep neural network while reducing data transmission power consumption. Summary of the Invention

[0006] To address the shortcomings of existing technologies, this invention provides an image compression sensing signal processing method integrating metasurface CIS wafer bonding. This method solves the problem in existing image processing systems where the high-dimensional pixel data output by the image sensor increases the data transmission bandwidth usage of the analog-to-digital conversion interface and system power consumption. Furthermore, the separation of the optical sensing stage from the backend neural network computation, requiring data to flow between different hardware components, increases the computational latency for real-time feature extraction.

[0007] To achieve the above objectives, the present invention provides the following technical solution: A method for image compression sensing signal processing of integrated metasurface CIS wafer bonding includes constructing a photoelectric joint mapping equation, solving it through an alternating optimization algorithm and fine-tuning it with hardware perception to generate a set of non-negative analog mask tensors, a set of affine calibration parameters, and fine-tuned back-end neural network model weights. Write the set of non-negative analog mask tensors into the mask register array of the image sensor module, set the working timing of the programmable sample-and-hold circuit, and deploy the set of affine calibration parameters and the fine-tuned back-end neural network model weights to the computing module. A broadband light field passes through the metasurface module, generating spatially differentiated diffraction distribution separation. This is then converted into charge distribution and pixel voltage signals in the image sensor module, forming a pre-nonlinear mapping. The image sensor module controls the programmable sample-and-hold circuit to perform weighted allocation and differential accumulation operations on the pixel voltage signal, compress and reduce the dimension to output an analog one-dimensional feature vector, and then quantize it to output a digital one-dimensional feature vector. The calculation module extracts the affine calibration parameter set and performs a one-dimensional feature affine reconstruction operation on the digital one-dimensional feature vector to generate a correction feature tensor, which is then passed to the backend deep neural network to perform subsequent task inference operations.

[0008] Furthermore, the construction of the photoelectric joint mapping equation includes: The feature extraction layer operator of the back-end deep neural network is extracted, and the inherent square law transformation characteristics of the image sensor module are introduced as a front-end physical nonlinearity to establish a hardware equivalent response model. The physical limitations of the image sensor module are combined with the digital compensation capability of the computing module to construct a global error model that includes a mean square error objective function; The mean square error objective function is jointly determined by the ideal one-dimensional eigenvector expected by the back-end network, the input complex amplitude vector, the one-dimensional affine scale gain vector, the DC bias vector, the positive and negative non-negative analog mask matrices, the fixed spatial transfer function matrix, and the physical square-law response function of the photoelectric conversion link, thereby forming the photoelectric joint mapping equation.

[0009] Furthermore, the solution obtained through the alternating optimization algorithm includes: Under the conditions of fixed optical transfer function and physical square-law response, an alternating optimization iteration is performed on the programmable analog mask and the set of affine calibration parameters; By keeping the affine calibration parameter set unchanged, and based on the physical constraint that the analog weighted capacitors only represent non-negative weights, the positive and negative polarity non-negative analog mask matrices that minimize the total mean square error are solved. By keeping the positive and negative polarity non-negative analog mask matrices unchanged, the affine calibration parameter set that minimizes the error is calculated using a multivariate linear regression method. This set is used to correct the zero-point offset caused by forced non-negative projection and physical truncation.

[0010] Furthermore, the step of writing the set of non-negative analog mask tensors into the mask register array of the image sensor module includes: Perform binary quantization on the set of non-negative analog mask tensors; The address mapping rule using a multi-dimensional joint addressing mode is used to write the quantized mask data into the mask register array; The hardware addressing index of the multidimensional joint addressing mode covers the index of output features, wavelength channel index, pixel row address, pixel column address, and polarity index.

[0011] Furthermore, the step of deploying the affine calibration parameter set and the fine-tuned backend neural network model weights to the computing module includes: The affine calibration parameter set is loaded into the built-in parameter cache, and data memory alignment and allocation operations based on the output feature index are performed. The affine calibration parameter set includes one-dimensional affine scale gain values ​​and DC bias values. Based on the total number of output feature elements, the one-dimensional affine scale gain value and the DC bias value corresponding to the same feature index are merged into a continuous memory word, so that the calculation module can concurrently read the affine calibration parameters of the corresponding channel in a single instruction cycle when receiving digital features.

[0012] Furthermore, the broadband light field passing through the metasurface module generates spatially differentiated diffraction distribution separation, including: The phase delay modulation of the subwavelength nanoantenna array of the metasurface module is used to guide light fields of different wavelengths to form spatially differentiated diffraction distributions that can be distinguished from each other, thereby forming independent physical transmission channels. The number of photoelectrons generated in the charge distribution is proportional to the square of the modulus of the complex amplitude of the incident light field. The pre-nonlinear mapping is completed by using the square law characteristics of the bottom photoelectric absorption in conjunction with the end-to-end feature fitting parameters.

[0013] Furthermore, the image sensor module controls the programmable sample-and-hold circuit to perform weighted allocation and differential accumulation operations on the pixel voltage signal, including: A non-destructive voltage sampling mode is adopted. In the first sampling stage, the voltage signal of the entire array space is taken out and connected to the positive polarity binary weighted capacitor array for proportional scaling. The signal is then aggregated to form the column-level positive polarity part and then simulated across columns to form the global positive polarity voltage on the first integrating capacitor. In the second sampling stage, the voltage signal in the same space is taken out and connected to the negative polarity binary weighted capacitor array for proportional scaling, forming a column-level negative polarity part and accumulating across columns into the second integrating capacitor.

[0014] Furthermore, the compressed dimensionality reduction output simulates a one-dimensional feature vector, including: A differential amplifier circuit is used to perform an analog subtraction operation on the voltage signals on the first integrating capacitor and the second integrating capacitor, and outputs a differential voltage signal representing a single scalar feature element within a single cycle, which constitutes the analog one-dimensional feature vector.

[0015] Furthermore, the calculation module extracts the affine calibration parameter set and performs a one-dimensional feature affine reconstruction operation on the digital one-dimensional feature vector to generate a correction feature tensor, including: According to the feature channel index, each element in the digital one-dimensional feature vector is multiplied by the one-dimensional affine scale gain value of the corresponding channel in the affine calibration parameter set, and then added by the corresponding DC bias value in the set, thereby obtaining the output feature tensor after affine reconstruction as the correction feature tensor.

[0016] Furthermore, the step of passing the data to the backend deep neural network for subsequent task inference operations includes: The corrected feature tensor is fed into a back-end network layer with a corresponding interface for processing and remapped into an intermediate layer input feature map that is compatible with the back-end network structure. After deep semantic feature extraction is completed by feeding the backbone network, the data is further fed into the task output network, which outputs the task inference result containing the target bounding box and the category confidence.

[0017] This invention provides a method for image compression sensing signal processing using integrated metasurface CIS wafer bonding. It offers the following advantages: 1. This invention writes a set of non-negative analog mask tensors into the mask register array of the image sensor module, controlling the internal programmable sample-and-hold circuit to perform weighted allocation and differential accumulation operations on the pixel voltage signals before the analog-to-digital converter (ADC) starts. This technical feature allows high-dimensional pixel data to be directly compressed and reduced to a low-dimensional analog one-dimensional feature vector in the analog circuit domain, followed by single-feature quantization output. This reduces the data transmission bandwidth occupied by the system's ADC interface, while also reducing the amount of data read / write at the digital backend and the overall system power consumption.

[0018] 2. This invention utilizes a metasurface module to perform spatial modulation on a broadband optical field, generating a spatially differentiated diffraction distribution. Combined with the inherent square-law response characteristics of the photoelectric absorption stage of the image sensor module, it directly achieves the pre-nonlinear mapping of spatial optical field features at the optical physics level. This structure transfers the high-dimensional feature extraction and activation operators, which rely on digital computing modules in the target deep neural network, to the physical sensing layer for execution. While ensuring the effectiveness of image feature extraction, it reduces the number of forward inference nodes in the digital computing module.

[0019] 3. This invention constructs a global error model that incorporates the physical limitations of the image sensor hardware and the compensation capabilities of the digital computing module. Corresponding hardware deployment parameters are generated through alternating optimization algorithms and hardware-aware fine-tuning. The computing module performs a one-dimensional feature affine reconstruction operation on the digital one-dimensional feature vector based on a cached set of affine calibration parameters, thereby calculating and generating a corrected feature tensor. This operation corrects the computational errors and zero-point offsets caused by the forced non-negative projection and physical truncation of the front-end analog circuit, maintaining the accuracy of the back-end deep neural network task inference output while adapting to the physical characteristics of the underlying analog circuit. Attached Figure Description

[0020] Figure 1 This is a schematic diagram of the structure of a signal processing system provided in an embodiment of the present invention; Figure 2 This is an overall flowchart of the image compression sensing signal processing method provided in an embodiment of the present invention; Figure 3 A flowchart illustrating the process of constructing and alternating optimization of the photoelectric joint mapping equation provided in this embodiment of the invention; Figure 4 A flowchart illustrating the hardware parameter writing and timing configuration process provided in this embodiment of the invention; Figure 5 A flowchart of the wavelength dispersion coupling forward propagation and square-law mapping process provided in an embodiment of the present invention; Figure 6 This is a flowchart of the simulation domain difference calculation and compressed feature generation process provided in an embodiment of the present invention; Figure 7 A flowchart of the digital affine reconstruction and deep network docking process provided in an embodiment of the present invention; Figure 8 This is a numerical flow diagram of image compression sensing and feature calculation according to the present invention; Figure 9 This is a graph showing the convergence state of the mean square error and the change in inference accuracy during the hardware perception fine-tuning process of the present invention. Figure 10 This is a data comparison chart showing the energy consumption distribution and interface bandwidth usage between the conventional all-digital processing system and the present invention. Detailed Implementation

[0021] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0022] See attached document Figure 1 The present invention provides a signal processing system, which may include a metasurface module, an image sensor module, and a computing module.

[0023] The metasurface module is located on the light-incident side of the front end and is bonded to the surface of the integrated circuit chip using a wafer-level packaging process. This metasurface module has wavelength-dependent characteristics and is used to spatially modulate the incident light field to provide a fixed high-dimensional broadband transfer function.

[0024] The image sensor module is positioned behind the transmission optical path of the metasurface module. The image sensor module includes a color filter array, a programmable sample-and-hold circuit, and a mask register array. The image sensor module converts the incident light field into a charge distribution. Furthermore, the image sensor module utilizes the programmable sample-and-hold circuit to perform array-level weighted differential combining operations on the pixel signals before the analog-to-digital converter operates, directly outputting a compressed and dimensionality-reduced one-dimensional feature vector.

[0025] The computing module is electrically connected to the output of the image sensor module. The computing module has a built-in digital affine compensation parameter buffer. It receives the one-dimensional feature vector output from the image sensor module, performs affine correction calculations, and runs the forward inference computation of the deep neural network.

[0026] See attached document Figure 2 This invention provides an image compression sensing signal processing method, comprising the following steps: Step S100: Extract the feature extraction layer operator of the target deep neural network and construct the photoelectric joint mapping equation. Use an alternating optimization algorithm to solve for the initial parameters based on optical physical constraints, and after hardware-aware fine-tuning, generate the final deployed non-negative analog mask tensor set, affine calibration parameter set, and fine-tuned backend neural network model weights. The dimension of the non-negative analog mask tensor set matches the control interface of the image sensor module, and the dimension of the affine calibration parameter set matches the processing interface of the computation module.

[0027] In step S200, the set of non-negative analog mask tensors is written into the mask register array inside the image sensor module, and the positive and negative polarity operating timing of the programmable sample-and-hold circuit in the image sensor module is set according to the set of non-negative analog mask tensors. Simultaneously, the set of affine calibration parameters is written into the parameter buffer of the calculation module, and the fine-tuned back-end neural network model weights are deployed to the digital back-end model storage area of ​​the calculation module.

[0028] Step S300: Receive the broadband light field of the target scene. The broadband light field passes through the metasurface module, which, based on wavelength-dependent characteristics, causes spatially differentiated diffraction distributions of light fields of different wavelengths. After reaching the photosensitive surface of the image sensor module, the broadband light field is converted into a charge distribution. The photoelectric conversion process exhibits square-law response characteristics, thereby forming a pre-nonlinear mapping of the spatial light field characteristics.

[0029] In step S400, the image sensor module performs physical channel demultiplexing using the color filter array. Before the column-parallel analog-to-digital converter is started, the image sensor module controls the programmable sample-and-hold circuit to perform analog domain weighted allocation and differential accumulation operations on the spatially distributed pixel voltage signals according to the timing configuration in the mask register array.

[0030] The image sensor module directly compresses and reduces the dimensionality of high-dimensional light field data into a sequence of outputs containing a preset total number of feature elements. The analog-to-digital converter (ADC) uses a single-feature-element real-time quantization mode, immediately quantizing and buffering the individual analog features generated in each computation cycle, accumulating and outputting a complete digital one-dimensional feature vector, in which the total number of output feature elements is [number missing]. It is much smaller than the total number of input pixels.

[0031] In step S500, the calculation module receives the digital one-dimensional feature vector, extracts the affine calibration parameter set from the parameter buffer, performs a one-dimensional feature affine reconstruction operation on it, and calculates and generates a correction feature tensor. The calculation module then passes the correction feature tensor to the backend deep neural network for subsequent task inference operations.

[0032] See attached document Figure 3 In some embodiments, the specific implementation process of constructing and alternating optimization of the photoelectric joint mapping equation in step S100 above may include the following steps S110 to S130.

[0033] Step S110: Extract the feature extraction layer operator of the target deep neural network and establish a hardware equivalent response model that includes the underlying physical characteristics.

[0034] Optionally, the target deep neural network is a model for image classification or object detection. Such models have already undergone pre-training on a digital computing platform using labeled datasets during the pre-deployment phase. The required input data dimension for the network is characterized by containing high... ,width and number of wavelength channels The three-dimensional spatial tensor corresponds to a broadband image field that has been phase-coded by an active coherent light source or a spatial light modulator. Each wavelength component of the light field signal incident on the sensor is physically independent and exhibits a complex amplitude with a definite phase.

[0035] The computation module reads the target model structure and establishes a hardware equivalent response model based on the physical diffraction characteristics of the metasurface module and the photoelectric conversion characteristics of the image sensor module. The computation module incorporates the inherent square-law conversion characteristics of the image sensor module as a pre-existing physical nonlinearity into the hardware equivalent response model.

[0036] Since the number of photoelectrons detected and generated by the image sensor module is proportional to the square of the modulus of the complex amplitude of the light field, the process no longer relies on the standard activation function in the digital domain, but instead uses the squared modulus response of the underlying optoelectronic physics as the basis for the overall nonlinear mapping.

[0037] Step S120: Construct the objective function of the digital joint and form the photoelectric joint mapping equation.

[0038] Considering that fitting the target feature space using the analog circuitry within the image sensor module would introduce truncation error and zero-point shift, the computation module combines the physical limitations of the image sensor module with the digital compensation capabilities of the computation module to construct a global error model. The established mean square error objective function... Represented as:

[0039] In this formula, This represents the objective function for the total mean square error. This represents the calibration dataset used for model compilation and deployment. Representing the The input complex amplitude vector of each wavelength channel, are positive integers and .

[0040] The desired length of the target network is the total number of feature elements. The ideal one-dimensional feature vector. This ideal feature vector is derived from the original high-dimensional feature tensor of the target network. After reference linear projection operator Dimensional reduction expansion yields the result that satisfies... .

[0041] The total number of wavelength channels demultiplexed by the image sensor module corresponds to the number of channels in the input data mentioned above. and They represent lengths of The one-dimensional affine scale gain vector and DC bias vector, the elements of which are arranged according to the index of the output feature element; This represents element-wise multiplication.

[0042] and Representing channels The corresponding positive and negative polarity non-negative analog mask matrices; Representative metasurface module for channels The fixed-space transfer function matrix; This represents the physical square-law response function of the photoelectric conversion process.

[0043] In this photoelectric joint mapping equation, the projection operator is fixed. Together with programmable masks, they form the hardware foundation for feature extraction, enabling the overall output to approximate the target operator.

[0044] Step S130: Under the condition of hardware physical constraints, perform alternating optimization iterative calculations on the established photoelectric joint mapping equation and output the decoupled hardware deployment parameters.

[0045] Specifically, the computation module uses an alternating least squares algorithm to perform alternating optimization iterations on the programmable analog mask and digital affine parameters under the condition of fixed optical transfer function and physical square law response.

[0046] In the first sub-step of a single iteration, the computation module locks the one-dimensional affine scale gain vector. With the DC bias vector To keep this constant, the solution to the overall objective is transformed into a convex optimization problem. The computation module, based on the physical constraint that the simulated weighted capacitors in the underlying circuitry of the image sensor module only represent non-negative weights, calculates and generates a solution that minimizes the total mean square error. Minimal positive polarity mask matrix With negative polarity mask matrix This operation solves for an analog mask matrix containing only non-negative numbers to meet the write criteria of the mask register array.

[0047] In the second sub-step of a single iteration, the computation module keeps the mask matrix unchanged and uses a multivariate linear regression method to calculate and generate the affine parameters that minimize the error. and It is used to correct zero-point offsets caused by forced non-negative projection and physical truncation.

[0048] The calculation module iteratively executes the above two sub-steps, setting the descent gradient threshold to 10. -4 The total mean square error generated by two consecutive iterations. When the absolute value of the difference is less than the set gradient threshold, the model is considered to have converged, and the total mean square error is retained in the historical records. The smallest set of parameters is used as the initial deployment parameters.

[0049] Subsequently, the system enters the hardware-aware fine-tuning phase. The computation module constructs a simulation computation graph containing optical physical constraints using the calibration dataset used for model compilation as samples, while fixing the mask data in the metasurface physical parameters and initial deployment parameters. Using backpropagation algorithms and optimizers, such as adaptive moment estimation optimizers or stochastic gradient descent optimizers, the weights of the backend deep network and the set of affine calibration parameters corresponding to each feature element are jointly updated.

[0050] After fine-tuning convergence, the system exports the final set of non-negative analog mask tensors, the set of affine calibration parameters, and the weights of the fine-tuned backend neural network model, thereby completing the hardware executable parameter mapping of the deep neural network operator and the overall model parameter update.

[0051] See attached document Figure 4 In some embodiments, the specific implementation process of writing hardware parameters and configuring timing in step S200 above may include the following steps S210 to S230.

[0052] In step S210, the calculation module performs binary quantization on the set of non-negative analog mask tensors and writes the quantized mask data into the mask register array inside the image sensor module according to the address mapping rules using the internal data bus.

[0053] The calculation module uses a 4-bit or 8-bit quantization width to convert the floating-point values ​​of the mask matrix into a fixed-point digital bit stream, depending on the precision limitations of the digital-to-analog converter or control interface inside the image sensor module.

[0054] Based on this, the address mapping rule adopts a multi-dimensional joint addressing mode. The hardware addressing index of this multi-dimensional joint addressing mode covers the index of output features, wavelength channel index, pixel row address, pixel column address, and polarity index.

[0055] For the full image size is High-dimensional input direction of pixels The mapping requirement for dimensionality reduction and compression of low-dimensional feature outputs necessitates a mask register array configured for multi-cycle cyclic read mode. In the... In each feature calculation cycle are positive integers and The system outputs a fixed feature index. It sequentially reads the mask data corresponding to each wavelength channel and each spatial pixel position to complete the mask loading of a single specified output feature channel.

[0056] In step S220, the control logic unit inside the image sensor module reads the mask data stored in the mask register array and sets the positive and negative polarity operating timing of the programmable sample-and-hold circuit according to the data level state.

[0057] The programmable sample-and-hold circuit is located on the column readout bus of the image sensor module and includes a controlled multiplexer network and an integrating capacitor assembly. The control logic unit controls the combined state of the binary weighted capacitor array connected to each branch of the multiplexer network, effectively matching the numerical values ​​of the mask weights after fixed-point quantization using the charge redistribution principle. The pixel unit employs a non-destructive voltage sampling mode, ensuring that the voltage signal of the same pixel can be repeatedly sampled across multiple calculation cycles. Multiply-accumulate operations on different output feature channels.

[0058] In the first sampling phase of timing control, the capacitor array corresponding to the positive polarity mask is turned on to perform accumulation; in the second sampling phase, the capacitor array corresponding to the negative polarity mask is turned on. The above timing settings control the image sensor module to complete the selective extraction and weighted allocation of spatial pixel states.

[0059] In step S230, the calculation module loads the affine calibration parameter set into the built-in parameter cache and performs data memory alignment and allocation operations based on the output feature index.

[0060] The calculation module calculates based on the total number of output feature elements. For a one-dimensional affine scale gain vector With DC bias vector Elements are filled using memory alignment rules. The allocation mechanism merges the one-dimensional affine scale gain values ​​and DC bias values ​​corresponding to the same feature index into contiguous memory words.

[0061] When the data bus width is configured to 32 bits, the calculation module extracts a 16-bit gain fixed-point value, shifts it left by 16 bits, and then performs a bitwise OR operation with a 16-bit bias fixed-point value to construct the 32-bit parameter configuration data. This data alignment operation enables the calculation module to concurrently read the calibration parameters of the corresponding channel in a single instruction cycle when receiving digital features, thereby reducing memory access latency.

[0062] See attached document Figure 5 In some embodiments, the specific implementation process of wavelength dispersion coupling forward propagation and square law mapping in step S300 above may include the following steps S310 to S330.

[0063] In step S310, the metasurface module receives the input light field and uses the phase response difference of the micro-nano structure to achieve physical channel separation of the dispersion effect.

[0064] The metasurface module receives a broadband input optical field containing multiple discrete wavelength components, each with a spatial field distribution that can be described by complex amplitude. The subwavelength nanoantenna array of the metasurface module is fabricated using a high-refractive-index dielectric material. When optical field components of different wavelengths are electromagnetically coupled to nanoantennas with specified geometric parameters, they are subjected to phase delay modulation with corresponding gradients. This phase delay modulation guides the optical fields of different wavelengths to form spatially distinct diffraction distributions, thereby creating independent physical transmission channels.

[0065] In step S320, the metasurface module projects the separated wavelength light field components onto the image sensor module to complete broadband optical field spatial modulation.

[0066] The modulated physical light field propagates to the output complex amplitude vector at the focal plane of the image sensor module. This vector is equal to the product of the fixed spatial transfer function matrix of the metasurface module for that channel and the input complex amplitude vector. This process is equivalent to the fixed spatial projection operator at the hardware front end.

[0067] In step S330, the image sensor module receives the spatially modulated light field and completes the square-law physical response mapping through a photoelectric conversion process.

[0068] The photodiodes in the pixel array perform physical conversion based on the photoelectric effect, and the number of photoelectrons generated is proportional to the square of the modulus of the complex amplitude of the incident light field. The light intensity distribution vector generated at the focal plane of the image sensor module provides the nonlinear features necessary for feature extraction in the system data stream. Utilizing the square-law characteristics of the underlying photoelectric absorption, and in conjunction with the end-to-end feature fitting parameters determined in the pre-configuration stage, the image sensor module completes front-end nonlinear feature extraction without the intervention of a digital network activation function.

[0069] See attached document Figure 6 In some embodiments, the specific implementation process of the simulation domain difference calculation and compressed feature generation in step S400 above may include the following steps S410 to S420.

[0070] In step S410, the image sensor module performs wavelength demultiplexing on the incident light field through a pixel-level filter array to extract the analog voltage signal of each wavelength channel.

[0071] The image sensor module integrates a multispectral filter array. After spatially modulated light fields pass through the corresponding wavelength filter microstructures, they are converted into voltage signals at the floating diffusion region nodes. This physical filtering process suppresses residual crosstalk generated during the diffraction propagation of light fields of different wavelengths, thereby performing wavelength demultiplexing of the light field signal.

[0072] In step S420, the image sensor module, in conjunction with the mask timing control logic, uses the differential amplifier circuit on the column readout bus to perform array-level multiplication and addition and direct dimensionality reduction operations to generate low-dimensional feature elements.

[0073] Based on the set positive and negative polarity operating timing, the image sensor module uses a non-destructive voltage sampling mode to extract features.

[0074] In the first sampling phase, the entire array space, i.e., dimension 1, is... The voltage signal corresponding to the space is extracted and connected to the positive polarity binary weighted capacitor array for proportional scaling. The weighted signals of each column of pixels first converge on the column-level readout bus to form the column-level positive polarity part sum, and then each column readout bus is further connected to the common integration node for cross-column simulated accumulation, which is summarized to form the global positive polarity voltage on the first integration capacitor.

[0075] In the second sampling stage, the voltage signal in the same space is extracted and connected to the negative polarity binary weighted capacitor array for proportional scaling, forming a column-level negative polarity part and accumulating across columns into the second integrating capacitor.

[0076] The differential amplifier circuit performs an analog subtraction operation on the voltage signals across the first and second integrating capacitors, outputting a differential voltage signal representing a single scalar characteristic element within a single cycle. Subsequently, the analog-to-digital converter immediately quantizes this scalar characteristic element digitally and stores it in a buffer, while the system resets the first and second integrating capacitors to prepare for the calculation in the next cycle.

[0077] Through multiple cycles of the above steps, the image sensor module directly compresses the high-resolution input light field into a format containing... A one-dimensional feature vector with n elements, where the number of output features is... Much smaller than the total number of input pixels This workflow directly converges high-dimensional feature maps into low-dimensional optical features, thereby reducing the data bus bandwidth usage of the analog-to-digital conversion module.

[0078] See attached document Figure 7 In some embodiments, the specific implementation process of digital affine reconstruction and deep network docking in step S500 above may include the following steps S510 to S530.

[0079] In step S510, the calculation module receives the digital one-dimensional feature vector output by the image sensor module after analog-to-digital conversion.

[0080] Since the optical spatial modulation and high-compression-ratio multiply-accumulate operations have been completed in the preceding physical stages, the data output by the analog-to-digital converter circuit is directly represented as a low-dimensional one-dimensional feature vector. The computing module receives this feature vector through the mobile industrial processor interface and moves it to a preset static random access memory buffer.

[0081] In step S520, the calculation module reads the affine calibration parameters from the built-in parameter buffer, performs error compensation calculation on the digital one-dimensional feature vector, and reconstructs the target feature distribution.

[0082] The arithmetic logic unit performs multiplication operations on each element of the one-dimensional feature vector with the corresponding affine scale gain value according to the feature channel index, and then performs addition operations with the corresponding DC bias value, thereby obtaining the output feature tensor after affine reconstruction. These affine parameters originate from the hardware-aware fine-tuning process during the system pre-configuration phase. Since the above fine-tuning has been completed during the deployment preparation phase, the computation module directly calls the final affine scale gain value and DC bias value fixed in the parameter buffer during this forward inference process, ensuring that the reconstructed feature distribution effectively approximates the ideal feature space.

[0083] In step S530, the calculation module passes the reconstructed output feature tensor to the deep neural network deployed in the digital domain, and uses the finely tuned backend neural network model weights to trigger network feature transfer and task inference calculation.

[0084] Based on the topological connections of the deep neural network model, the computation module directly feeds the one-dimensional feature vector after affine reconstruction into the back-end network layer with the corresponding interface for processing. For example, it can input it into a dedicated interface for visual features, or remap it into an intermediate layer input feature map that is compatible with the back-end network structure through reshaping operations and fully connected layers.

[0085] After the data stream is input to the backbone network for deep semantic feature extraction, it is further fed into the task output network for task regression and classification calculations. After the calculation module outputs the final task inference result, which includes elements such as the target bounding box and class confidence, it releases the memory buffer resources and waits for the reception of the next frame of digital feature data, thereby maintaining the continuous flow of forward inference in the system.

[0086] Specific application examples: See attached document Figure 8 This invention provides a specific application embodiment of image compression sensing and feature decomposition numerical transformation. To facilitate intuitive analysis of the data dimensionality reduction process, this embodiment uses local spatial tensor operations as an example for numerical demonstration.

[0087] Define the input space height corresponding to the receptive field of the object detection model in a single computation. 8 and width The number of input wavelength channels is 8. The value is 3, totaling 192 physical pixel inputs. This sets the total number of feature elements required by the target network. The value is 4.

[0088] A broadband light field is applied to the metasurface module. (In the channel index) In the red light field channel with a value of 1, the spatial transfer function matrix of the metasurface module performs a fixed spatial projection on the input complex amplitude vector. The light field reaching the surface of the image sensor module after propagation undergoes photoelectric conversion. According to the physical square-law response function, the square of the modulus of the incident light field's complex amplitude is converted into photoelectrons, forming a pre-nonlinear mapping. For ease of explanation, it is assumed that after exposure integration, the channel at this point... The voltage signals generated by the 64 pixel units with a value of 1 in the floating diffusion region node are evenly distributed with an average value of 1.5V.

[0089] The computation module pre-generates hardware deployment parameters using an alternating optimization algorithm, employing 4-bit quantization precision. For the output feature index... For feature elements that are 1, the system reads the corresponding positive and negative non-negative analog mask matrices from the mask register array. Assume the total quantized value of the positive weights for these 64 pixel positions is 45, and the total quantized value of the negative weights is 20.

[0090] The image sensor module controls the programmable sample-and-hold circuit to perform timing operations. In the first sampling phase, the 1.5V voltage signals of 64 pixels are connected to a positive binary weighted capacitor array for scaling and cross-column analog accumulation, resulting in a global positive voltage equivalent to 67.5V on the first integrating capacitor. In the second sampling phase, the voltage signals of the same pixel node are connected to a negative binary weighted capacitor array, resulting in a global negative voltage equivalent to 30.0V on the second integrating capacitor.

[0091] The differential amplifier circuit performs an analog subtraction operation, and the output differential voltage signal has an equivalent value of 37.5V. After a single element of this analog one-dimensional feature vector is quantized by an analog-to-digital converter, the output digital one-dimensional feature vector has a value of 38. At this point, the high-dimensional input data originally consisting of 192 pixels has been compressed into a digital one-dimensional feature vector of length 4.

[0092] The calculation module extracts the parameter cache and feature index. Let 1 be the set of affine calibration parameters. Assume the fine-tuned one-dimensional affine scale gain is 1.2 and the DC bias is -5.0. The calculation module performs a one-dimensional feature affine reconstruction operation, which involves multiplying 1.2 by 38 and adding -5.0, resulting in a single-element value of 40.6 for the reconstructed corrected feature tensor. This value is then passed to the backend deep neural network for target bounding box prediction.

[0093] See attached document Figure 9 With appendix Figure 10 To verify the authenticity of the technical solution of the present invention, this embodiment provides experimental verification data and effect comparison based on a standard image classification dataset.

[0094] The experiment used a full-color image sequence with an input resolution of 256 x 256 and 3 channels. The total number of one-dimensional feature elements in the output was set. It is 1024.

[0095] See attached document Figure 9 This figure illustrates the convergence state of the mean square error and the change in inference accuracy during the hardware-aware fine-tuning process. The horizontal axis represents the training iteration cycle, the left vertical axis represents the total mean square error constructed by the photoelectric joint mapping equation, and the right vertical axis represents the validation set accuracy of the image classification task. In the first 20 iteration cycles, the system fixes the physical parameters of the metasurface module and the set of non-negative analog mask tensors, and only jointly updates the deep network weights and the affine calibration parameter set.

[0096] Because the initial values ​​were determined using a multiple linear regression method, the total mean squared error (MSE) started at 0.45 and decreased stepwise. By the 45th iteration, the absolute value of the difference between two adjacent MSEs reached the threshold of 10 to the power of -4, and the MSE converged to below 0.02. Correspondingly, the system's classification accuracy on the validation set stabilized at 89.4% during the forward inference phase, approaching the theoretical upper limit of 90.1% for a full-precision floating-point model running entirely on a digital computing platform. The data indicates that by extracting hardware physical characteristics and performing alternating optimization solutions, the signal truncation error generated by the front-end physical simulation was compensated for by the affine reconstruction operation of the computation module.

[0097] See attached document Figure 10 The figure shows a comparison of energy consumption distribution and interface bandwidth usage between a traditional all-digital processing system and the system of this invention.

[0098] Regarding data transmission bandwidth, traditional systems require analog-to-digital conversion (ADC) to transmit a 256x256x3 pixel array to the digital backend, resulting in 196,608 feature words transmitted per frame. This invention, however, directly outputs a compressed, dimensionality-reduced one-dimensional digital feature vector by performing weighted allocation and differential accumulation operations within a programmable sample-and-hold circuit. The data transmission volume per frame is fixed at the set total number of feature elements, i.e., 1024 feature words. The bandwidth reduction ratio of the ADC interface is 192 to 1.

[0099] Regarding system power consumption, the average current values ​​obtained from the hardware testing environment were used for calculation. The power consumption distribution of the traditional system is as follows: 15mW for the image sensor module, 120mW for the high-frequency quantization of the analog-to-digital conversion circuit, and 180mW for the backend digital neural network to perform the first layer of high-dimensional convolution calculations.

[0100] The power consumption distribution of the system in this invention is as follows: the total power consumption of the front-end photosensitive and analog domain differential computation of the image sensor module and metasurface module is 25mW; the power consumption of the analog-to-digital conversion circuit decreases to 5mW due to the reduction in the number of quantization operations; and the power consumption of the back-end digital neural network decreases to 45mW due to the omission of the first-layer high-dimensional convolution operator. The total end-to-end computational power consumption of the system is reduced from 315mW in the traditional architecture to 75mW. The above experimental test data verify the objective engineering performance of the processing method of this invention in reducing the bandwidth pressure of the analog-to-digital conversion interface and the forward inference power consumption of the system.

Claims

1. A method for image compression sensing signal processing using integrated metasurface CIS wafer bonding, characterized in that, include: A photoelectric joint mapping equation is constructed, and after solving it with an alternating optimization algorithm and hardware-aware fine-tuning, a set of non-negative analog mask tensors, a set of affine calibration parameters, and the weights of the fine-tuned back-end neural network model are generated. Write the set of non-negative analog mask tensors into the mask register array of the image sensor module, set the working timing of the programmable sample-and-hold circuit, and deploy the set of affine calibration parameters and the fine-tuned back-end neural network model weights to the computing module. A broadband light field passes through the metasurface module, generating spatially differentiated diffraction distribution separation. This is then converted into charge distribution and pixel voltage signals in the image sensor module, forming a pre-nonlinear mapping. The image sensor module controls the programmable sample-and-hold circuit to perform weighted allocation and differential accumulation operations on the pixel voltage signal, compress and reduce the dimension to output an analog one-dimensional feature vector, and then quantize it to output a digital one-dimensional feature vector. The calculation module extracts the affine calibration parameter set and performs a one-dimensional feature affine reconstruction operation on the digital one-dimensional feature vector to generate a correction feature tensor, which is then passed to the backend deep neural network to perform subsequent task inference operations.

2. The image compression sensing signal processing method for integrated metasurface CIS wafer bonding according to claim 1, characterized in that, The construction of the photoelectric joint mapping equation includes: The feature extraction layer operator of the back-end deep neural network is extracted, and the inherent square law transformation characteristics of the image sensor module are introduced as a front-end physical nonlinearity to establish a hardware equivalent response model. The physical limitations of the image sensor module are combined with the digital compensation capability of the computing module to construct a global error model that includes a mean square error objective function; The mean square error objective function is jointly determined by the ideal one-dimensional eigenvector expected by the back-end network, the input complex amplitude vector, the one-dimensional affine scale gain vector, the DC bias vector, the positive and negative non-negative analog mask matrices, the fixed spatial transfer function matrix, and the physical square-law response function of the photoelectric conversion link, thereby forming the photoelectric joint mapping equation.

3. The image compression sensing signal processing method for integrated metasurface CIS wafer bonding according to claim 2, characterized in that, The solution obtained through the alternating optimization algorithm includes: Under the conditions of fixed optical transfer function and physical square-law response, an alternating optimization iteration is performed on the programmable analog mask and the set of affine calibration parameters; By keeping the affine calibration parameter set unchanged, and based on the physical constraint that the analog weighted capacitors only represent non-negative weights, the positive and negative polarity non-negative analog mask matrices that minimize the total mean square error are solved. By keeping the positive and negative polarity non-negative analog mask matrices unchanged, the affine calibration parameter set that minimizes the error is calculated using a multivariate linear regression method. This set is used to correct the zero-point offset caused by forced non-negative projection and physical truncation.

4. The image compression sensing signal processing method for integrated metasurface CIS wafer bonding according to claim 1, characterized in that, The step of writing the set of non-negative analog mask tensors into the mask register array of the image sensor module includes: Perform binary quantization on the set of non-negative analog mask tensors; The address mapping rule using a multi-dimensional joint addressing mode is used to write the quantized mask data into the mask register array; The hardware addressing index of the multidimensional joint addressing mode covers the index of output features, wavelength channel index, pixel row address, pixel column address, and polarity index.

5. The image compression sensing signal processing method for integrated metasurface CIS wafer bonding according to claim 1, characterized in that, The step of deploying the affine calibration parameter set and the fine-tuned backend neural network model weights to the computing module includes: The affine calibration parameter set is loaded into the built-in parameter cache, and data memory alignment and allocation operations based on the output feature index are performed. The affine calibration parameter set includes one-dimensional affine scale gain values ​​and DC bias values. Based on the total number of output feature elements, the one-dimensional affine scale gain value and the DC bias value corresponding to the same feature index are merged into a continuous memory word, so that the calculation module can concurrently read the affine calibration parameters of the corresponding channel in a single instruction cycle when receiving digital features.

6. The image compression sensing signal processing method for integrated metasurface CIS wafer bonding according to claim 1, characterized in that, The broadband optical field passing through the metasurface module produces spatially differentiated diffraction distribution separation, including: The phase delay modulation of the subwavelength nanoantenna array of the metasurface module is used to guide light fields of different wavelengths to form spatially differentiated diffraction distributions that can be distinguished from each other, thereby forming independent physical transmission channels. The number of photoelectrons generated in the charge distribution is proportional to the square of the modulus of the complex amplitude of the incident light field. The pre-nonlinear mapping is completed by using the square law characteristics of the bottom photoelectric absorption in conjunction with the end-to-end feature fitting parameters.

7. The image compression sensing signal processing method for integrated metasurface CIS wafer bonding according to claim 1, characterized in that, The image sensor module controls the programmable sample-and-hold circuit to perform weighted allocation and differential accumulation operations on the pixel voltage signals, including: A non-destructive voltage sampling mode is adopted. In the first sampling stage, the voltage signal of the entire array space is taken out and connected to the positive polarity binary weighted capacitor array for proportional scaling. The signal is then aggregated to form the column-level positive polarity part and then simulated across columns to form the global positive polarity voltage on the first integrating capacitor. In the second sampling stage, the voltage signal in the same space is taken out and connected to the negative polarity binary weighted capacitor array for proportional scaling, forming a column-level negative polarity part and accumulating across columns into the second integrating capacitor.

8. The image compression sensing signal processing method for integrated metasurface CIS wafer bonding according to claim 7, characterized in that, The compressed dimensionality reduction output simulates a one-dimensional feature vector, including: A differential amplifier circuit is used to perform an analog subtraction operation on the voltage signals on the first integrating capacitor and the second integrating capacitor, and outputs a differential voltage signal representing a single scalar feature element within a single cycle, which constitutes the analog one-dimensional feature vector.

9. The image compression sensing signal processing method for integrated metasurface CIS wafer bonding according to claim 1, characterized in that, The calculation module extracts the affine calibration parameter set and performs a one-dimensional feature affine reconstruction operation on the digital one-dimensional feature vector to generate a correction feature tensor, including: According to the feature channel index, each element in the digital one-dimensional feature vector is multiplied by the one-dimensional affine scale gain value of the corresponding channel in the affine calibration parameter set, and then added by the corresponding DC bias value in the set, thereby obtaining the output feature tensor after affine reconstruction as the correction feature tensor.

10. The image compression sensing signal processing method for integrated metasurface CIS wafer bonding according to claim 1, characterized in that, The process of passing the data to the backend deep neural network for subsequent inference operations includes: The corrected feature tensor is fed into a back-end network layer with a corresponding interface for processing and remapped into an intermediate layer input feature map that is compatible with the back-end network structure. After deep semantic feature extraction is completed by feeding the backbone network, the data is further fed into the task output network, which outputs the task inference result containing the target bounding box and the category confidence.