Lensless optical neural image processing system

Abstract

The application relates to the technical field of optoelectronic hybrid computing and image processing, and discloses a lens-free optical neural image processing system, which comprises an electrically addressable programmable spatial light modulator, an image sensor and a digital backend network arranged in sequence; incoherent light emitted by a target scene is encoded by a modulation layer and directly received by the image sensor to form a speckle pattern and is input into the digital backend network to complete a task; meanwhile, for non-differentiable optical modulation layer parameters, a weight simultaneous disturbance algorithm is used to directly estimate gradients on a physical system for optimization; for differentiable digital network parameters, a gradient descent method is used for optimization; the two are alternately updated in a unified framework to realize global collaborative optimization of optical domain and electrical domain parameters. The application avoids the need for accurate physical modeling, eliminates a secondary calibration link after system deployment, and significantly improves the task adaptability, deployment flexibility and overall performance of the lens-free computational imaging system.

Description

Technical Field

[0001] This invention relates to the field of optoelectronic hybrid computing and image processing technology, and in particular to a lensless optical neural image processing system. Background Technology

[0002] With the rapid development of artificial intelligence technology, especially in computer vision tasks such as image recognition, object detection, and image reconstruction, algorithms based on deep neural networks have achieved unprecedented success. However, traditional pure electronic computing architectures face the "von Neumann bottleneck" challenge when processing massive amounts of visual data, characterized by high power consumption, high latency, and exponentially increasing computing power requirements. To overcome this limitation, optoelectronic hybrid computing architectures, which combine the high parallelism, low power consumption, and ultra-fast speed of optical computing with the flexibility of electronic computing, have emerged and become a research hotspot in academia and industry.

[0003] In this interdisciplinary field, lensless optical neural image processing systems have demonstrated enormous application potential. In both production and daily life, these systems are expected to be deployed in various scenarios with strict limitations on size, cost, and power consumption. For example, in mobile terminal devices (such as smartphones and tablets), they can replace traditional bulky lens modules to achieve ultra-thin cameras for applications such as facial recognition and augmented reality (AR) perception. In embedded vision systems, such as auxiliary sensors for drones and autonomous vehicles, and defect detection in industrial products, their compact structure and low power consumption can significantly improve the device's battery life and deployment flexibility. In the Internet of Things (IoT) and smart monitoring fields, extremely low-cost, easily attachable miniature vision sensors can be manufactured for environmental monitoring and fall detection in the elderly. Furthermore, in the healthcare field, miniaturized, lensless endoscopic imaging systems can reduce patient discomfort and enable wider screening applications. The core of these applications lies in using a programmable optical mask layer to physically encode incident light, which is then decoded by a lightweight digital neural network to complete classification or reconstruction tasks, thereby achieving the integration of perception and computation.

[0004] Despite the promising future of lensless optoelectronic hybrid systems, existing technologies still face the following technical challenges in practical applications: First, the biggest technical bottleneck is the mismatch between optical modeling errors and the physical system. The current mainstream design paradigm is "digital design first, then physical solidification." Researchers typically design and train multilayer diffractive optical elements or metasurfaces on computers based on ideal optical propagation models (such as Fresnel diffraction and Fraunhofer diffraction). However, once the trained parameters are "solidified" onto the physical device, due to manufacturing tolerances, non-ideal material dispersion characteristics, assembly errors, and the approximation of the optical field propagation model, there will inevitably be differences between the digital model and the real physical system that are difficult to quantify precisely. This "analog-physical" gap will lead to a sharp decline in the performance of the system after deployment, such as reduced image classification accuracy or deterioration of reconstructed image quality. Secondly, to address the aforementioned mismatch issue, existing technologies introduce a complex secondary calibration process. To compensate for the mismatch between the physical front-end and the digital model, the digital back-end network typically requires tedious secondary calibration or retraining after system deployment. This process necessitates collecting a large amount of output data from real physical systems to fine-tune the digital network, which not only increases additional time and manpower costs but also, because the calibration process struggles to cover all possible operating conditions and environmental changes, often fails to fundamentally solve the problem. Consequently, the system's performance ceiling is limited by the quality and coverage of the calibration dataset. This lengthens the system's development and application cycle and makes it difficult to maintain stable and reliable performance in volatile environments. Furthermore, existing hardware-in-the-loop training methods are still limited by the accuracy of the digital surrogate model. To incorporate physical effects into the training process, some advanced solutions employ hardware-in-the-loop training, which involves the physical system participating in forward computation. However, when updating optical layer parameters through backpropagation, these methods still rely on a differentiable digital surrogate model (or "digital twin") to calculate gradients. In other words, the algorithm updates parameters on the digital model and then deploys it to the physical system for verification. This approach does not eliminate the dependence on an accurate physical model. Any inaccuracies in the digital surrogate model will cause the calculated gradient direction to deviate from the true physical optimal direction. Optimization errors accumulate during iterations, ultimately limiting the peak performance that the system can achieve. This "semi-in-the-loop" approach, in essence, still does not achieve a true deep integration of physics and computation. Finally, the above problems collectively lead to poor system task adaptability and insufficient flexibility. Once the optical front end is designed and manufactured, its physical characteristics are fixed, making it difficult to dynamically adjust for new tasks or environments. Although it can be partially adapted by retraining the digital back end, the overall system performance ceiling is also determined because the optical coding mode is already solidified, making it difficult to flexibly adapt to a variety of different complex tasks (such as switching from simple classification to high-precision image reconstruction).

[0005] Therefore, those skilled in the art urgently need a novel optical neural processing system architecture and training solution that can truly achieve end-to-end collaborative optimization of optical and electrical parameters, eliminate the need for precise physical modeling, and avoid secondary calibration. Summary of the Invention

[0006] The core technical problem solved by this invention is: how to completely break the dependence of existing optoelectronic hybrid systems on accurate physical modeling, fundamentally eliminate the system mismatch between the optical front end and the digital back end, avoid cumbersome secondary calibration process, and realize global, end-to-end collaborative optimization of optical and electrical parameters in the real physical system, thereby improving the system's task adaptability, deployment flexibility and final performance.

[0007] In order to solve the above-mentioned core technical problems, this invention designs a lensless optical neural image processing system. Its purpose is to build an intelligent imaging architecture that does not require precise modeling or secondary calibration, and can directly drive the physical system to perform joint optimization starting from the task objective. The core of this invention lies in the fact that the system is tightly integrated with three parts: an electrically programmable spatial light field modulation layer, an image sensor, and a digital back-end processing network. Incoherent light from the target scene propagates in free space, is directly encoded and modulated by the light field modulation layer, received by the image sensor to form a speckle pattern, and finally input into the digital back-end network to complete the designated task.

[0008] The key to this invention lies in employing a weighted simultaneous perturbation algorithm for the non-differentiable physical optical modulation layer parameters. This algorithm performs two forward propagations on the real physical system (once for the original parameters and once for the parameters after applying a small random perturbation), directly estimating the approximate gradient of the physical parameters based on the change in the loss function. For the differentiable digital backend network parameters, the standard gradient descent method is used for optimization. Both are iteratively updated alternately within the same training framework, achieving global collaborative optimization of the optical domain physical parameters and the electrical domain network parameters. This method directly incorporates real physical effects into the optimization loop, avoiding precise physical modeling and secondary calibration, and significantly improving the overall performance, adaptability, and flexibility of the system.

[0009] To achieve the above objectives, the specific technical solution of the present invention is a lensless optical image neural processing system, comprising: an electrically addressable and programmable spatial light modulator, an image sensor, and an electronic digital processor; the spatial light modulator, the image sensor, and the electronic digital processor are arranged sequentially, the spatial light modulator and the image sensor constitute an optical front end, and the electronic digital processor and its built-in electronic digital neural network constitute an electronic digital back end. The system achieves integrated optimization of the optical front-end and the electronic digital back-end by synchronously inversely designing and end-to-end collaboratively training the modulation parameters of the spatial light modulator in multiple color channels and the weight parameters of the electronic digital neural network, thereby completing the color image processing task.

[0010] It should be noted that the programmable light field modulation layer of the spatial light modulator is located directly in front of the image sensor, and the physical distance between the two is configured to be on the order of millimeters (in the embodiments of the present invention, the distance is set to be within the range of less than 1 millimeter). This spatial structure constitutes the basic configuration of lensless optical imaging. In this configuration, the visible light field emitted from the target scene propagates in free space, first passing through the programmable light field modulation layer, and then entering the image sensor. The programmable light field modulation layer is configured to perform spatial modulation operation on the incident light field. Its modulation effect encodes each object point in the target scene and maps it to multiple pixel unit regions of the image sensor, thereby forming a light intensity distribution pattern with high spatial contrast characteristics on the photosensitive plane of the image sensor.

[0011] In the operating environment of the system, when the target scene is under incoherent illumination, it can be equivalently represented as a set of multiple point light sources, each with different spectral characteristics, light intensity, and spatial location. The image ultimately acquired by the image sensor is essentially a linear superposition of the light intensity patterns generated on the image sensor plane after the light fields emitted by these point light sources are spatially modulated by the programmable light field modulation layer. This image, as a physically encoded optical signal, is transmitted to the digital backend and network (i.e., the electrical domain digital backend and its built-in electrical domain digital neural network) for subsequent processing to complete specified tasks, including image classification or image reconstruction.

[0012] The programmable light field modulation layer has learnable parameters for its light field modulation distribution. By adjusting the parameters of the programmable light field modulation layer online, it can adapt to different imaging tasks or environmental conditions. The algorithm and network of the digital backend also have learnable network parameters. The parameters of the algorithm and network of the digital backend can also be fine-tuned according to the actual application scenario to further improve system performance.

[0013] The spatial light modulator has the ability to independently adjust the transmittance and transmittance spectrum of each spatial pixel, and its structure includes the integration of a liquid crystal modulation layer and a color filter layer.

[0014] The spatial light modulator is the only optical modulation element in the system. The distance between the spatial light modulator and the image sensor is on the order of millimeters, forming a compact lensless imaging optical path.

[0015] The image sensor is an array-type CMOS or CCD sensor, which directly receives the coded light field modulated by the spatial light modulator and outputs the corresponding light intensity distribution image to the electronic domain digital backend. The light intensity distribution image is transmitted to the electronic domain digital processor of the electronic domain digital backend through a data interface. The electronic domain digital processor executes the algorithm and network of the digital backend and outputs the classification result or reconstructed image.

[0016] The image data acquired by the image sensor is transmitted to the electronic domain digital processor through a data interface. The electronic domain digital processor executes the algorithms and networks of the digital backend and outputs classification results or reconstructed images.

[0017] The electrical domain digital neural network of the electrical domain digital backend is a trainable neural network structure, which can be selected from one or a combination of fully connected networks, convolutional neural networks or U-Net structures depending on the task type.

[0018] The end-to-end collaborative training method employs a hardware-in-the-loop training mechanism and includes the following steps: a) Perform forward propagation in the physical system to acquire the light intensity pattern output by the sensor; b) Input the light intensity pattern into the digital backend network and calculate the loss function; c) Update the network parameters of the digital backend using gradient descent; d) Calculate and update the approximate gradient of the modulatorable parameters of the spatial light modulator using a weighted simultaneous perturbation algorithm; e) Repeat steps a) to d) until the loss function converges.

[0019] It is important to note that the hardware-in-the-loop end-to-end collaborative training method involves the collaborative optimization of optical modulation parameters and digital processing parameters. In the specific optimization process, the parameters of the programmable optical field modulation layer are updated using a weighted simultaneous perturbation algorithm, while the parameters of the digital backend algorithm and network are updated using a gradient-based backpropagation algorithm. Through iterative optimization, the optical modulation mode and the backend computing network are matched to the task objectives, thereby achieving global performance optimization. After training, the system can be directly used for actual image processing tasks without the need for secondary optical calibration or parameter adjustment.

[0020] The weighted simultaneous perturbation algorithm estimates the gradient based on the loss change during two forward propagations by applying random small perturbations to the optical parameters. The formula is as follows: In the formula, For parameters The approximate gradient, Represents the system's loss function. These are the moduliable parameters of the optical field modulation layer. For small perturbations that follow a normal distribution.

[0021] The system is suitable for incoherent light imaging scenarios, where the input light field is the target scene under natural light or LED illumination, and no additional optical lenses or imaging lenses are required.

[0022] The system can be deployed directly after training without the need for secondary calibration or retraining for specific environments, and has good task adaptability and system stability.

[0023] The system supports multi-task processing, including but not limited to: one or more of the following: handwritten digit recognition, object classification, optical image reconstruction, and multispectral image processing.

[0024] Compared with the prior art, the technical solution disclosed in this application has the following non-obvious technical features: First, this application adopts a hardware-in-the-loop collaborative optimization architecture without a surrogate model. Existing hardware-in-the-loop methods rely on a differentiable digital surrogate model to update optical parameters. This invention breaks away from this mindset and proposes to directly abandon the digital surrogate model when updating optical parameters, and adopt a weighted perturbation algorithm to estimate the gradient through a finite number of differences on a real physical system. This "true in-the-loop" rather than "semi-in-the-loop" approach fundamentally solves the model mismatch problem and is a breakthrough design that is not easily thought of by those skilled in the art. Second, this application achieves a heterogeneous combination of non-gradient optimization and gradient optimization in the physical-digital domain. This application creatively integrates a non-gradient optimization algorithm (with simultaneous weight perturbation) for non-differentiable physical systems with a gradient descent method for differentiable digital networks into a unified end-to-end training framework. The two are optimized asynchronously and alternately in the physical and digital domains, respectively, achieving efficient search of the global parameter space. This ingenious combination of heterogeneous optimization strategies is not a simple superposition, but a system-level solution designed to overcome the core problem of non-differentiable physical parameters. Third, this application adopts parametric-level optoelectronic co-training and deployment. The training process of this application realizes synchronous alignment and co-evolution of the optical domain front end (mask parameters) and the electrical domain back end (network weights) at the parametric level. This means that the optical coding mode is "tailor-made" for specific digital network structures and task objectives, and vice versa. After training, the system reaches a globally optimal matching state, so no secondary calibration is required during deployment. This deep, co-design concept that spans the physical and digital domains surpasses the traditional approach of processing the two separately or only performing surface adaptation. Fourth, this application uses a mechanism in which the physical system directly participates in gradient estimation. By employing a weight perturbation algorithm, the physical system itself is treated as a "black box" gradient information generator. By applying small perturbations to the physical hardware and observing changes in the output loss, the update direction that reflects the real physical response can be directly obtained. This method transforms the physical system from a passive executor to an active optimization participant, a technical path that is difficult for those skilled in the art to conceive under the traditional model-based design approach. Fifth, the programmable mask in this application serves as a "task-adaptive" optical processor: In this application, the spatial light modulator is no longer a fixed encoding template, but a "task-adaptive" optical processor that is obtained through co-training and can automatically learn the optimal encoding strategy for different tasks (such as classification or reconstruction). This method of defining mask functions through learning rather than physical formulas greatly expands the flexibility and application scope of the system, reflecting the innovative idea of ​​deep integration of optical computing and artificial intelligence.

[0025] Compared with the prior art, the present invention has the following beneficial effects: 1. This invention achieves end-to-end global parameter optimization: By using a collaborative optimization parameter training scheme, the optical physical layer and the digital computing layer are incorporated into a unified optimization framework, enabling the system to directly start from the task objective and alternately optimize the optical field coding strategy and neural network parameters in reverse. This breaks the limitations of the traditional scheme's separation of physics and computation, thereby effectively improving the system's performance on the target task. 2. This invention reduces the reliance on precise and complex optical physics modeling: the system uses a data-driven approach to allow the programmable optical field modulation layer to autonomously learn and adapt to the optimal coding mode for backend tasks, avoiding the cumbersome process of precise modeling and calibration of the imaging system in traditional methods, and enhancing the practicality of the technical solution and its robustness to non-ideal conditions. 3. This invention eliminates the training requirement and performance bottleneck of secondary calibration: Since the optical front end and digital back end achieve parameter-level collaborative alignment during training, the system no longer needs to perform secondary calibration of the digital back end independently of the training process when deployed. This not only shortens the cycle from system development to application, but also effectively avoids system performance degradation caused by calibration errors or mismatches. 4. This invention has good task adaptability and system flexibility: thanks to the parameterized design of the programmable light field modulation layer and the modular back-end network, the same optical hardware platform can be quickly adapted to a variety of different image processing tasks by changing the back-end algorithm network and adjusting the programmable light field modulation layer mode, such as switching from image classification to target detection, image reconstruction, etc., thus expanding the application scope and use value of the system. 5. This invention promotes the miniaturization and integration of systems: the lensless compact design simplifies the optical path structure, reduces the system's size, weight, and the cost of optical components, making this technical solution particularly suitable for cutting-edge application areas such as embedded devices, mobile terminals, UAV payloads, and wearable devices where size, power consumption, and cost are strictly limited, thus driving the development of intelligent sensing technology. Attached Figure Description

[0026] Figure 1 This is a schematic diagram of the system described in Embodiment 1 of the present invention; In the diagram: 1. Scene image detection; 2. Programmable light field modulation layer; 3. Image sensor; 4. Digital back-end algorithm and network. Figure 2 This is the flow of the algorithm described in Embodiment 2 of the present invention; Figure 3 This is an architecture diagram of the system described in Embodiment 3 of the present invention; Figure 4 This is a schematic diagram of the parameter training process of the system described in Embodiment 3 of the present invention; Figure 5 These are the image classification and recognition results and image reconstruction effect diagrams of the system described in Embodiment 3 of the present invention. Detailed Implementation

[0027] The embodiments of the present invention will now be described in detail with reference to the accompanying drawings; Example 1: A lensless optical neural image processing system, such as Figure 1 As shown, the system comprises three core components: an electrically addressable and programmable spatial light modulator, an image sensor, and an electrical domain digital processor (with a built-in electrical domain digital neural network and an end-to-end co-training method).

[0028] In the workflow, the light field carrying the target scene information is first physically encoded and modulated by the programmable light field modulation layer of the spatial light modulator. This modulation process encodes the light field spatially, so that it is accurately mapped onto the photosensitive surface of the image sensor. Subsequently, the image sensor is responsible for detecting and recording the modulated light field intensity distribution information to form a speckle image with specific coding features; Finally, the algorithm (mainly the weight perturbation algorithm and gradient descent method in the end-to-end collaborative training method) and network (mainly the electric domain digital neural network) of the digital backend trained with parameters receive the speckle image as input, select and apply the corresponding algorithm model and network architecture according to the specific processing task, perform deep calculation and analysis, and finally output the image classification and recognition result or calculate and reconstruct the image.

[0029] Specifically, the working mechanism and structure of this system can be understood through... Figure 1 A detailed explanation follows: The incoherent natural light field emitted by the detected scene image 1 can be physically modeled as a collection of multiple incoherent point light sources with different wavelength characteristics, spatial distributions, and radiation intensities. The light waves emitted by each point light source propagate independently in free space, reaching the programmable light field modulation layer 2. After amplitude or phase modulation based on a programmable mask, the light field is further mapped onto the photosensitive surface of the image sensor 3. The light field undergoes specific encoding processing in the modulation layer, thereby forming a complex speckle pattern with high contrast and high information density on the sensor surface.

[0030] It is important to note that this system can approximately satisfy the translation invariance condition within a certain spatial range. This means that object points at different spatial locations in the target scene, after being processed by the modulation layer, will produce speckle patterns on image sensor 3 with similar spatial structural features, differing only in spatial location. Based on this characteristic, the image finally recorded by image sensor 3 can be regarded as a linear superposition of the speckle patterns corresponding to all object points in the target scene on the sensor plane. This linearly superimposed image not only preserves the global light intensity distribution characteristics of the original scene but also incorporates rich structural information and spatial features through speckle coding, providing high-dimensional, high-information feature input for backend algorithm processing.

[0031] The recorded speckle image is converted into a digital signal by an analog-to-digital converter and then transmitted to the digital back-end algorithm and network 4 for processing. This digital back-end can be flexibly configured according to the actual task requirements. For example, in image classification tasks, a deep convolutional neural network containing multiple convolutional and pooling layers can be used to extract image features and output the class probability distribution through a fully connected layer; in image reconstruction tasks, a neural network with encoder and decoder structures, such as the U-Net network, can be used to realize the conversion from speckle image to sharp original image.

[0032] The entire system employs a hardware-in-the-loop end-to-end optimization strategy during the training phase. By alternately updating the physical parameters of the optical field modulation layer and the trainable weights of the digital neural network, it achieves a globally optimal match between the optical encoding process and the algorithm decoding process, thereby effectively improving the system's performance in various tasks. After training, the system can be directly deployed to real-world application scenarios without additional calibration steps, demonstrating high practicality and operational stability.

[0033] In practical implementation, the programmable light field modulation layer can be implemented using programmable optical elements such as liquid crystal spatial light modulators or digital micromirror devices. These elements can change their optical properties in real time according to control signals, thereby achieving precise modulation of the incident light field. The image sensor can be a high-resolution, high-sensitivity CMOS or CCD sensor to ensure accurate capture of the speckle pattern formed after modulation. The digital backend algorithms and networks can be deployed on general-purpose computing platforms or dedicated neural network processors, and optimized for processing speed and power consumption requirements according to the actual application scenario.

[0034] During operation, the system first encodes the incident light field through a programmable optical field modulation layer. This encoding can be amplitude encoding, phase encoding, or a combination of both. The encoded light field forms a specific speckle pattern on the image sensor, which contains compressed encoded information of the original scene. By learning the mapping relationship between this encoding mode and the original image, the digital back-end network can recover the information of the original scene from the speckle pattern, achieving image classification or reconstruction. The overall system performance depends on the degree of synergistic optimization between optical encoding and digital decoding; an end-to-end training method can achieve optimal matching between the two.

[0035] Example 2: A parameter joint optimization training algorithm for a lensless optical neural image processing system, the process of which is as follows: Figure 2 As shown, the algorithm aims to improve the system's performance on the target task by jointly optimizing the optical modulation layer parameters and digital back-end network parameters through hardware-in-the-loop end-to-end training.

[0036] During training, the incoherent light field from the target scene propagates in free space, first undergoing spatial modulation through a programmable light field modulation layer, and then reaching the image sensor to form a speckle image. To simulate this physical process in the digital domain, this algorithm uses the point spread function (PSF) to model the light field transmission. The distribution of the light field received by the image sensor plane is shown when the system satisfies translation invariance within a certain spatial range. It can be obtained from the input light field The result is obtained by convolving the system's point spread function, as shown in formula (1.1).

[0037] (1.1) In formula (1.1), This represents the spatial intensity distribution of the input light field. The programmable light field modulation layer has a set of modulated parameters. These parameters determine the spatial distribution of the mask pattern produced by the modulation layer. When the physical distance between the modulation layer and the image sensor is small enough (typically on the order of millimeters), the point spread function pattern of the system can be approximated by the distribution of the mask layer, thus simplifying the complexity of the physical model.

[0038] The lensless speckle image recorded by the image sensor is then fed into a learnable digital backend algorithm and network for processing. The digital backend network has a set of learnable network parameters. Depending on the task being performed, different network structures can be used in the digital backend. For image classification or recognition tasks, the digital backend typically consists of several fully connected layers, non-linear activation layers, and an output layer. The non-linear activation layers generally use functions such as ReLU, and the number of neurons in the output layer is determined based on the number of classification categories. For image reconstruction tasks, the digital backend can use encoder-decoder structures such as U-Net, with its network parameters... This mainly includes the weights and biases of each convolutional layer. Additionally, digital simulation methods of physical systems can be used to reconstruct images using the learning-based alternating direction multiplier method (LE-ADMM), where network parameters can be learned. This corresponds to the scalar penalty parameter in each iteration of the algorithm.

[0039] After the speckle image is processed by the digital back-end network, the system output is obtained. The loss function can be calculated by comparing the output with the corresponding label in the training set. For image classification tasks, the cross-entropy loss function is often used to measure the difference between the network output and the true label; for image reconstruction tasks, a hybrid loss function composed of mean squared error (MSE) and learned perceptual patch similarity (LPIPS) is often used. The hybrid loss function can simultaneously constrain pixel-level error and perceptual similarity, which helps to improve the visual quality of the reconstructed image. The hybrid loss function for image reconstruction tasks is shown in Equation (1.2).

[0040] (1.2) In formula (1.2), For the label image, To balance the hyperparameters of the two indicators.

[0041] The MSE index is calculated as shown in formula (1.3).

[0042] (1.3) In formula (1.3), This represents the total number of pixels in the image. Indicates the first The value of each pixel in the system output image. No. The value of each pixel in the reference image (real label).

[0043] The LPIPS index is calculated as shown in formula (1.4).

[0044] (1.4) In formula (1.4), Indicates the pre-trained network's... Feature maps extracted from layers, and Indicates the first Spatial dimensions (height and width) of layer feature maps. and These represent the spatial indexes of the feature map in the height and width directions, respectively.

[0045] After completing the forward calculation of the loss function, the network parameters of the digital backend need to be processed separately. Modulable parameters of the optical modulation layer Update the network parameters for the digital backend. Since its model is differentiable, the stochastic gradient descent algorithm can be used to calculate the loss function relative to the network parameters. The gradient is calculated, and the parameters are updated in the opposite direction of the gradient, thereby minimizing the loss function.

[0046] Modulable parameters of the programmable optical field modulation layer Since it corresponds to a real physical system, its gradient cannot be directly obtained analytically. Therefore, this algorithm employs a weighted simultaneous perturbation algorithm to estimate the approximate gradient of this parameter. This method estimates the gradient of the modulated parameter by... Apply a random perturbation Then, based on the change in the loss function of the system output before and after the perturbation, the magnitude and direction of the gradient are estimated, as shown in formula (1.5).

[0047] (1.5) In formula (1.5), Represents the system's loss function. Indicates the application of moduloable parameters The small perturbation. This perturbation is typically sampled from a sample with a mean of zero and a standard deviation of [value missing]. The normal distribution, where It is a small positive value set empirically to control the magnitude of the perturbation. This perturbation method allows us to obtain the effective direction of parameter updates without needing to know the precise analytical model of the system. make Then we get: Stochastic gradient descent and simultaneous weight perturbation algorithms are used to iteratively update the network parameters of the digital backend. Modulable parameters of physical systems As shown in formulas (1.6) and (1.7).

[0048] (1.6) (1.7) In formula (1.6), Indicates the state before the update parameter, Indicates the updated parameter, Indicates update The learning rate of the parameters.

[0049] In formula (1.7), Indicates the state before the update parameter, Indicates the updated parameter, Indicates update The learning rate of the parameters.

[0050] In each iteration, the calculated parameter updates are deployed in real time to the programmable mask layer and the digital back-end algorithm and network, before the next forward computation. This process is repeated until the preset convergence condition is met or the maximum number of iterations is reached. Through this online, end-to-end training method, global collaborative optimization of the parameters of the optical modulation layer and the digital back-end network is achieved. This enables the system to automatically learn optical coding patterns and digital processing strategies adapted to specific tasks, thereby effectively improving the overall performance of the system and eliminating the multi-stage calibration steps required in traditional methods.

[0051] Example 3: A hardware-in-the-loop training system, the system architecture of which is as follows: Figure 3As shown, the system adopts a distributed control structure: a high-performance computer serves as the central control unit, responsible for scheduling, calculation, and management of testing tasks in the overall parameter training process. This computer establishes a remote communication connection with the Raspberry Pi development board via the SSH protocol, enabling indirect control of various physical modules within the system, including the collaborative work of the image display module, light field modulation module, and photoelectric conversion module. To achieve rapid switching and dynamic loading of the target scene during online training, a high-resolution LCD display is used as the dynamic display carrier for training images. This display directly presents the original images from the training dataset, serving as the optical input signal for the optical neural image processing system. A specially processed backlight-free LCD panel is used as the physical device for the programmable light field modulation layer. This LCD panel has pixel-level programmable dimming capabilities and can generate a light field mask pattern with a specific transmittance distribution based on electrical signals. A lensless Raspberry Pi high-definition camera is deployed at a distance of less than 1 mm from this modulation layer as the photoelectric conversion module. This camera is directly aimed behind the LCD panel, responsible for recording the lensless speckle image formed after encoding and mapping by the programmable light field modulation layer, completing the conversion from optical signal to digital image.

[0052] The system employs an end-to-end hardware-in-the-loop optimization strategy (i.e., the parameter joint optimization training algorithm described in Example 2) during parameter training. The overall training process is as follows: Figure 4 As shown, the specific process of a single parameter iteration is as follows: Step 1. The computer first transmits one input image from the training set to the Raspberry Pi development board through the communication interface and displays it in real time on the LCD screen to form a standardized optical input scene.

[0053] Step 2. The computer uses the learnable parameters of the current programmable optical field modulation layer. The corresponding LCD panel mask distribution pattern is calculated, and the control signal is sent to the Raspberry Pi to drive the LCD panel to display the corresponding modulation pattern, thus completing the physical encoding of the light field.

[0054] Step 3. The computer sends a photo-taking command to the Raspberry Pi, and the Raspberry Pi camera simultaneously captures a modulated, lensless speckle image, and then transmits the image data back to the computer via the communication interface.

[0055] Step 4. The digital backend algorithm and network deployed on the computer receive the speckle image as input, and after forward propagation calculation, generate the system's output result under the current parameters (such as classification probability distribution or reconstructed image).

[0056] Step 5. Calculate the loss function value for the current iteration based on the difference between the output result and the true label or target image.

[0057] Step 6. Using the stochastic gradient descent algorithm, calculate the learnable parameters of the digital backend algorithm and network through backpropagation. The precise gradient.

[0058] Step 7. Physical parameters of the programmable optical field modulation layer Apply small perturbation Then repeat the forward calculation process from steps 1 to 5 above to obtain the value of the perturbated loss function.

[0059] Step 8. Based on the change in the loss function before and after parameter perturbation, calculate the learnable parameters of the physical system using the weighted simultaneous perturbation algorithm. The approximate gradient.

[0060] Step 9. Finally, use the calculated parameters gradient and parameters The approximate gradient is obtained, and the parameters of the digital backend network are updated simultaneously. and parameters of the optical field modulation layer This completes a full parameter iteration optimization.

[0061] After the above training process optimization, the system demonstrated good performance in multiple benchmark tasks. For example... Figure 5 As shown, the lensless optical neural image processing system trained in this specific embodiment achieved a test set classification accuracy of 94.6% in the image classification task of the MNIST handwritten digit dataset, demonstrating its high-precision processing capability in pattern recognition tasks. In the image reconstruction task of the MNIST handwritten digit dataset, the test set consisted of monochrome MNIST images with random RGB color mixing. The system's image reconstruction performance on the test set was tested, and the mean squared error (MSE) reached 0.032, while the learned perceptual patch similarity (LPIPS) reached 0.181. This indicates that the system can efficiently recover high-quality image content from coded speckle, maintaining low reconstruction error while exhibiting good visual perception quality.

[0062] This embodiment verifies the feasibility and effectiveness of the system architecture described in this invention. Through the hardware-in-the-loop collaborative training mechanism, it achieves seamless integration of optical coding and digital processing, providing a reliable technical solution for the practical application of lensless computational imaging systems in tasks such as classification and image reconstruction.

[0063] Example 4: A support structure is provided between the programmable light field modulation layer and the image sensor described in Embodiments 1 and 3 to maintain a stable millimeter-level spacing. The support structure is made of a material with a low coefficient of thermal expansion to reduce the impact of ambient temperature changes on imaging stability.

[0064] Example 5: A readable storage medium storing a computer program, characterized in that, when the computer program is executed by a processor, it is capable of implementing the parameter joint optimization training algorithm in Embodiment 2.

[0065] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, the phrase "comprising an element defined as..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0066] The above technical solutions only embody the preferred technical solutions of the present invention. Any modifications that may be made by those skilled in the art to certain parts thereof embody the principles of the present invention and fall within the protection scope of the present invention.

Claims

1. A lensless optical image neural processing system, characterized in that, The system includes: an electrically addressable and programmable spatial light modulator, an image sensor, and an electronic digital processor; the spatial light modulator, the image sensor, and the electronic digital processor are arranged sequentially, the spatial light modulator and the image sensor constitute the optical domain front end, and the electronic digital processor and its built-in electronic digital neural network constitute the electronic digital back end. The system achieves integrated optimization of the optical front-end and the electronic digital back-end by synchronously inversely designing and end-to-end collaboratively training the modulation parameters of the spatial light modulator in multiple color channels and the weight parameters of the electronic digital neural network, thereby completing the color image processing task.

2. The system according to claim 1, characterized in that, The spatial light modulator has the ability to independently adjust the transmittance and transmittance spectrum of each spatial pixel, and its structure includes the integration of a liquid crystal modulation layer and a color filter layer.

3. The system according to claim 1 or 2, characterized in that, The spatial light modulator is the only optical modulation element in the system. The distance between the spatial light modulator and the image sensor is on the order of millimeters, forming a compact lensless imaging optical path.

4. The system according to claim 1, characterized in that, The image sensor is an array-type CMOS or CCD sensor that directly receives the encoded light field modulated by the spatial light modulator and outputs the corresponding light intensity distribution image to the electronic domain digital back-end.

5. The system according to claim 1 or 4, characterized in that, The electrical domain digital neural network of the electrical domain digital backend is a trainable neural network structure, which can be selected from one or a combination of fully connected networks, convolutional neural networks or U-Net structures depending on the task type.

6. The system according to claim 1, characterized in that, The end-to-end collaborative training method employs a hardware-in-the-loop training mechanism and includes the following steps: a) Perform forward propagation in the physical system to acquire the light intensity pattern output by the sensor; b) Input the light intensity pattern into the digital backend network and calculate the loss function; c) Update the network parameters of the digital backend using gradient descent; d) Calculate and update the approximate gradient of the modulatorable parameters of the spatial light modulator using a weighted simultaneous perturbation algorithm; e) Repeat steps a) to d) until the loss function converges.

7. The system according to claim 6, characterized in that, The weighted simultaneous perturbation algorithm estimates the gradient based on the loss change during two forward propagations by applying random small perturbations to the optical parameters. The formula is as follows: In the formula, For parameters The approximate gradient, Represents the system's loss function. These are the moduliable parameters of the optical field modulation layer. For small perturbations that follow a normal distribution.

8. The system according to claim 1, characterized in that, The system is suitable for incoherent light imaging scenarios, where the input light field is the target scene under natural light or LED illumination, and no additional optical lenses or imaging lenses are required.

9. The system according to claim 1, characterized in that, The system can be deployed directly after training without the need for secondary calibration or retraining for specific environments.

10. The system according to claim 1, characterized in that, The system supports multi-task processing, including but not limited to: one or more of the following: handwritten digit recognition, object classification, optical image reconstruction, and multispectral image processing.

Images