Camera modulation transfer function estimation method and system based on pulse neural network

Abstract

The invention belongs to the technical field of image quality online detection, and particularly relates to a camera modulation transfer function estimation method and system based on a pulse neural network, and the method comprises the steps: obtaining a test image collected by a camera, and selecting a region of interest from the test image; performing gray scale differential coding on the ROI image in the continuous time window to generate a sparse pulse sequence; the pulse sequence is input into a pulse neural network, and the pulse neural network is constructed based on a leakage integration distribution model and is used for carrying out feature extraction and dynamic contrast reconstruction on the pulse sequence and outputting a corresponding pulse response; gaussian smoothing and deconvolution are carried out on the impulse response, and a high-precision line spread function is reconstructed; carrying out fast Fourier transform on the modulation transfer function curve to obtain a complete modulation transfer function curve; by using an event-driven mechanism and sparse calculation characteristics of the spiking neural network, ultralow-power-consumption and ultralow-delay MTF estimation at the embedded front end is realized, and the requirement of online full inspection of a high-speed production line is met.

Description

Technical Field

[0001] This invention relates to the field of online image quality detection technology, specifically to a method and system for estimating the camera modulation transfer function based on a spiking neural network. Background Technology

[0002] Modulation Transfer Function (MTF), as a core standard for evaluating the resolution of imaging systems, has been listed as a mandatory test item in the factory inspection of cameras, lenses, mobile phone modules, and automotive vision sensors by standards such as ISO 12233 and GB / T 29298. Traditional testing methods rely on precision optical platforms and complex post-processing algorithms, which are cumbersome and require harsh environments, and can no longer meet the requirements of full-line inspection and real-time evaluation of embedded systems. Existing MTF testing technologies can be divided into three categories: 1. Image processing-based physical measurement method: This method strictly follows the principles of optical measurement. It requires a controlled laboratory environment, using a high-precision standard target (such as a bevel, slit, or star target), to acquire the target image through an imaging system, and then using digital image processing algorithms (such as edge detection, line spread function calculation, and Fourier transform) to derive the MTF curve step by step.

[0003] 2. CNN-based Deep Learning Regression Methods: These methods employ Convolutional Neural Networks (CNNs) to extract global features from images and regress single metrics such as MTF50. For example, Chinese patent CN121033025A discloses a quality assessment method for optical imaging systems based on multi-task deep learning. This method extracts features through a shared encoder and outputs metrics such as MTF and distortion in parallel by multiple decoders. While this method simplifies the process, its essence remains a "frame-and-post-processing" architecture. It requires caching the complete grayscale image first, followed by offline computation by the CPU / GPU. The entire process consumes a considerable amount of time due to computation after transmission and is sensitive to vibration, noise, and lighting fluctuations.

[0004] 3. General Processing Methods Based on SNNs: SNNs, due to their event-driven sparse computation, show significant potential for low-power hardware implementation. For example, Chinese patent CN115374924A discloses an SNN optimization method and hardware accelerator based on ternary pulses. By introducing a dual-membrane threshold and dedicated hardware design, it effectively improves network throughput and reduces inference latency. Existing SNN technologies all focus on solving general computer vision tasks such as image classification and recognition, and their network structures, encoding methods, and learning objectives are optimized for these tasks.

[0005] Currently, applying SNN to the specific regression task of MTF evaluation has the following problems: (1) Existing SNN schemes do not provide a method to effectively encode standard test images (such as slant edges) used for MTF evaluation into spatiotemporal pulse sequences. General encoding methods cannot focus on edge contrast information that is sensitive to MTF.

[0006] (2) The output of SNN in classification tasks is the class probability, while MTF evaluation requires the output of accurate line spread function curve and frequency domain response. There is no effective mapping and decoding mechanism between the two.

[0007] (3) The complete computational link from sparse pulses to high-precision MTF curves (including LSF reconstruction, FFT transformation, etc.) is still a blank in the existing SNN technology system.

[0008] Although SNNs have the inherent advantage of low power consumption and traditional deep learning provides an end-to-end evaluation approach, existing technologies have failed to combine the advantages of both. Therefore, there is an urgent need in this field for a feasible technical solution that can perform ultra-low latency and ultra-low power consumption and accurate MTF estimation in embedded front-ends in order to systematically solve the above problems. Summary of the Invention

[0009] In view of this, the present invention aims to overcome the problems of large ESF fitting error in high noise or low illumination of the improved slant method, the high floating-point computing power required by traditional deep learning methods based on convolutional neural networks (CNNs), making it difficult to run on embedded DSPs or FPGAs with low power consumption, and the lack of adaptation and optimization of existing spiking neural network (SNN) technology for the specific regression task of MTF evaluation. It provides a camera modulation transfer function estimation method and system based on spiking neural networks, and constructs an event-driven embedded processing pipeline from image acquisition to MTF curve generation, replacing the traditional "frame storage-post-processing" architecture.

[0010] On the one hand, a method for estimating the camera modulation transfer function based on a spiking neural network is provided, including the following steps: S1. Image Acquisition and Region Selection: Acquire test images captured by the camera. And select a local image of the ROI region containing the features of the test target from the test image. ; S2, Spatiotemporal Differential Coding: For continuous time windows Δt... Gray-scale difference calculation is performed, and the difference result is compared with a preset threshold to generate a binary spatiotemporal pulse sequence. This step converts the image data into an event stream suitable for SNN processing; S3. Spiking Neural Network Processing: The spatiotemporal pulse sequence is input into a pre-trained spiking neural network. The spiking neural network is constructed based on a leaky ensemble firing model and is used to perform nonlinear mapping and feature extraction on the input pulse sequence, ultimately outputting a set of pulse response sequences characterizing the dynamic changes in image edge contrast. ; S4. Line Spread Function Reconstruction: Gaussian smoothing is applied to the impulse response sequence output by the SNN to suppress noise, and then deconvolution is performed to reconstruct a high-precision, continuous line spread function (LSF). S5. Modulation Transfer Function Calculation: Perform Fast Fourier Transform (FFT) on the reconstructed line spread function and normalize the transformation result to obtain the complete modulation transfer function (MTF) curve, and further extract key evaluation indicators such as MTF50.

[0011] On the other hand, a camera modulation transfer function estimation system for implementing the above method is also provided. The system is deployed on an embedded hardware platform and includes, in sequence: (1) Spatiotemporal coding module, used to receive the image stream acquired by the camera, perform differential coding on the continuous image frames and generate pulse sequences; (2) A spiking neural network processing module, consisting of neuron connections based on a leaky integrated firing model, is used to receive and process the spiking sequence and output a spiking response; (3) Line spread function reconstruction module, used to perform smoothing and deconvolution operations on the impulse response to reconstruct the line spread function; (4) Modulation transfer function calculation module, used to perform frequency domain transformation and normalization on the line spread function and output the modulation transfer function curve.

[0012] Furthermore, the spatiotemporal coding module directly interfaces with the image sensor interface, receives data in the form of a pixel stream, and uses a differential calculation unit to process the pixels in the region of interest in parallel, thereby achieving real-time pulse coding without buffering complete image frames; The synaptic weights and neuron parameters used by the spiking neural network processing module are fixed-point parameters that have been pre-trained and stored in the embedded hardware platform's storage unit. The deconvolution operation in the line spread function reconstruction module is implemented using an algorithm based on Wiener filtering.

[0013] Furthermore, the system is preferably deployed on a field-programmable gate array, an application-specific integrated circuit, or a system-on-a-chip containing a neuromorphic computing unit.

[0014] Furthermore, the system also includes a communication module for uploading the calculated modulation transfer function curve and key indicators to a host computer or production line control system, supporting communication protocols based on UDP or serial ports.

[0015] Compared with the prior art, the present invention has the following beneficial effects: The spatiotemporal differential coding used in this invention can effectively suppress fixed pattern noise and slow illumination changes; combined with the dynamic feature extraction of SNN and the dedicated LSF reconstruction algorithm, it can still maintain high MTF estimation accuracy in complex environments and provide complete MTF curves for in-depth analysis.

[0016] Spatiotemporal differential coding directly processes the pixel stream while utilizing sparse event-driven computation of SNN, eliminating the need for complete image frame buffering and transmission processes, enabling MTF estimation to be completed within milliseconds, meeting the cycle time requirements of high-speed production lines; Furthermore, the embedded hardware deployment method adopted in this solution significantly reduces the overall power consumption of the system. The processing pipeline does not require a high-performance GPU or CPU and can be completed directly in the resource-constrained embedded front end, simplifying the system structure and reducing costs.

[0017] Other advantages, objectives, and features of the invention will be set forth in part in the description which follows, and in part will be apparent to those skilled in the art from the following examination, or may be learned from practice of the invention. The objectives and other advantages of the invention can be realized and obtained through the following description. Attached Figure Description

[0018] To make the objectives, technical solutions, and advantages of the present invention clearer, the preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings, wherein: Figure 1 The diagram shows the system block diagram for estimating the modulation transfer function of a camera based on a spiking neural network. Detailed Implementation

[0019] The following specific examples illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of the present invention. Unless otherwise specified, the following embodiments and features can be combined with each other.

[0020] The accompanying drawings are for illustrative purposes only and are schematic diagrams, not actual pictures. They should not be construed as limiting the invention. To better illustrate the embodiments of the invention, some parts in the drawings may be omitted, enlarged, or reduced, and do not represent the actual product dimensions. It is understandable to those skilled in the art that some well-known structures and their descriptions may be omitted in the drawings.

[0021] This invention provides an MTF testing method and system based on a spiking neural network (SNN). The invention will be further described in detail below with reference to the accompanying drawings.

[0022] Example 1: An MTF testing system architecture based on spiking neural networks like Figure 1 As shown, this system mainly includes five core modules: a spatiotemporal coding module, a spiking neural network (SNN) processing module, a line spread function (LSF) reconstruction module, an MTF curve estimation module, and a host computer communication module.

[0023] After the standard test source (such as a slanted target) is imaged by the camera under test, the image data is not buffered as a complete frame, but directly enters the processing chain of this system: First, the spatiotemporal coding module converts the image into a sparse pulse stream; then, the SNN processing module performs feature extraction and dynamic contrast reconstruction on the pulse stream; the LSF reconstruction module performs Gaussian smoothing and deconvolution on the SNN output to obtain a high-precision line spread function; finally, the MTF curve estimation module obtains the complete MTF curve through Fast Fourier Transform (FFT). The entire process does not require GPU participation and relies on the event-driven mechanism of pulse computing, which can be directly deployed on FPGA, ARM, SoC, or neuromorphic chips.

[0024] 1. Spatiotemporal coding module This module processes image data output from the camera sensor and is responsible for converting continuous image frame data into a sparse spatiotemporal pulse sequence, which is the first step in realizing event-driven processing.

[0025] Input: Local image patches captured by the camera To focus on key information, the system extracts a region of interest (ROI) from the source image, such as the region containing the standard beveled target edge.

[0026]

[0027] in, To acquire source images for the camera, This is a local image of the ROI region in the source image. for The coordinates (x, y) and width (w) and height (h) in the source image.

[0028] Processing: Image data preprocessing; this module performs differential encoding on the image within the Δt time window.

[0029] The difference result is compared with a preset threshold T to generate a sparse pulse stream, which greatly reduces the computational load.

[0030] in, for pixel coordinates, The ROI region of the image acquired at time t. The ROI region of the image acquired at time t-Δt. Differential encoding for the standard test source interval Δt time window, The threshold value set, Let be the sparse pulse flow at time t.

[0031] The final output is a sparse pulse matrix. , which serves as the input to the SNN processing module.

[0032] 2. SNN Processing Module This module consists of Leaky-Integrate-and-Fire (LIF) neurons, which are responsible for processing pulse sequences and extracting deep features for MTF estimation.

[0033] The LIF model is a simplified mathematical model of neuronal electrical activity used to simulate changes in neuronal membrane potential and impulse firing mechanisms. Its discrete expression is as follows:

[0034]

[0035] in, Let be the membrane potential at time t. For coefficients, For matrix parameters, Input the value at time t. For the threshold, This is the result of pulsed generation.

[0036] This module receives a pulse matrix from the spatiotemporal coding module. Through the transmission and nonlinear integration of multiple LIF neurons, a set of pulse sequences that characterize the changes in image edge contrast are finally output. .

[0037] 3. LSF Reconstruction Module This module reconstructs the discrete pulse sequence output by the SNN into a continuous line spread function.

[0038] First, the pulse sequence output by the SNN Accumulation and filtering are performed using a Gaussian convolution kernel. Perform smoothing: ,in, This represents the convolution operation. This is the initial outline after smoothing.

[0039] Then, to improve accuracy, Perform a deconvolution operation. Calculate the size of the feature map after deconvolution:

[0040]

[0041] in, Step size, For filling, The kernel size is [size]. for width, for high, The feature width after deconvolution, The feature height is obtained after deconvolution. Through this process, the high-precision line spread function result after reconstruction is obtained. .

[0042] 4. MTF Curve Estimation Module This module performs standard frequency domain transformation to obtain the MTF curve from the LSF.

[0043] For the reconstructed line spread function Perform a Fast Fourier Transform (FFT) to obtain its frequency domain representation. Normalize the amplitude of the frequency domain representation to obtain the modulation transfer function (MTF(f)). The system outputs a complete 32-point discrete MTF curve and calculates key evaluation metrics such as MTF50 (the spatial frequency corresponding to when the MTF value drops to 0.5).

[0044] Each module in this embodiment is replaceable: SNN can be replaced with SRM (Spike Response Model) or TNN (Time-coded Neural Network); FFT calculation can use DFT filter chain or lookup table transformation to further improve embedded efficiency.

[0045] Example 2: Final Inspection Results of a Certain Model of Downhole Camera Module on the Production Line Taking the final inspection of a certain type of downhole camera module on the production line as an example, the specific implementation parameters are as follows: 2.1 Hardware Platform Setup: FPGA is used as the core processor. Spatiotemporal coding, SNN inference, LSF reconstruction and FFT calculation are all implemented entirely by the FPGA's internal DSP and LUT.

[0046] 2.2 Parameter settings: The ROI region is selected near the edge of the slanted target. The size needs to be dynamically set according to the target imaging size. In this experiment, it is set to 64×256 pixels. The differential time window Δt is set to 20us, and the threshold T is set to 4 based on the sensor's background noise (64 gray levels, each gray level has 4 gray values).

[0047] The SNN uses a lightweight network consisting of an input layer, a hidden layer (128 LIF neurons), and an output layer (32 LIF neurons). The weights are pre-trained and fixed. The LIF neuron parameters are set to β=0.8 and θ=1.0.

[0048] 2.3 Work Process: A single test is initiated by a production line trigger signal. The camera acquires the test image, and the data stream goes directly into the FPGA. The spatiotemporal coding module completes differential and pulse conversion within approximately 50µs.

[0049] Subsequently, the pulse sequence is input into the SNN processing module, and forward propagation is completed in approximately 100µs. LSF reconstruction and MTF estimation are completed within T≤16 time steps (approximately 100µs).

[0050] The total latency of the entire processing flow is approximately 800µs, and the peak power consumption of the system was measured to be approximately 1.2W.

[0051] 2.4 Performance Results: In 1000 consecutive tests, the root mean square error (RMSE) of the MTF50 value estimated by the system and the result of the laboratory high-precision MTF measuring instrument was less than 2%, and the repeatability standard deviation was less than 0.5%, which met the cycle time and accuracy requirements of the high-speed online production line.

[0052] In summary, this invention acquires test images captured by a camera and selects regions of interest (ROIs). It then performs grayscale differential encoding on the ROI images within a continuous time window to generate sparse pulse sequences, which are input into a spiking neural network (SNN). The SNN, built on a leaky integrated firing model, is used for feature extraction and dynamic contrast reconstruction of the pulse sequences, outputting the corresponding pulse response. This response is then Gaussian smoothed and deconvolved to reconstruct a high-precision line spread function (LPF). A Fast Fourier Transform (FFT) is applied to obtain the complete modulation transfer function (MTF) curve. Utilizing the event-driven and sparse computation characteristics of the SNN, ultra-low power consumption and ultra-low latency MTF estimation are achieved in embedded front-ends, meeting the requirements of high-speed production lines for online and full-inspection operations.

[0053] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A method for estimating the modulation transfer function of a camera based on a spiking neural network, characterized in that, Includes the following steps: S1. Image Acquisition and Region Selection: Acquire test images captured by the camera. And select a local image of the ROI region containing the features of the test target from the test image. ; S2, Spatiotemporal Differential Coding: For continuous time windows Δt... Gray-scale difference calculation is performed, and the difference result is compared with a preset threshold to generate a binary spatiotemporal pulse sequence. ; S3. Spiking Neural Network Processing: The spatiotemporal pulse sequence is input into a pre-trained spiking neural network; the spiking neural network is constructed with a leaky integrated firing model to perform nonlinear mapping and feature extraction on the input pulse sequence, and finally outputs a set of pulse response sequences characterizing the dynamic changes in image edge contrast. ; S4. Line spread function reconstruction: Gaussian smoothing is applied to the impulse response sequence, followed by deconvolution to reconstruct a continuous line spread function (LSF). S5. Modulation transfer function calculation: Perform a fast Fourier transform on the reconstructed line spread function and normalize the amplitude spectrum of the transform result to obtain the modulation transfer function curve.

2. The method according to claim 1, characterized in that, In step S2, the generation of the binary spatiotemporal pulse sequence is specifically achieved through the following formula: in, for pixel coordinates, The ROI region of the image acquired at time t. The ROI region of the image acquired at time t-Δt. Differential encoding for the standard test source interval Δt time window, The threshold value set, Let be the sparse pulse flow at time t.

3. The method according to claim 1, characterized in that, In step S3, the dynamic behavior of the LIF model is described by the following discrete equations: in, Let be the membrane potential at time t. For coefficients, For matrix parameters, Input the value at time t. For the threshold, This is the result of pulsed generation.

4. The method according to claim 1, characterized in that, In step S4, the calculation method for the output feature map size in the deconvolution operation is as follows: in, Step size, For filling, The kernel size is [size]. for width, for high, The feature width after deconvolution, The height of the feature after deconvolution is shown. The reconstructed feature result is obtained after padding after deconvolution. .

5. A camera modulation transfer function estimation system for implementing the method of any one of claims 1 to 4, characterized in that, The system is deployed on an embedded hardware platform and includes: (1) Spatiotemporal coding module, used to receive the image stream acquired by the camera, perform differential coding on the continuous image frames and generate pulse sequences; (2) A spiking neural network processing module, consisting of neuron connections based on a leaky integrated firing model, is used to receive and process the spiking sequence and output a spiking response; (3) Line spread function reconstruction module, used to perform smoothing and deconvolution operations on the impulse response to reconstruct the line spread function; (4) Modulation transfer function calculation module, used to perform frequency domain transformation and normalization on the line spread function and output the modulation transfer function curve.

6. The system according to claim 5, characterized in that, The spatiotemporal coding module directly interfaces with the image sensor, receives data in the form of a pixel stream, and uses a differential computing unit to process pixels in the region of interest in parallel, thereby achieving real-time pulse coding without buffering complete image frames.

7. The system according to claim 5, characterized in that, The synaptic weights and neuron parameters used by the spiking neural network processing module are fixed-point parameters that have been pre-trained and stored in the embedded hardware platform's storage unit.

8. The system according to claim 5, characterized in that, The deconvolution operation in the line spread function reconstruction module is implemented using an algorithm based on Wiener filtering.

9. The system according to any one of claims 5 to 8, characterized in that, The embedded hardware platform is a field-programmable gate array, an application-specific integrated circuit, or a system-on-a-chip containing a neuromorphic computing unit.

10. The system according to any one of claims 5 to 8, characterized in that, The system also includes a communication module, which is used to upload the calculated modulation transfer function curve and key indicators to the host computer or production line control system, and supports communication protocols based on UDP or serial port.

Images