System and method for integrated spiking neural network training in sensor
By integrating a spiking neural network into the sensor and using the CMOS sensing module to generate image light intensity signals and random numbers in different modes, the problems of single sensor function and data migration are solved, and a high-efficiency, low-latency sensing and computing closed loop is realized.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-12-10
- Publication Date
- 2026-03-13
AI Technical Summary
Existing intelligent sensing technologies suffer from limited sensor functionality, leading to energy consumption and latency issues due to frequent data migration, as well as the inability of sensors to provide random computing resources.
By integrating a spiking neural network into the sensor, the CMOS sensing module operates in different modes to generate image light intensity signals and random numbers, and is trained through pulse conversion, weight storage, and weight update modules to achieve autonomous perception and computation.
It achieves an efficient closed loop of sensing, computing and learning within the sensor, reduces energy consumption and latency, improves hardware resource utilization and integration, and provides autonomy and computational completeness.
Smart Images
Figure CN121660003A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of sensor integration technology, and in particular to a system and method for training a spiking neural network integrated within a sensor. Background Technology
[0002] In existing intelligent sensing technologies, sensors are generally defined as a single-function signal conversion device. Its core task is to convert analog signals such as light, sound, heat, and force from the external physical world into standardized electrical or digital signals through specific sensing principles (such as photoelectric effect and piezoelectric effect).
[0003] In the traditional sensing-transmission-processing separation architecture of intelligent sensing technology, the sensor function is limited to converting physical signals into electrical signals. This traditional separation architecture has inherent performance bottlenecks: frequent and large amounts of data migrate between sensing, storage and processing units, resulting in significant energy consumption and latency, and frequent memory wall problems; the sensor itself is positioned as a passive data source, which makes it lack the ability to actively and initially understand information. Summary of the Invention
[0004] The purpose of this invention is to improve the problem of the limited function of sensors as passive data sources in the prior art, and to provide a system and method for training spiking neural networks integrated within a sensor.
[0005] To achieve the above-mentioned objectives, the embodiments of the present invention provide the following technical solutions:
[0006] A method for training an integrated spiking neural network within a sensor includes the following sub-steps:
[0007] A spiking neural network is constructed based on a CMOS sensing module, a pulse conversion circuit, a weight storage module, a neuron module, and a weight update module.
[0008] Unsupervised training of spiking neural networks is performed based on the stochastic STDP mechanism, and neurons are classified after training is completed;
[0009] The unsupervised training specifically involves the CMOS sensing module operating in shutter-open mode to acquire external image information and generate an image light intensity electrical signal to the pulse conversion circuit.
[0010] The pulse conversion circuit encodes the image light intensity electrical signal and converts it into a presynaptic pulse sequence, which is then input to the weight storage module.
[0011] The stored weights in the weight storage module are multiplied and accumulated with the corresponding presynaptic pulses, and the result is accumulated onto the membrane potential of the corresponding postsynaptic neuron in the neuron module.
[0012] The postsynaptic pulse is sent to the weight update module, and at the same time the weight storage module sends the historical presynaptic pulse sequence within the most recent n cycles to the weight update module;
[0013] The weight update module receives postsynaptic pulses and random numbers generated by the CMOS sensing module, triggering a learning process based on the stochastic STDP mechanism to update the synaptic weights and threshold voltages of neurons.
[0014] To address the frequent memory wall problem caused by the separation of sensing, transmission, and processing architecture in existing intelligent sensing systems, this invention integrates a CMOS sensing module, input-pulse conversion circuit, weight storage module, neuron module, and weight update module to construct a spiking neural network. This allows the entire process, from image information acquisition, pulse encoding, multiplication and accumulation calculation to pulse generation and learning, to form a closed loop within the sensor chip. This completely avoids the frequent transfer of massive amounts of raw image data or intermediate calculation results to off-chip storage and processing units, fundamentally solving the power consumption and latency bottlenecks caused by data migration. Furthermore, addressing the issue of sensors in traditional architectures serving only as passive data sources with limited functionality, this invention enables the CMOS sensing module to maintain basic image sensing functions while transforming into a physical true random number generator based on quantum random processes when needed. This expands the core functionality of the sensor, transforming it from a passive signal converter into an active unit capable of both sensing and supplying key computing resources (random entropy). The integrated construction of a spiking neural network and a dual-mode reconfigurable CMOS sensor not only eliminates the memory wall through localized processing, but also enhances the autonomy and computational completeness of the sensor through functional fusion. It enables the complete iteration from information perception to adaptive intelligent learning within the physical boundaries of the sensor, achieving high energy efficiency, low latency, and high privacy edge intelligent perception.
[0015] Furthermore, a method for training an integrated spiking neural network within a sensor is provided. The spiking neural network comprises N presynaptic neurons and M postsynaptic neurons. All presynaptic and postsynaptic neurons are fully connected to form N×M synapses, and each postsynaptic neuron is connected to a weight update module.
[0016] Furthermore, a method for training a pulse neural network integrated within a sensor, wherein the CMOS sensing module has two modes: shutter open mode and shutter closed mode, and includes an optical shutter, lens, LED, and sensor photosensitive pixel array assembly;
[0017] In the shutter opening mode, external light is focused by the lens onto the CMOS sensing module to generate an image light intensity electrical signal, which serves as the raw input to the neural network.
[0018] In the shutter-closed mode, external light is blocked, creating a darkroom environment. An integrated LED emits a weak light of controllable intensity to activate the quantum random process to generate random numbers.
[0019] The existing sensors described above typically operate in a single photoelectric conversion mode, their function fixed to continuously acquire and output analog or digital light intensity signals. Essentially, they are passive data suppliers. This fixed mode prevents the sensor from providing any other critical computing resources to subsequent processing units besides the sensed data. When performing tasks requiring high-quality randomness, such as training random neural networks, an additional independent random number generator must be relied upon, leading to increased system complexity, reduced integration, and additional power consumption and latency. This invention addresses this by enabling the CMOS sensing module to perform its basic image sensing function through a shutter-open mode, converting external light signals into the raw input electrical signals required by the network. Through a shutter-closed mode, the same photosensitive array is used to excite an inherent quantum random process in a controllable low-light environment, transforming it into a physically true random number generator that operates on demand. This solves the problem that existing sensors, due to their single function and fixed mode, cannot provide random numbers beyond sensing, forcing the system to rely on external discrete modules, thus sacrificing integration and energy efficiency. This invention combines image perception and random number generation into a deep and dynamically switchable combination of functions, enabling a single sensor hardware to be flexibly reconfigured between data sources and entropy sources. This not only greatly improves the utilization of hardware resources and the integration of sensor chips, but more importantly, it provides a self-contained and secure source of randomness for completing a complete "perception-computation-learning" closed loop within the sensor, which is the core foundation for realizing integrated training within the sensor.
[0020] Furthermore, a method for training a spiking neural network integrated within a sensor, wherein encoding the image light intensity electrical signal includes the following sub-steps:
[0021] If the input is a grayscale image, the grayscale value of each pixel is linearly mapped to a pulse sequence of a specific frequency;
[0022] If the input is an RGB color image, the pixel values of the R, G, and B channels are independently and linearly mapped to pulse sequences of corresponding frequencies, and the three-channel signals are encoded into three independent pulse sequences.
[0023] The existing technology, when inputting continuous image light intensity signals acquired by the sensor into the computing unit, typically uses analog-to-digital conversion followed by direct transmission of the original pixel data array. This encoding method generates a large amount of redundant and dense data, whose data format is severely incompatible with the sparse, event-driven computing paradigm favored by subsequent spiking neural networks. This results in significant internal data handling burden and static power consumption even when the computing unit is integrated within the sensor. This invention addresses this by employing an encoding method that linearly maps pixel grayscale values to pulse frequencies for grayscale images and a three-channel independent parallel linear mapping encoding method for RGB color images. This directly converts continuous analog light intensity information into discrete, frequency-modulated pulse event sequences. It solves the technical problems of high data redundancy and incompatibility with the computational characteristics of spiking neural networks in existing encoding methods, leading to data congestion and energy loss within the integrated architecture. This invention constructs a universal pulse frequency encoding framework applicable to both single-channel (grayscale) and multi-channel (color) images, achieving fundamental data compression and format conversion from the information source. Its unified mapping principle (linear frequency encoding) provides a simple and consistent hardware implementation path for processing different types of image inputs.
[0024] Furthermore, in a method for training a spiking neural network integrated within a sensor, the formula for multiplying and accumulating the stored weights and their corresponding presynaptic pulses is as follows:
[0025] ;
[0026] in, Let be the membrane potential at time t. For leakage voltage, Let be the weight value of the i-th synapse at time t. This represents the presynaptic pulse released by the i-th presynaptic neuron, with a value of either 0 or 1. This is the reset voltage. is the threshold voltage, and N is the number of presynaptic neurons;
[0027] Each neuron has a dynamic threshold voltage. When the conditions are met At this time, the neuron fires a postsynaptic pulse, and its membrane potential is then reset.
[0028] The above-described solutions address the challenge of efficiently coupling pulse events with weights and accurately simulating neuron integration-firing dynamics when processing forward computation in spiking neural networks. This invention, by precisely defining the membrane potential calculation formula, integrates the multiplication and accumulation of weights with pulse inputs, real-time comparison with dynamic thresholds, and membrane potential reset after firing into a clear, discrete mathematical process. This solves the technical problem of existing solutions struggling to balance accuracy with hardware overhead and analog-to-digital fusion, hindering the effective operation of spiking neural networks within resource-constrained sensor chips. This invention, based on the multiplication and accumulation of pulse events, real-time comparison with dynamic thresholds, and post-firing reset, utilizes discrete time steps and clear logic gate operations. This allows the timing information of pulses to be effectively captured through discrete membrane potential integration, while ensuring efficient implementation through low-power digital or mixed-signal circuits. This provides crucial mathematical and circuit design basis for building high-performance spiking neural network computing cores within sensors and is a necessary computational foundation for achieving efficient sensor-computer fusion.
[0029] Furthermore, a method for training an integrated spiking neural network within a sensor, wherein the random numbers generated by the CMOS sensing module include the following sub-steps:
[0030] When a postsynaptic pulse is generated, the CMOS sensing module switches from shutter open mode to shutter closed mode, activating the integrated LED to emit a weak light of a preset intensity;
[0031] Under dim light conditions in a dark room, the random quantum events of photon arrival are superimposed with the thermal noise of the sensor to generate inherently unpredictable and discrete random charge signals on the CMOS pixel array.
[0032] The CMOS sensor module in shutter-off mode is periodically sampled and quantized to obtain a random digital sequence.
[0033] The above-mentioned solutions, in order to support random STDP training of the spiking neural network integrated in the sensor, require the provision of high-quality random numbers. The usual solution is to integrate a separate dedicated physical true random number generator circuit outside or on top of the sensor chip. This results in a significant increase in the overall chip area, design complexity and power consumption of the system, which is contrary to the development trend of sensor chip miniaturization and low power consumption. The physical random process of the external or independent physical true random number generator is completely separated from the sensor's optical signal sensing in terms of physical mechanism and working timing. It is difficult to achieve accurate and low-latency synchronization with the internal spiking events (learning trigger points) of the neural network, which may affect the accurate execution of the learning rules. This invention switches the sensor's operating mode from sensing to random number generation by triggering a postsynaptic pulse. In a constructed dark, low-light environment, it directly utilizes the inherent physical noise of the CMOS pixel array, stemming from quantum effects and thermodynamic fluctuations, as the physical source of random entropy. Through the readout and quantization circuitry integrated within the sensor, this physical noise signal is periodically sampled and quantized, directly outputting a digital random sequence. This solves the technical problems of existing solutions, which rely on independent external hardware for random number generation, resulting in low system integration, high overhead, and difficulty in tightly coordinating the random number stream with internal learning events. This invention, through the combined technical features of using postsynaptic pulses to drive sensor mode reconstruction and developing the inherent physical noise of the CMOS sensor into controllable random numbers, achieves random numbers not provided by external modules but derived from the sensor itself under specific instructions. This not only achieves a high-density fusion of sensing and randomness generation functions at the physical level, saving hardware resources, but also ensures that the randomness driving learning is strictly aligned with the learning events in time and physically bound to the sensor hardware in terms of its source. This provides an indispensable and highly integrated randomness guarantee for efficient and secure autonomous learning within the sensor.
[0034] Furthermore, a method for training a spiking neural network integrated within a sensor, wherein the superposition of random quantum events arriving with sensor thermal noise includes the following sub-steps:
[0035] The photosensitive pixel array of a CMOS sensor is exposed to the weak light field generated by an LED. The particle characteristics of light dominate the photoelectric conversion process. The time, position, and ability of a single photon to reach a specific pixel point to generate a photogenerated carrier result in a completely random quantum event.
[0036] Meanwhile, the thermally excited charge carriers generated by the thermal effect of the CMOS sensor itself become non-negligible background noise. The randomness of photon arrival and the randomness of thermal noise are independently superimposed, together generating an inherently unpredictable and discrete random charge signal on each photosensitive pixel.
[0037] Through the above-described solutions, existing technologies typically rely on independent dedicated random number generation modules or processor-based pseudo-random algorithms to provide randomness. These modules are physically separate from the sensor as the front end, and the generation of random numbers is unrelated to the core physical processes of the sensor. The sensor must pay additional hardware overhead or computing resources for generating randomness, leading to reduced integration and energy efficiency. This invention explicitly reveals and utilizes the inherent, rather than externally injected, physical randomness of CMOS sensors under specific operating conditions, specifically elucidating the generation principle of their random charge signals: In a controllable low-light environment, the output signal of the photosensitive pixel array of a CMOS sensor is inherently unpredictable because it is dominated by two independent random processes originating from the physical nature of the device—the quantum random event of photon arrival and the carrier noise thermally excited by silicon material. This solves the fundamental technical problem in existing technologies where sensors themselves lack the ability to generate random signals, increasing system complexity, power consumption, and latency. This invention, by actively constructing conditions to manifest and utilize the core characteristics of the sensor's inherent quantum noise and thermal noise, transforms the CMOS sensor for the first time from a purely deterministic sensing device into a dual-function "sensing-entropy source" device that can generate high-quality physical random signals on demand. Randomness is not added to the sensor, but extracted from its own physical processes, making the generation of random numbers deeply bound to the sensor's physical entity and operating state.
[0038] Furthermore, a method for training an integrated spiking neural network within a sensor, wherein triggering a learning process based on a stochastic STDP mechanism to update the synaptic weights and threshold voltages of neurons includes the following sub-steps:
[0039] Perform long-term enhanced LTP updates, and when a postsynaptic pulse occurs, probabilistically enhance synapses that have previously experienced a presynaptic pulse within a specific time window;
[0040] For a synapse with a time interval of i, the currently obtained random number is compared with the corresponding preset probability parameter Pi. If the random number is less than Pi, the synapse weight is increased by a fixed value.
[0041] Perform a long-term suppressed LTD update, and after the long-term suppressed LTD update, apply a global small decay to all synaptic weights;
[0042] Increase the threshold voltage of the neuron that has just fired a pulse by increasing the threshold voltage by an amount that increases the threshold voltage.
[0043] Furthermore, a method for training an integrated spiking neural network within a sensor, wherein the synaptic weights and threshold voltages of the updated neurons are calculated using the following formula:
[0044] ;
[0045] in, , and These are constants, representing the increase in LTP, the decrease in LTD, and the increase in threshold voltage, respectively. The postsynaptic pulse at time t, with a value of 0 or 1; Let be the threshold voltage value at time t. These are the weights of the spiking neural network. This is an indicator variable based on the result of random number comparison, taking the value 0 or 1. i is the time interval between the presynaptic pulse and the postsynaptic pulse; n is the number of historically stored presynaptic pulse cycles.
[0046] The above approach relies on a deterministic mechanism when implementing STDP rules in hardware using existing spiking neural networks. This mechanism directly calculates and updates weights using a fixed function based on the precise time difference between the pulses before and after a synapse. This requires the circuit to have high-precision time detection and complex function calculation capabilities, resulting in high hardware implementation complexity, high power consumption, sensitivity to timing jitter, and poor robustness. This invention addresses the technical problem of existing deterministic STDP rules being difficult to integrate and apply efficiently and stably within sensor chips due to their complex hardware implementation and excessive time sensitivity. By defining a probabilistic LTP decision-making mechanism driven by physical random numbers, this invention transforms the original weight updates, which relied on precise timing and complex calculations, into simple comparisons and fixed-value addition / subtraction operations driven by local random events. This significantly reduces the implementation difficulty and power consumption of the learning circuit. Furthermore, by introducing controllable randomness, it enhances the network's exploration ability and robustness, enabling the spiking neural network within the sensor to perform adaptive learning in a hardware-friendly and energy-efficient manner.
[0047] A system for integrating spiking neural network training within a sensor includes a spiking neural network construction module and a spiking neural network training module;
[0048] The spiking neural network module is constructed based on a CMOS sensing module, a pulse conversion circuit, a weight storage module, a neuron module, and a weight update module.
[0049] The spiking neural network training module performs unsupervised training of the spiking neural network based on the stochastic STDP mechanism, and classifies the neurons after training is completed.
[0050] By addressing the limitations of existing intelligent sensing systems caused by the separation of sensing, transmission, and processing architectures, as well as the technical problem of limited sensor functionality, the system provided by this invention concretizes the integrated method and process into two core functional entities by constructing a "spiking neural network module and a spiking neural network training module." Through modular integration of sensing, computing, and learning functions, this system ensures that the entire intelligent processing flow from signal input to classification is executed in a closed loop within the sensor system. This also achieves the technical effects of eliminating data off-chip handling and expanding the intelligent functions of the sensor, providing a clear system architecture and engineering implementation path for manufacturing intelligent sensor chips that integrate sensing and computing. Attached Figure Description
[0051] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.
[0052] Figure 1 This is a flowchart of a method for training a spiking neural network integrated within a sensor.
[0053] Figure 2 This is a diagram of the neuron structure of a spiking neural network.
[0054] Figure 3 This is a schematic diagram of the CMOS sensing module.
[0055] Figure 4 This is a flowchart of the unsupervised training process for a spiking neural network.
[0056] Figure 5 This is a schematic diagram illustrating the learning effect of a spiking neural network in an example. Detailed Implementation
[0057] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.
[0058] It should be noted that similar reference numerals and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. Furthermore, in the description of this invention, the terms "first," "second," etc., are used only for distinguishing descriptions and should not be construed as indicating or implying relative importance, or suggesting any such actual relationship or order between these entities or operations. Additionally, the terms "connected," "linked," etc., can refer to a direct connection between elements or an indirect connection via other elements.
[0059] This invention is achieved through the following technical solutions, such as... Figure 1 As shown, a method for training an integrated spiking neural network within a sensor includes the following sub-steps:
[0060] S1: Construct a spiking neural network based on a CMOS sensing module, a pulse conversion circuit, a weight storage module, a neuron module, and a weight update module;
[0061] like Figure 2 As shown, the spiking neural network includes N presynaptic neurons and M postsynaptic neurons. All presynaptic neurons and postsynaptic neurons are connected in a fully connected manner to form N×M synapses. Each postsynaptic neuron is connected to a weight update module.
[0062] The logical relationship of the spiking neural network is as follows: when any postsynaptic neuron emits a pulse, the weight update module resets the membrane potential of all postsynaptic neurons to the initial value and updates the relevant synaptic weights and threshold voltages.
[0063] like Figure 3 As shown, the CMOS sensing module has two modes: shutter open mode and shutter closed mode, and includes components such as an optical shutter, lens, LED and sensor photosensitive pixel array;
[0064] In shutter-open mode, ambient light is focused by the lens onto the CMOS sensor module, generating an image light intensity electrical signal, which serves as the raw input to the neural network.
[0065] In shutter-closed mode, external light is blocked, creating a darkroom environment. Integrated LEDs emit a weak light of controllable intensity to activate the quantum random process to generate random numbers.
[0066] The weight storage module is used to store and manage network state information, specifically including:
[0067] Store the current weight values of all N×M synapses, which are used for multiplication and accumulation calculations with the input pulses during forward propagation;
[0068] The pulse history sequence of all presynaptic neurons within the last n clock cycles is cached and output to provide the input data required by the weight update module to determine the temporal correlation of presynaptic and postsynaptic pulses.
[0069] The neuron module is used to simulate the dynamic characteristics of biological neurons and realize the functions of membrane potential integration and pulse firing. It includes a membrane potential accumulation unit, a threshold comparator, and a pulse generation and reset control module.
[0070] The membrane potential accumulation unit receives the multiplication and accumulation result from the weighted storage module, adds the previous membrane potential to the multiplication and accumulation result and subtracts the leakage voltage to obtain the current membrane potential.
[0071] The threshold comparator compares the accumulated membrane potential with the current threshold voltage, and triggers a pulse firing event when the membrane potential reaches or exceeds the threshold.
[0072] The pulse generation and reset control module generates a control signal to reset its own membrane potential to the initial value while issuing the pulse, and sends the output postsynaptic pulse to the weight update module.
[0073] The weight update module is responsible for executing the learning rules based on the random STDP mechanism, including a random number interface, a time-series correlation judgment logic module, and a parameter update logic module.
[0074] The random number interface receives a random number sequence generated by the CMOS sensor module in shutter-closed mode;
[0075] The temporal correlation judgment logic module determines which synapses meet the enhancement conditions and their corresponding time intervals based on the historical presynaptic pulse sequence obtained from the weight storage module and the received postsynaptic pulse.
[0076] The parameter update logic module performs probabilistic calculations using random numbers according to the random STDP rule, and adjusts the synaptic weights that meet the conditions through long-term enhancement (LTP), long-term suppression (LTD), and threshold voltage adjustment.
[0077] It should be noted that the pulse conversion circuit, weight storage module, neuron module, and weight update module are integrated with the CMOS sensing module on the same sensor chip in the form of embedded hardware circuits.
[0078] like Figure 4 As shown, S2: Unsupervised training of the spiking neural network is performed based on the stochastic STDP mechanism, and the neurons are classified after training.
[0079] S21: The CMOS sensor module operates in shutter-open mode, acquiring external image information, generating corresponding image light intensity electrical signals (single-channel intensity signal for grayscale images, and three-channel R, G, and B signals for color images), and outputting them to the pulse conversion circuit.
[0080] S22: The pulse conversion circuit encodes the image light intensity electrical signal and converts it into a presynaptic pulse sequence that can be processed by the spiking neural network, which is then input to the weight storage module.
[0081] The encoding includes the following sub-steps:
[0082] If the input is a grayscale image, the grayscale value of each pixel is linearly mapped to a pulse sequence of a specific frequency (for example, grayscale values 0 and 255 are encoded as no pulse and a single pulse, respectively).
[0083] If the input is an RGB color image, the pixel values of the R, G, and B channels are independently and linearly mapped to pulse sequences of corresponding frequencies, and the three-channel signals are encoded into three independent pulse sequences.
[0084] S23: The stored weights in the weight storage module are multiplied and accumulated with the corresponding presynaptic pulses, and the result is accumulated on the membrane potential of the corresponding postsynaptic neuron in the neuron module.
[0085] Specifically, S23 refers to the membrane potential of each postsynaptic neuron at time t, expressed as:
[0086] ;
[0087] in, Let be the membrane potential at time t. For leakage voltage, Let be the weight value of the i-th synapse at time t. This represents the presynaptic pulse released by the i-th presynaptic neuron, with a value of either 0 or 1. This is the reset voltage (which takes effect after the neuron fires a pulse). is the threshold voltage, and N is the number of presynaptic neurons;
[0088] Each neuron has a dynamic threshold voltage. When the conditions are met At this time, the neuron fires a postsynaptic pulse, and its membrane potential is then reset.
[0089] In this embodiment, the initial value of the threshold voltage is set to 200 (analog or digital units based on a specific circuit design), and is dynamically adjusted during training according to step S25. (During training, after each pulse fired by the neuron, its threshold voltage is increased by the amount of threshold voltage boost according to step S2527.) ,For example =5).
[0090] S24: When a postsynaptic neuron in the neuron module emits a pulse, the postsynaptic pulse is sent to the weight update module. At the same time, the weight storage module sends the historical presynaptic pulse sequence within the most recent n cycles to the weight update module.
[0091] S25: The weight update module receives postsynaptic pulses and random numbers generated by the CMOS sensing module, triggers a learning process based on the stochastic STDP mechanism, and updates the synaptic weights and threshold voltages of neurons.
[0092] The random number generated by the CMOS sensing module includes the following sub-steps:
[0093] S251: When a postsynaptic pulse is generated, the CMOS sensing module switches from shutter open mode to shutter closed mode, activating the integrated LED to emit a weak light of a preset intensity;
[0094] S252: Under low-light conditions in a dark room, the random quantum events of photon arrival are superimposed with the thermal noise of the sensor to generate an inherently unpredictable and discrete random charge signal on the CMOS pixel array.
[0095] S253: Periodically sample and quantize the CMOS sensor module in shutter-off mode to obtain a random digital sequence.
[0096] S252 includes the following steps:
[0097] S2521: The photosensitive pixel array of the CMOS sensor is exposed to the weak light field generated by the LED. The particle characteristics of light dominate the photoelectric conversion process. The time, position and whether a single photon can generate a photogenerated carrier when it arrives at a specific pixel point are completely random quantum events.
[0098] S252: At the same time, the thermally excited charge carriers generated by the CMOS sensor itself due to thermal effects become non-negligible background noise. The randomness of photon arrival and the randomness of thermal noise are independently superimposed on each other, together generating an inherently unpredictable and discrete random charge signal on each photosensitive pixel.
[0099] In this embodiment, the sampling period is set to 1μs and the quantization bit width is 8 bits. Each sampling obtains a random analog voltage from a specified pixel area and converts it into a random integer output between 0 and 255 to form a random number sequence.
[0100] The triggering of the learning process based on the stochastic STDP mechanism, which updates the synaptic weights and threshold voltages of neurons, includes the following sub-steps:
[0101] S2524: Perform long-term enhancement LTP updates, when a postsynaptic pulse occurs, probabilistically enhance synapses that have previously experienced a presynaptic pulse within a specific time window (e.g., the most recent n=4 cycles stored);
[0102] S2525: For a synapse with a time interval of i (i=1,2,3,4, representing the cycle from near to far), compare the currently obtained random number with the corresponding preset probability parameter Pi. If the random number is less than Pi, the synapse weight is increased by a fixed value.
[0103] S2526: Perform a Long-Term Suppression (LTD) update, followed by a global small decay applied to all synaptic weights.
[0104] S2527: Increases the threshold voltage of the neuron that has just fired a pulse, increasing the threshold voltage by an amount.
[0105] Wherein, the fixed value = C1, the minute attenuation = C2, and the preset parameter Pi satisfies P1>P2>P3>P4, so that the shorter the time interval between the presynaptic pulse and the postsynaptic pulse, the greater the probability that the connection weight is enhanced.
[0106] The formulas for S2524-S257 are as follows:
[0107] ;
[0108] in, , and These are constants, representing the increase in LTP, the decrease in LTD, and the increase in threshold voltage, respectively. The postsynaptic pulse at time t, with a value of 0 or 1; Let be the threshold voltage value at time t. These are the weights of the spiking neural network. This is an indicator variable based on the result of random number comparison, taking the value 0 or 1. i is the time interval between the presynaptic pulse and the postsynaptic pulse; n is the number of historically stored presynaptic pulse cycles.
[0109] In the embodiments, the learning effect of the spiking neural network is as follows: Figure 5 As shown, this mainly illustrates the dynamic changes in synaptic weight and threshold voltage of a postsynaptic neuron after the release of a pulse. pre1, pre2, and pre3 are input pulse sequences from three different presynaptic neurons, respectively. , , is the weight value of the corresponding synapse, and post is the pulse signal output by the postsynaptic neuron; This indicates the threshold voltage of the neuron, prior to the generation of the postsynaptic pulse post, and the presynaptic pulses pre1, pre2, and pre3, respectively, and their corresponding weights. , , The multiplication and accumulation operation is performed, and the calculation result is continuously accumulated to the membrane potential of the postsynaptic neuron, causing it to gradually rise until it exceeds the threshold voltage. At this time, the postsynaptic neuron releases a post impulse. After the post impulse is generated, the synaptic weights are updated through long-term potentiation (LTP), which manifests as... and The curve shows upward step jumps of varying magnitudes, followed by a slight global decay of all weights to achieve long-term suppression (LTD), which manifests as... , and A step drop of the same amplitude on the curve corresponds to an adjustment in the neuron's threshold voltage, which manifests as... The curve rises.
[0110] S26: Repeat steps S21-S25 to iteratively train the spiking neural network. After training, class labels are assigned to neurons by observing the response patterns of each postsynaptic neuron to samples of known categories.
[0111] The logic for determining whether iterative training is complete is as follows: It simultaneously uses an accuracy threshold or the maximum number of training iterations for judgment.
[0112] If the output accuracy of the spiking neural network exceeds the accuracy threshold (e.g., 90% classification accuracy), then the training of the spiking neural network is complete.
[0113] The training of the spiking neural network is complete when the maximum number of training iterations (e.g., 10,000) is reached.
[0114] In this embodiment, after training, different postsynaptic neurons in the spiking neural network selectively respond to image feature patterns directly acquired by the sensor. For example, in a scenario used for surveillance or object recognition, the sensor is fixedly facing an area. During the training phase, the CMOS sensor, through a shutter-open mode, allows the spiking neural network to learn unsupervised different visual patterns (such as "contours," "brightness changes caused by motion trajectories," etc.) corresponding to typical objects (such as "people," "vehicles," and "background") appearing in that area. After training, to configure the classification function, a set of samples with known categories are input into the spiking neural network: a person and a vehicle are sequentially allowed to enter the area, and the CMOS sensing module is triggered to acquire images respectively. Simultaneously, the category is manually labeled for each acquisition. When the spiking neural network processes these labeled samples, it observes that some neurons continuously fire when the "person" sample appears, while other neurons specifically respond to the "vehicle" sample. Based on this response pattern, these neurons can be assigned the category labels "human" or "vehicle." Neurons that are always active but do not respond to specific objects can be classified as "background." Through this process, the mapping configuration from the sensor's raw signals to high-level semantic categories ("people," "vehicles," and "background") is completed.
[0115] Embodiments of this application provide a system for integrating spiking neural network training within a sensor, including a spiking neural network construction module and a spiking neural network training module;
[0116] The spiking neural network module is constructed based on a CMOS sensing module, a pulse conversion circuit, a weight storage module, a neuron module, and a weight update module.
[0117] The spiking neural network training module performs unsupervised training of the spiking neural network based on the stochastic STDP mechanism, and classifies the neurons after training is completed.
[0118] Embodiments of this application provide a computer-readable storage medium storing instructions that, when executed by a computer, perform a method for training an integrated spiking neural network within a sensor.
[0119] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.
Claims
1. A method for training an integrated spiking neural network within a sensor, characterized in that, Includes the following sub-steps: A spiking neural network is constructed based on a CMOS sensing module, a pulse conversion circuit, a weight storage module, a neuron module, and a weight update module. Unsupervised training of spiking neural networks is performed based on the stochastic STDP mechanism, and neurons are classified after training is completed; The unsupervised training specifically involves the CMOS sensing module operating in shutter-open mode to acquire external image information and generate an image light intensity electrical signal to the pulse conversion circuit. The pulse conversion circuit encodes the image light intensity electrical signal and converts it into a presynaptic pulse sequence, which is then input into the weight storage module. The stored weights in the weight storage module are multiplied and accumulated with the corresponding presynaptic pulses, and the result is accumulated onto the membrane potential of the corresponding postsynaptic neuron in the neuron module. The postsynaptic pulse is sent to the weight update module, and at the same time the weight storage module sends the historical presynaptic pulse sequence within the most recent n cycles to the weight update module; The weight update module receives postsynaptic pulses and random numbers generated by the CMOS sensing module, triggering a learning process based on the stochastic STDP mechanism to update the synaptic weights and threshold voltages of neurons.
2. The method for training an integrated spiking neural network within a sensor according to claim 1, characterized in that, The spiking neural network includes N presynaptic neurons and M postsynaptic neurons. All presynaptic and postsynaptic neurons are fully connected to form N×M synapses. Each postsynaptic neuron is connected to a weight update module.
3. The method for training an integrated spiking neural network within a sensor according to claim 1, characterized in that, The CMOS sensing module has two modes: shutter open mode and shutter closed mode, and includes an optical shutter, lens, LED and sensor photosensitive pixel array assembly. In the shutter opening mode, external light is focused by the lens onto the CMOS sensing module to generate an image light intensity electrical signal, which serves as the raw input to the neural network. In the shutter-closed mode, external light is blocked, creating a darkroom environment. An integrated LED emits a weak light of controllable intensity to activate the quantum random process to generate random numbers.
4. The method for training an integrated spiking neural network within a sensor according to claim 1, characterized in that, The encoding of the image light intensity electrical signal includes the following sub-steps: If the input is a grayscale image, the grayscale value of each pixel is linearly mapped to a pulse sequence of a specific frequency; If the input is an RGB color image, the pixel values of the R, G, and B channels are independently and linearly mapped to pulse sequences of corresponding frequencies, and the three-channel signals are encoded into three independent pulse sequences.
5. The method for training an integrated spiking neural network within a sensor according to claim 1, characterized in that, The formula for multiplying and accumulating the stored weight and the corresponding presynaptic pulse is as follows: ； in, Let be the membrane potential at time t. For leakage voltage, Let be the weight value of the i-th synapse at time t. This represents the presynaptic pulse released by the i-th presynaptic neuron, with a value of either 0 or 1. This is the reset voltage. is the threshold voltage, and N is the number of presynaptic neurons; Each neuron has a dynamic threshold voltage. When the conditions are met At this time, the neuron fires a postsynaptic pulse, and its membrane potential is then reset.
6. The method for training an integrated spiking neural network within a sensor according to claim 2, characterized in that, The random number generated by the CMOS sensing module includes the following sub-steps: When a postsynaptic pulse is generated, the CMOS sensing module switches from shutter open mode to shutter closed mode, activating the integrated LED to emit a weak light of a preset intensity; Under dim light conditions in a dark room, the random quantum events of photon arrival are superimposed with the thermal noise of the sensor to generate inherently unpredictable and discrete random charge signals on the CMOS pixel array. The CMOS sensor module in shutter-off mode is periodically sampled and quantized to obtain a random digital sequence.
7. The method for training an integrated spiking neural network within a sensor according to claim 2, characterized in that, The superposition of random quantum events arriving by photons with sensor thermal noise includes the following sub-steps: The photosensitive pixel array of a CMOS sensor is exposed to the weak light field generated by an LED. The particle characteristics of light dominate the photoelectric conversion process. The time, position, and ability of a single photon to reach a specific pixel point to generate a photogenerated carrier result in a completely random quantum event. Meanwhile, the thermally excited charge carriers generated by the thermal effect of the CMOS sensor itself become non-negligible background noise. The randomness of photon arrival and the randomness of thermal noise are independently superimposed, together generating an inherently unpredictable and discrete random charge signal on each photosensitive pixel.
8. The method for training an integrated spiking neural network within a sensor according to claim 1, characterized in that, The triggering of the learning process based on the stochastic STDP mechanism, which updates the synaptic weights and threshold voltages of neurons, includes the following sub-steps: Perform long-term enhanced LTP updates, and when a postsynaptic pulse occurs, probabilistically enhance synapses that have previously experienced a presynaptic pulse within a specific time window; For a synapse with a time interval of i, the currently obtained random number is compared with the corresponding preset probability parameter Pi. If the random number is less than Pi, the synapse weight is increased by a fixed value. Perform a long-term suppressed LTD update, and after the long-term suppressed LTD update, apply a global small decay to all synaptic weights; Increase the threshold voltage of the neuron that has just fired a pulse by increasing the threshold voltage by an amount that increases the threshold voltage.
9. The method for training an integrated spiking neural network within a sensor according to claim 7, characterized in that, The formula for updating the synaptic weights and threshold voltages of the neurons is as follows: ； in, , and These are constants, representing the increase in LTP, the decrease in LTD, and the increase in threshold voltage, respectively. The postsynaptic pulse at time t, with a value of 0 or 1; Let be the threshold voltage value at time t. For the weights of the spiking neural network, This is an indicator variable based on the result of random number comparison, taking the value 0 or 1. i is the time interval between the presynaptic pulse and the postsynaptic pulse; n is the number of historically stored presynaptic pulse cycles.
10. A system for training a spiking neural network integrated within a sensor, characterized in that, This includes a spiking neural network module and a spiking neural network training module; The spiking neural network module is constructed based on a CMOS sensing module, a pulse conversion circuit, a weight storage module, a neuron module, and a weight update module. The spiking neural network training module performs unsupervised training of the spiking neural network based on the stochastic STDP mechanism, and classifies the neurons after training is completed.