Lightweight gesture recognition system, recognition method and gesture recognition system training system

Abstract

This invention provides a lightweight gesture recognition system, recognition method, and gesture recognition system training system. The system is located on the terminal side and includes: a video acquisition module for acquiring a video stream; an event-driven processing module for calculating the brightness difference between pixels in adjacent frames within the video stream, comparing the brightness difference with a dynamic trigger threshold, and selecting, based on the comparison result, to output data packets that do not contain zero events to the NPU or not generate the data packets; the NPU is equipped with a spiking neural network; after updating the state of the spiking neurons that receive the data packets, the NPU outputs the gesture recognition result through the neural network. This system reduces computational waste in invalid background areas, and while maintaining gesture recognition accuracy and response speed, reduces the overall power consumption of the system.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and human-computer interaction, and in particular to a lightweight gesture recognition system, recognition method, and gesture recognition system training system. Background Technology

[0002] With the development of full-screen smartphones, users' demand for air gesture interaction is increasing, often used in scenarios such as cooking and screen-off operation. Existing solutions mostly rely on the front-facing camera to continuously capture images and use CNN models to perform full-frame computation to achieve gesture recognition. This results in the NPU / DSP running at high load for a long time, causing serious power consumption and heat generation, which affects battery life and system smoothness.

[0003] Furthermore, while existing technologies employ optimization methods such as frame difference and low-resolution sampling, they cannot fundamentally reduce computational waste in invalid regions. Lightweight models on mobile devices have weak temporal modeling capabilities and insufficient accuracy in recognizing complex gestures; while high-precision temporal models require significant computational power, making them difficult to run in real-time on low-power hardware.

[0004] Meanwhile, the poor coordination between gesture recognition algorithms and device system hardware and software makes it impossible to dynamically adjust the calculation strategy based on battery power and device temperature. This makes it difficult to balance the requirements of all-weather perception, recognition accuracy, low power consumption, and temperature control, thus hindering the widespread application of air gesture interaction technology on mobile terminals. Summary of the Invention

[0005] This invention provides a lightweight gesture recognition system, recognition method, and gesture recognition system training system, which reduces the overall power consumption of the system while maintaining gesture recognition accuracy and response speed.

[0006] This invention provides a lightweight gesture recognition system, the system being installed on a terminal side, comprising: The video capture module is used to capture video streams; The event-driven processing module is used to calculate the brightness difference between two adjacent frames of pixels in the video stream, compare the brightness difference with a dynamic trigger threshold, and select to output a data packet that does not contain zero events to the NPU or not generate the data packet based on the comparison result. The NPU contains a spiking neural network; after updating the state of the spiking neurons that receive the data packet, the NPU outputs the gesture recognition result through the spiking neural network.

[0007] Furthermore, when the spiking neural network does not receive the data packet, the NPU enters a sleep state.

[0008] Furthermore, the NPU also includes: A memory management unit is used to store the membrane potential state of the spiking neurons; The state update unit is used to read the current membrane potential value stored by the spiking neuron in the memory management unit, add the synaptic weight value corresponding to the data packet to the current membrane potential value, and when the current membrane potential value exceeds the firing threshold of the spiking neuron, control the spiking neuron to output a pulse and reset the membrane potential value, and write the updated membrane potential value back to the corresponding storage address of the memory management unit.

[0009] Furthermore, the NPU also includes: On-chip storage unit for storing the weight parameters of the spiking neural network and the membrane potential state of the spiking neurons; The dynamic sparse computing unit is used to parse the data packet, extract the spatial coordinate information in the data packet, locate the weight parameter region corresponding to the spatial coordinate information in the on-chip storage unit, execute the dynamic sparse activation instruction, and perform addressing and operation on the weight parameter region; skip the calculation operation of the spiking neuron that has not received event input.

[0010] Furthermore, the system also includes: The dynamic resource management module is used to dynamically adjust the dynamic trigger threshold according to the temperature and power status of the terminal side, so as to control the number of data packets input to the spiking neural network and the terminal side's response sensitivity to gestures.

[0011] Furthermore, the dynamic resource management module includes: A temperature-threshold mapping table storage unit is used to store a temperature-threshold lookup table; wherein, the temperature-threshold lookup table is used to reflect the mapping relationship between different temperature levels and different trigger thresholds; The temperature level determination unit searches for the current temperature level of the terminal side based on the temperature-threshold lookup table; The threshold adjustment execution unit searches for the corresponding trigger threshold in the temperature-threshold lookup table based on the current temperature level of the terminal side, and sets the trigger threshold as a dynamic trigger threshold.

[0012] Furthermore, the system also includes: The SOC temperature sensor is used to transmit the temperature data from the terminal side to the dynamic resource management module.

[0013] On the other hand, the present invention also discloses a lightweight gesture recognition method, the method comprising: Capture video streams; Calculate the brightness difference between two adjacent frames of pixels in the video stream, compare the brightness difference with a dynamic trigger threshold, and based on the comparison result, select to output a data packet that does not contain zero events to the NPU or not generate the data packet. After updating the state of the sparse event characteristics of the data packet, the spiking neuron that received the event outputs the gesture recognition result through the spiking neuron.

[0014] On the other hand, the present invention also discloses a lightweight gesture recognition system training system, the system comprising: A hybrid architecture teacher network is used to extract and fuse temporal dependency features and spatial image features of gesture actions to obtain fused features represented in floating-point. A weighted binarization bridge network is used to convert the fused features represented by the floating point into discrete features, and input the discrete features into the student model of the spiking neural network; The spiking neural network student model is used to learn the expressive power of the discrete features through knowledge distillation, and to transfer the learned expressive power to the spiking neural network on the terminal side.

[0015] Furthermore, the hybrid architecture teacher network adopts a hybrid model of convolutional neural network and visual state space model.

[0016] Compared with the prior art, the present invention has at least the following technical effects: The lightweight gesture recognition system disclosed in this invention achieves ultra-low standby power consumption through the collaborative operation of an event-driven processing module and a spiking neural network. Specifically, when a user performs a gesture, the pixel brightness of the moving area changes significantly. When the pixel brightness difference exceeds a dynamic trigger threshold, the event-driven processing module generates a data packet and outputs it to the NPU. The spiking neural network only updates the state and propagates pulses for neurons that receive the event, making the actual computing power consumption of the NPU proportional to the sparsity of the input event. This event-driven asynchronous computing mode reduces computational waste in invalid background areas, lowering the overall system power consumption while maintaining gesture recognition accuracy and response speed. Attached Figure Description

[0017] Figure 1 This is a simplified schematic diagram of the lightweight gesture recognition system in Embodiment 1 of the present invention; Figure 2 This is a simplified schematic diagram of another lightweight gesture recognition system in Embodiment 1 of the present invention. Figure 3 This is a simplified flowchart of the dynamic resource management module in Embodiment 1 of the present invention; Figure 4 This is a simplified flowchart illustrating the lightweight gesture recognition method in Embodiment 2 of the present invention; Figure 5 This is a simplified schematic diagram of the modules of the lightweight gesture recognition system training system in Embodiment 3 of the present invention. Detailed Implementation

[0018] The following description, with reference to schematic diagrams, illustrates a lightweight gesture recognition system, recognition method, and gesture recognition system training system according to the present invention. Preferred embodiments of the invention are shown. It should be understood that those skilled in the art can modify the invention described herein while still achieving its advantageous effects. Therefore, the following description should be understood as being of general knowledge to those skilled in the art and is not intended to limit the invention.

[0019] The invention is described more specifically by way of example in the following paragraphs with reference to the accompanying drawings. The advantages and features of the invention will become clearer from the following description. It should be noted that the drawings are in a very simplified form and use non-precise proportions, and are only used to facilitate and clarify the illustration of the embodiments of the invention.

[0020] Example 1 Please refer to Figure 1 This embodiment discloses a lightweight gesture recognition system, which is located on the terminal side and includes: The video capture module is used to capture video streams; The event-driven processing module is used to calculate the brightness difference between two adjacent frames of pixels in the video stream. The brightness difference The data packet is compared with the dynamic trigger threshold, and based on the comparison result, the data packet that does not contain zero events is output to the NPU (Neural Processing Unit) or the data packet is not generated. The NPU contains a spiking neural network (SNN); after updating the state of the spiking neurons that receive the data packet, the NPU outputs the gesture recognition result through the spiking neural network.

[0021] In this embodiment, the lightweight gesture recognition system achieves ultra-low standby power consumption through the collaborative operation of the event-driven processing module and the spiking neural network. Specifically, when a user performs a gesture, the pixel brightness of the moving area changes significantly. When the pixel brightness difference exceeds the dynamic trigger threshold, the event-driven processing module generates a data packet and outputs it to the NPU. The spiking neural network only updates the state and propagates pulses for neurons that receive the event, making the actual computing power consumption of the NPU proportional to the sparsity of the input event. This event-driven asynchronous computing mode reduces the computational waste in invalid background areas, reducing the overall power consumption of the system while maintaining gesture recognition accuracy and response speed.

[0022] In this embodiment, after the NPU updates the state of the spiking neurons that receive the data packet, it outputs the gesture recognition result through the spiking neural network.

[0023] In this embodiment, the NPU (Neural Processing Unit) is a hardware accelerator on the mobile terminal side dedicated to accelerating neural network inference computation. The NPU is integrated into the SoC (System on Chip) on the mobile terminal side, possessing parallel computing capabilities and low power consumption, enabling it to efficiently execute sparse computation tasks of spiking neural networks.

[0024] In another specific example, the event-driven processing module is implemented using an ISP (Image Signal Processor) or a dedicated preprocessing unit on the mobile terminal side.

[0025] The ISP is responsible for acquiring raw video stream data from the camera, storing pixel data of the current and previous frames through its built-in frame buffer, and quickly calculating the brightness difference between adjacent frames using a hardware-level pixel difference calculation unit. The system uses a programmable threshold comparator to compare the calculated results with a dynamically triggered threshold in real time, thereby generating sparse event data packets. This hardware-level processing avoids the bandwidth overhead of transmitting complete video frames to the application processor, reducing the overall power consumption of the system.

[0026] In another specific example, a concrete implementation of comparing the brightness difference with a dynamic trigger threshold and selecting, based on the comparison result, to output a data packet that does not contain zero events to the NPU or not to generate the data packet is as follows: The event-driven processing module calculates the brightness difference of the pixel at coordinates (x, y) in the video stream between time t and time t-1. ; When ΔL(x,y)≤Cth, it is determined that there is no significant movement at the pixel position, the event corresponding to the pixel is marked as a zero event and discarded, and no data packet is generated; where Cth is the dynamic trigger threshold.

[0027] When ΔL(x,y)>Cth, it is determined that there is significant motion at the pixel position, and the data packet (x,y,t,p) containing spatial coordinates x,y, timestamp t and polarity identifier p is generated and the event data packet is output to the NPU; wherein, the polarity identifier p is used to indicate the direction of brightness change, p=+1 when brightness increases and p=-1 when brightness decreases.

[0028] In this embodiment, when the spiking neural network does not receive the data packet, the NPU enters a sleep state.

[0029] In this embodiment, the sleep state refers to the NPU shutting down the clock signal of its internal computing units using clock gating technology, ceasing state updates and pulse propagation calculations for spiking neurons, and only maintaining key state data such as neuron membrane potentials in on-chip storage units. This reduces the dynamic power consumption of the NPU to near the static leakage power consumption level. In this sleep state, because the background area in the video stream remains static, the pixel brightness difference ΔL(x,y) calculated by the event-driven processing module is generally less than the dynamic trigger threshold Cth, and no valid event data packets are generated.

[0030] In this embodiment, the NPU further includes: A memory management unit is used to store the membrane potential state of the spiking neurons.

[0031] The state update unit is used to read the current membrane potential value stored by the spiking neuron in the memory management unit, add the synaptic weight value corresponding to the data packet to the current membrane potential value, and when the current membrane potential value exceeds the firing threshold of the spiking neuron, control the spiking neuron to output a pulse and reset the membrane potential value, and write the updated membrane potential value back to the corresponding storage address of the memory management unit.

[0032] It is understandable that after the spiking neurons transmit the pulse signals layer by layer to the final layer, the final layer of the spiking neural network determines the gesture type based on the number of pulse signals.

[0033] In this embodiment, the two units described above enable efficient storage and dynamic updating of the membrane potential states of spiking neurons. The memory management unit employs memory pooling technology to centrally store the membrane potential data of multiple spiking neurons in the on-chip storage unit of the NPU, avoiding the latency and power consumption overhead caused by frequent access to off-chip memory. Furthermore, the state update unit enables asynchronous event-driven computation of the spiking neural network, allowing neurons to remain in a resting state when there is no event input, effectively reducing computational resource consumption.

[0034] Furthermore, the NPU also includes: An on-chip storage unit is used to store the weight parameters of the spiking neural network and the membrane potential state of the spiking neurons.

[0035] The dynamic sparse computing unit is used to parse the data packet, extract the spatial coordinate information in the data packet, locate the weight parameter region corresponding to the spatial coordinate information in the on-chip storage unit, execute the dynamic sparse activation instruction, and perform addressing and operation on the weight parameter region; skip the calculation operation of the spiking neuron that has not received event input.

[0036] In a specific example, the on-chip storage unit includes SRAM (Static Random Access Memory), on-chip cache, and register file, used to store the membrane potential state of spiking neurons, synaptic weight parameters, event data packets, and activation values ​​of each layer of the neural network.

[0037] In another specific example, the dynamic sparse computing unit includes a sparse matrix multiplication accelerator, a conditional execution controller, a zero-value detection circuit, and an address generation unit. The sparse matrix multiplication accelerator is responsible for performing multiplication and addition operations between non-zero weights and event inputs. The conditional execution controller determines whether to activate the computational pathway of the corresponding neuron based on the event data packet. The zero-value detection circuit monitors zero events in the input event stream and triggers a skip instruction. The address generation unit dynamically calculates the physical addresses of neurons and synaptic weights based on the spatial coordinates of sparse events.

[0038] In this embodiment, the two units described above enable dynamic sparse computation and zero-value skipping optimization during the spiking neural network inference process. Specifically, the on-chip storage unit pre-loads all weight parameters and neuron state data of the spiking neural network into the on-chip storage unit of the NPU, avoiding data transmission delays during inference. The dynamic sparse computation unit avoids performing invalid multiplication and addition operations. The actual computational load of the NPU is proportional to the sparsity of the input events, and the computational load is close to zero in static background scenes, reducing standby power consumption.

[0039] For further details, please refer to... Figure 2 In this embodiment, the system further includes a SOC temperature sensor, used to transmit the temperature data from the terminal side to the dynamic resource management module.

[0040] In this embodiment, the SoC temperature sensor collects real-time chip core temperature data on the mobile terminal side and transmits the temperature data to the dynamic resource management module via the system bus. The dynamic resource management module, based on the received temperature data and a temperature-threshold lookup table, controls the dynamic trigger threshold of the event-driven processing module, thereby adjusting the sparsity of the event stream input to the spiking neural network. This mechanism forms a closed-loop control loop for temperature sensing, which can proactively reduce the computational load when the chip temperature rises, avoiding chip overheating and performance lag caused by continuous high-load computation, and maintaining system smoothness during multi-task concurrency. The temperature data is a key input parameter for triggering the dynamic resource management strategy; its real-time performance and accuracy directly affect the system's thermal management effect and user experience.

[0041] Please continue to refer to this. Figure 2In this embodiment, the system further includes a dynamic resource management module. Before the NPU unit performs the corresponding operation, the dynamic resource management module is used to dynamically adjust the dynamic trigger threshold according to the temperature and power status of the terminal side, so as to control the number of data packets input to the spiking neural network and the response sensitivity of the terminal side to gesture actions.

[0042] In this embodiment, the dynamic resource management module includes: A temperature-threshold mapping table storage unit is used to store a temperature-threshold lookup table; wherein, the temperature-threshold lookup table is used to reflect the mapping relationship between different temperature levels and different trigger thresholds.

[0043] The temperature level determination unit searches for the current temperature level of the terminal side based on the temperature-threshold lookup table.

[0044] The threshold adjustment execution unit searches for the corresponding trigger threshold in the temperature-threshold lookup table based on the current temperature level of the terminal side, and sets the trigger threshold as a dynamic trigger threshold.

[0045] In this embodiment, the temperature-threshold lookup table is configured to increase the temperature level step by step, thereby reducing the system's event throughput.

[0046] In a specific example, the dynamic resource management module is the Thermal Management Service or power management daemon in the mobile terminal's operating system. This module subscribes to temperature sensor data and battery management system power information in real time through a system-level API interface, and dynamically adjusts the operating parameters of the gesture recognition algorithm according to preset policy rules.

[0047] Please refer to Figure 3 In a specific example, the dynamic resource management module dynamically adjusts the dynamic trigger threshold based on the temperature and battery status of the terminal to control the number of data packets input to the spiking neural network and the terminal's response sensitivity to gestures. The specific implementation includes the following steps: Step 1: Monitor terminal-side status parameters: The dynamic resource management module reads the current temperature data T_current from the SoC temperature sensor and the current battery percentage P_current from the battery management system in real time through the system interface.

[0048] Step 2: Determine if the temperature exceeds the limit threshold: Compare the current temperature data T_current with the preset temperature limit threshold T_limit; wherein, the temperature limit threshold T_limit is set according to the thermal design power and safe temperature range of the mobile terminal, and the typical value is between 42℃ and 45℃.

[0049] Step 3: Determine if the battery level is below the warning threshold: Compare the current battery percentage P_current with the preset battery warning threshold P_limit; wherein, the battery warning threshold P_limit is set according to the user's battery life requirements, with a typical value between 20% and 30%.

[0050] Step 4: Implement the threshold adjustment strategy: When at least one of the conditions T_current > T_limit or P_current ≤ P_limit is met, the terminal is determined to be in an overheated or low-power state, and a strategy to reduce processing accuracy is executed. Specifically, the higher trigger threshold Cth_high corresponding to the current temperature level is found in the temperature-threshold lookup table, and the dynamic trigger threshold of the event-driven processing module is adjusted from the current value Cth_current to Cth_high. This adjustment causes the event-driven processing module to generate event data packets only for large pixel changes, ignoring small changes in light and shadow and subtle gestures, thereby physically reducing the number of events input to the NPU, reducing the NPU's inference throughput, and actively reducing computational load and chip heat generation.

[0051] When T_current ≤ T_limit and P_current > P_limit, the terminal is determined to be in a normal state, and the current dynamic trigger threshold Cth_current is maintained at a low value (e.g., Cth=10) to ensure the system's high sensitivity in recognizing micro-finger movements (such as pinching and tapping). If no gesture input is detected for a long time and the battery is sufficient, the trigger threshold can be further reduced to improve response sensitivity.

[0052] In this embodiment, the terminal device can be a smartphone, tablet, smartwatch, augmented reality (AR) glasses, virtual reality (VR) headset, smart vehicle device, smart home control terminal, or other mobile terminal device equipped with a camera, NPU, and touch display. All of the above terminal devices can employ the lightweight gesture recognition system disclosed in this application, which significantly reduces system power consumption and extends battery life while ensuring gesture recognition accuracy. Furthermore, a dynamic thermal management mechanism avoids device overheating and performance degradation caused by continuous visual computing, providing users with a smooth and stable air-based interactive experience.

[0053] Example 2 Based on the same inventive concept, this embodiment discloses a lightweight gesture recognition method. For details, please refer to... Figure 4 The method includes: S1. Capture video stream; S2. Calculate the brightness difference between two adjacent frames of pixels in the video stream, compare the brightness difference with the dynamic trigger threshold, and based on the comparison result, select to output a data packet that does not contain zero events to the NPU or not generate the data packet. S3. After updating the state of the spiking neuron that received the event by utilizing the sparse event characteristics of the data packet, the gesture recognition result is output through the spiking neuron.

[0054] Using the above method, when the video background is static, the number of event data packets generated is very small or zero, and the NPU is in a low-power state. When a gesture occurs, a dense event stream is generated in the motion area, driving the NPU to perform efficient pulse inference, realizing an event-driven asynchronous computing mode, which significantly reduces the power consumption of the gesture recognition system.

[0055] It is understood that the above-described lightweight gesture recognition method corresponds to the lightweight gesture recognition system disclosed in Embodiment 1, and the technical effects achieved are the same as those achieved by the lightweight gesture recognition system disclosed in Embodiment 1. The functional implementation of each module in the system can be applied to this method, and will not be elaborated here.

[0056] Example 3 Based on the same inventive concept, this embodiment discloses a lightweight gesture recognition system training system. For details, please refer to... Figure 5 The system includes: A hybrid architecture teacher network is used to extract and fuse temporal dependency features and spatial image features of gesture actions to obtain fused features in floating-point representation.

[0057] A weighted binarization bridge network is used to convert the fused features represented by the floating point into discrete features, and input the discrete features into the student model of the spiking neural network.

[0058] The spiking neural network student model is used to learn the expressive power of the discrete features through knowledge distillation, and to transfer the learned expressive power to the spiking neural network on the terminal side.

[0059] Once trained, the spiking neural network student model can be deployed in the NPU on the terminal side.

[0060] In this embodiment, the hybrid architecture teacher network utilizes large-scale parameters and complex structures to provide high-precision feature supervision, compensating for the difficulties in training spiking neural networks and the lack of temporal modeling capabilities. The weighted binarization bridge network, as an intermediate medium, simulates the discrete firing characteristics of spiking neurons, making the knowledge distillation process smoother and avoiding the performance loss of distilling directly from a floating-point network to a spiking network. The resulting spiking neural network student model has extremely low computational complexity and power consumption, while maintaining the ability to recognize complex temporal gestures, making it suitable for real-time operation on resource-constrained mobile terminals.

[0061] In one specific example, the hybrid architecture teacher network employs a hybrid model combining convolutional neural networks (CNNs) and a visual state-space model (Visual Mamba).

[0062] Specifically, the front end of the hybrid architecture teacher network uses a convolutional neural network to extract local spatial features of gesture images, and encodes the input video frames into a feature map sequence through multi-layer convolution and pooling operations; The backend of the hybrid architecture teacher network uses the Mamba-Transformer module for temporal modeling. The Mamba-Transformer module is designed based on the Visual Mamba model and uses the Selective State Space Mechanism to capture the long-distance spatiotemporal dependencies of gesture recognition results with linear complexity O(L) (where L is the sequence length), thus solving the gradient vanishing and gradient explosion problems existing in traditional recurrent neural networks (RNNs).

[0063] Furthermore, in this embodiment, the weighted binarization bridge network uses a sign function or a step function to binarize the fusion features represented by the floating point, in order to simulate the firing and non-firing states of spiking neurons.

[0064] In a specific example, the neuron activation values ​​of the weighted binarized bridge network are constrained by a binarization function. When the input activation value is greater than zero, the output is +1, indicating that the neuron fires a pulse; when the input activation value is less than or equal to zero, the output is -1 or 0, indicating that the neuron does not fire a pulse.

[0065] In this embodiment, the output features of the weighted binarized bridge network have discretization and sparsity characteristics, which are highly matched with the pulse coding method of the spiking neural network. This eliminates the modal differences from floating-point features to pulse features, enabling the spiking neural network student model to learn the knowledge representation of the teacher network more effectively and improve the effect of knowledge distillation.

[0066] Example 4 This embodiment uses the lightweight gesture recognition system and training system disclosed in Embodiments 1 and 3, taking a smart 5G mobile phone equipped with an NPU as an example, to illustrate the implementation process of "gesture-based page turning in a kitchen cooking scenario": 1. System Initialization Users enable the "Air Actions" function in the settings. The system loads the compressed spiking neural network student model into the on-chip storage unit of the NPU, initializing the dynamic trigger threshold to 10. The initial dynamic threshold can be set according to the camera signal characteristics, mobile phone hardware design, and light intensity in common usage scenarios.

[0067] 2. Standby monitoring phase In a specific use case, the phone is placed on a phone stand, and the user is viewing a cooking tutorial while the camera captures the kitchen background. Since the background remains static, the brightness difference calculated by the event generation unit is generally less than 10. No event is generated in this situation, the NPU is in a clock-gated state, and the overall power consumption of the vision subsystem is extremely low.

[0068] 3. Gesture Triggering Phase The user, with oily hands, waved to the right to indicate the "next video" option.

[0069] (1) Event generation: When the pixel changes drastically in the hand movement area and the brightness difference exceeds the dynamic trigger threshold, the event generation unit generates a data packet for the hand movement area that does not contain zero events.

[0070] (2) Event filtering: Hand model recognition is performed on the generated raw events to exclude interference data generated by non-target movements such as human movement and pet movement, and output valid hand movement event data packets.

[0071] (3) SNN Inference: The NPU is activated and invokes sparse operators to propagate pulses for data packets that do not contain zero events. The state update unit reads the current membrane potential value of the relevant spiking neurons from the memory management unit, adds the synaptic weight value corresponding to the event to the membrane potential value, and controls the neuron to output a pulse and reset the membrane potential when the membrane potential exceeds the firing threshold. The updated membrane potential is then written back to the memory management unit. After multiple layers of pulse propagation, the correct operation gesture is recognized within 50ms.

[0072] (4) Command execution: The mobile phone system receives the gesture recognition result, simulates the swipe operation, and realizes the video page turning function of the short video software.

[0073] 4. Dynamic Thermal Management Stage The phone is undergoing fast charging, and the ambient temperature is high. The SOC temperature sensor detected that the SoC core temperature reached 45°C.

[0074] (1) Feedback Adjustment: The temperature level judgment unit in the dynamic resource management module searches for the current temperature level of the terminal based on the temperature-threshold lookup table. The threshold adjustment execution unit adjusts the dynamic trigger threshold from 10 to 25 according to the search result.

[0075] (2) Load Reduction Effect: After the dynamic trigger threshold is increased, minor background leaf movements or light flickering (brightness difference ≈ 15) are no longer processed. Sparse event stream data is only generated and identified when the user makes a larger hand gesture (brightness difference > 25). By actively filtering weak signals, the processing load of the NPU is reduced by about 30%, helping to cool down the phone.

[0076] (3) Performance retention: The brightness difference generated by the user's conscious gesture recognition results far exceeds the new dynamic trigger threshold, so the gesture recognition function can still work normally and the recognition accuracy remains basically unchanged.

[0077] Through the above process, the present invention achieves low-power gesture recognition in a kitchen setting, and can dynamically adjust the dynamic trigger threshold according to the device temperature and power status, control the amount of sparse event stream data input to the spiking neural network and the response sensitivity of the terminal to the gesture recognition result, and achieve synergistic optimization of power consumption and temperature while ensuring functional availability.

[0078] Various modifications and variations are possible without departing from the spirit and scope of the invention. Thus, if these modifications and variations of the invention fall within the scope of the claims of the invention and their equivalents, the invention is also intended to include these modifications and variations.

Claims

1. A lightweight gesture recognition system, characterized in that, The system is located on the terminal side and includes: The video capture module is used to capture video streams; The event-driven processing module is used to calculate the brightness difference between two adjacent frames of pixels in the video stream, compare the brightness difference with a dynamic trigger threshold, and select to output a data packet that does not contain zero events to the NPU or not generate the data packet based on the comparison result. The NPU contains a spiking neural network; after updating the state of the spiking neurons that receive the data packet, the NPU outputs the gesture recognition result through the spiking neural network.

2. The lightweight gesture recognition system as described in claim 1, characterized in that, When the spiking neural network does not receive the data packet, the NPU enters a sleep state.

3. The lightweight gesture recognition system as described in claim 1, characterized in that, The NPU also includes: A memory management unit is used to store the membrane potential state of the spiking neurons; The state update unit is used to read the current membrane potential value stored by the spiking neuron in the memory management unit, add the synaptic weight value corresponding to the data packet to the current membrane potential value, and when the current membrane potential value exceeds the firing threshold of the spiking neuron, control the spiking neuron to output a pulse and reset the membrane potential value, and write the updated membrane potential value back to the corresponding storage address of the memory management unit.

4. The lightweight gesture recognition system as described in claim 3, characterized in that, The NPU also includes: On-chip storage unit for storing the weight parameters of the spiking neural network and the membrane potential state of the spiking neurons; The dynamic sparse computing unit is used to parse the data packet, extract the spatial coordinate information in the data packet, locate the weight parameter region corresponding to the spatial coordinate information in the on-chip storage unit, and execute the dynamic sparse activation instruction to address and operate the weight parameter region. And skip the computational operations of the spiking neurons that have not received event input.

5. The lightweight gesture recognition system as described in claim 1, characterized in that, The system also includes: The dynamic resource management module is used to dynamically adjust the dynamic trigger threshold according to the temperature and power status of the terminal side, so as to control the number of data packets input to the spiking neural network and the terminal side's response sensitivity to gestures.

6. The lightweight gesture recognition system as described in claim 5, characterized in that, The dynamic resource management module includes: A temperature-threshold mapping table storage unit is used to store a temperature-threshold lookup table; wherein, the temperature-threshold lookup table is used to reflect the mapping relationship between different temperature levels and different trigger thresholds; The temperature level determination unit searches for the current temperature level of the terminal side based on the temperature-threshold lookup table; The threshold adjustment execution unit searches for the corresponding trigger threshold in the temperature-threshold lookup table based on the current temperature level of the terminal side, and sets the trigger threshold as a dynamic trigger threshold.

7. The lightweight gesture recognition system as described in claim 5, characterized in that, The system also includes: The SOC temperature sensor is used to transmit the temperature data from the terminal side to the dynamic resource management module.

8. A lightweight gesture recognition method, characterized in that, The method includes: Capture video streams; Calculate the brightness difference between two adjacent frames of pixels in the video stream, compare the brightness difference with a dynamic trigger threshold, and based on the comparison result, select to output a data packet that does not contain zero events to the NPU or not generate the data packet. After updating the state of the sparse event characteristics of the data packet, the spiking neuron that received the event outputs the gesture recognition result through the spiking neuron.

9. A lightweight gesture recognition system training system, characterized in that, The system includes: A hybrid architecture teacher network is used to extract and fuse temporal dependency features and spatial image features of gesture actions to obtain fused features represented in floating-point. A weighted binarization bridge network is used to convert the fused features represented by the floating point into discrete features, and input the discrete features into the student model of the spiking neural network; The spiking neural network student model is used to learn the expressive power of the discrete features through knowledge distillation, and to transfer the learned expressive power to the spiking neural network on the terminal side.

10. The lightweight gesture recognition system training system as described in claim 9, characterized in that, The hybrid architecture teacher network adopts a hybrid model of convolutional neural network and visual state space model.

Images