Lightweight gesture recognition method and system fusing event stream
By using a lightweight gesture recognition method that integrates event streams and RGB data, the problem of low recognition accuracy of traditional RGB cameras in poor lighting and fast gesture movement scenarios is solved, enabling high-speed real-time gesture recognition for edge devices.
Patent Information
- Application Number
- CN202411141559.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-20
- Publication Date
- 2024-11-08
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
Existing gesture recognition methods based on traditional RGB cameras have low recognition accuracy in underexposed, overexposed, and fast gesture movement scenarios. Furthermore, deep neural networks have a large number of parameters and high computational complexity, making it difficult to achieve real-time high-speed computing on edge devices.
The system employs spatiotemporal synchronization alignment to acquire event stream data and RGB data. Through preprocessing, it generates event frames and RGB image three-channel data of uniform size. These are then input into the feature extraction module and merged into an advanced feature map. A lightweight fusion classification module is then used for gesture recognition.
Accurate gesture recognition was achieved in scenarios with poor lighting conditions and rapid gesture movement, and the problem of high-speed real-time computing on edge devices was solved by using a lightweight network.
Smart Images

Figure CN118918640A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of gesture recognition technology, and in particular to a lightweight gesture recognition method and system integrating event streams. Background Art
[0002] Gesture recognition technology is an important tool for human-computer interaction and can be used in fields such as virtual reality and augmented reality. Many deep neural networks use traditional optical RGB images as input and show good gesture recognition effects. However, for scenes such as underexposure, overexposure, and rapid gesture movement, the existing gesture detection system based on traditional optical RGB images has limitations and cannot accurately recognize gestures. The dynamic range of the sensor of traditional RGB cameras is limited, and it is difficult to capture very dark and very bright details at the same time. In the case of underexposure, dark details will be lost; in the case of overexposure, bright details will be overexposed, resulting in image distortion. Traditional RGB cameras capture images at a fixed frame rate. For scenes with rapid gesture movement, they may move a large distance between adjacent frames, resulting in motion blur or missing details. As a result, the existing gesture recognition methods that only use RGB cameras have low recognition accuracy in scenes such as underexposure, overexposure, and rapid gesture movement. At present, most deep neural networks for gesture recognition have large parameters and complex networks, and require high computing power for computing devices. For example, the popular small gesture recognition network using yolov4-tiny still contains 38 layers of convolution and the number of parameters reaches 6M, which cannot realize real-time high-speed operation of gesture recognition on edge devices. Summary of the invention
[0003] In order to solve the technical problems existing in the background technology, the present invention proposes a lightweight gesture recognition method and system integrating event streams.
[0004] In a first aspect, the present invention proposes a lightweight gesture recognition method integrating event streams, comprising the following steps:
[0005] S1, obtaining event stream data and RGB data of gesture actions collected by time-space synchronous alignment;
[0006] S2, pre-processing the event stream data and RGB data of the gesture action respectively to obtain corresponding event frames and RGB image three-channel data of uniform size;
[0007] S3, inputting the event frame of uniform size and the three-channel data of the RGB image into a pre-trained feature extraction module respectively, to obtain a high-level feature map of the event and a high-level feature map of RGB;
[0008] S4. Input the high-level feature map of the event and the high-level feature map of RGB into the pre-trained fusion classification module to obtain the final classification result of the gesture action.
[0009] Preferably, the preprocessing specifically includes:
[0010] The event stream data of the gesture action are processed in parallel, the event stream data are equally divided into multiple groups of event points according to the timestamp within the time threshold of the event stream data, each group of event points is superimposed according to the coordinates, the background is set to a solid color, and the superimposed event points are set to different brightness according to the polarity to obtain an event frame;
[0011] Bilinear interpolation is used to resize the RGB data image to obtain three-channel RGB image data.
[0012] Preferably, the feature extraction module includes an RGB feature extraction network and an event feature extraction network; step S3 specifically includes:
[0013] Input the event frame into the event feature extraction network to obtain a high-level feature map of the event;
[0014] The three-channel data of the RGB image is input into the RGB feature extraction network to obtain the high-level feature map of RGB.
[0015] Preferably, the fusion classification module includes a fusion layer, a fully connected layer, and a Softmax layer; step S4 specifically includes:
[0016] The high-level feature map of the event and the high-level feature map of RGB are input into the fusion layer for merging and splicing to obtain a feature map containing event and RGB information;
[0017] The feature map containing event and RGB information is input into the fully connected layer to obtain the category vector z of the gesture action;
[0018] Input the category vector z of the gesture action into the Softmax layer to obtain the probability of each category of the gesture action;
[0019] The category with the highest probability value among the gesture action categories is taken as the prediction result of the final gesture recognition.
[0020] Preferably, the Softmax function of the Softmax layer is specifically:
[0021]
[0022] Among them, z i is each element of the fully connected layer output vector z; k is the dimension of the fully connected layer output vector z, and e is the natural index.
[0023] Preferably, the training process of the fusion classification module is as follows:
[0024] Set N gesture action categories, and train the high-level feature maps of multiple groups of events and the high-level feature maps of RGB obtained after feature extraction processing to establish a connection network between gesture actions and N gesture action categories;
[0025] The gesture actions are classified and recognized according to the connection network.
[0026] Preferably, the N gesture action categories are specifically 26 gesture action categories.
[0027] In a second aspect, the present invention proposes a lightweight gesture recognition system integrating event streams, including:
[0028] A data acquisition module is used to acquire event stream data and RGB data of gesture actions collected by time-space synchronous alignment;
[0029] A data preprocessing module is used to preprocess the event stream data and RGB data of the gesture action respectively to obtain the corresponding event frame and RGB image three-channel data of uniform size;
[0030] A data extraction module is used to input the event frame of uniform size and the three-channel data of the RGB image into a pre-trained feature extraction module to obtain a high-level feature map of the event and a high-level feature map of RGB;
[0031] The gesture action classification module is used to input the high-level feature map of the event and the high-level feature map of RGB into the pre-trained fusion classification module to obtain the final classification result of the gesture action.
[0032] In a third aspect, an embodiment of the present invention provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of the method provided in the first aspect are implemented.
[0033] In a fourth aspect, an embodiment of the present invention provides a computer device, characterized in that the device includes a memory and a processor, wherein a computer program is stored in the memory, and when the processor runs the computer program stored in the memory, the processor executes the steps of the method provided in the first aspect.
[0034] In the present invention, the proposed lightweight gesture recognition method and system for fusion event streams obtains event stream data and RGB data of gesture actions collected by time-space synchronization alignment; pre-processes the event stream data and RGB data of gesture actions respectively to obtain corresponding event frames of uniform size and RGB image three-channel data; inputs the event frames of uniform size and RGB image three-channel data into a pre-trained feature extraction module respectively to obtain high-level feature maps of events and high-level feature maps of RGB; inputs the high-level feature maps of events and high-level feature maps of RGB into a pre-trained fusion classification module to obtain the final classification result of gesture actions. By fusing event streams, the problem that the current traditional RGB cameras and deep neural networks cannot accurately recognize gestures in scenes with poor lighting conditions and fast gesture movements is solved; by lightweight deep neural networks, the problem that the current deep neural networks for gesture recognition have large parameters, high computational complexity, and difficulty in high-speed real-time computing for edge devices is solved. BRIEF DESCRIPTION OF THE DRAWINGS
[0035] Figure 1 A deep neural network structure diagram of a lightweight gesture recognition method integrating event streams proposed by the present invention;
[0036] Figure 2 A schematic diagram of the hardware framework structure of a lightweight gesture recognition method integrating event streams proposed by the present invention;
[0037] Figure 3 A schematic diagram of a recursive array of a two-layer hierarchical structure deployed in an embodiment of a lightweight gesture recognition method integrating event streams proposed by the present invention;
[0038] Figure 4 A schematic diagram of peripheral devices deployed in an embodiment of a lightweight gesture recognition method integrating event streams proposed by the present invention;
[0039] Figure 5 A schematic diagram of the work flow structure of a lightweight gesture recognition method integrating event streams proposed by the present invention;
[0040] Figure 6 This is a schematic diagram of the system architecture of a lightweight gesture recognition system integrating event streams proposed by the present invention. DETAILED DESCRIPTION
[0041] Reference Figure 1-5 The present invention proposes a lightweight gesture recognition method integrating event streams, comprising the following steps:
[0042] S1. Acquire event stream data and RGB data of gesture actions collected by time-space synchronous alignment.
[0043] In this embodiment, an event camera and a traditional optical RGB camera are used to synchronously shoot gestures, and the RGB camera and the event camera data space are aligned through a beam splitter; the RGB camera and the event camera are triggered by an external synchronization signal to shoot at the same time to ensure data time alignment. The event camera sets a time threshold for shooting according to the excitation event point, and obtains event stream data within the time threshold. The event stream data includes the X coordinate, Y coordinate, polarity and timestamp of each event point.
[0044] S2. Preprocess the event stream data and RGB data of the gesture action respectively to obtain corresponding event frames and RGB image three-channel data of uniform size.
[0045] In this embodiment, the preprocessing specifically includes: parallel processing of event stream data of gesture actions, dividing the event stream data into multiple groups of event points according to timestamps within the time threshold of the event stream data, superimposing each group of event points according to coordinates, setting the background to a solid color, and setting the superimposed event points to different brightness according to polarity to obtain event frames; and using bilinear interpolation to resize the RGB data to obtain three-channel RGB image data.
[0046] In this embodiment, since the event camera is stimulated by events to asynchronously shoot and output data, when the serial CPU pre-processes the event stream data, it can only execute sequentially in the face of a large amount of short-term event data, which is slow and consumes high power. Therefore, a hardware circuit is used to implement frame compression operation, and the event stream data can be processed in parallel. The event stream data is divided into multiple groups according to timestamps within the time threshold, and each group of event points is superimposed according to coordinates, the background is set to a solid color, and the superimposed event points are set to different brightness according to polarity to obtain event frames, thereby realizing high-speed data processing.
[0047] Specifically, the event data is downsampled and framed by the hardware circuit. The event data format is [X, Y, T, P], where X and Y are the two-dimensional coordinates of the event point, T is the timestamp of the event point, and P is the polarity of the event point. The internal event data interface adopts the Address Event Representation (AER) bus communication protocol. Every time the unit on the transmitter device generates an event, it will be broadcast on the digital bus. The decoder on the receiver decodes the address and directs the signal to the appropriate location. Since the AER bus is an asynchronous communication protocol, the digital bus is controlled by the handshake line (confirmation and request signals). The data transmission uses the AER bus. According to the image resolution of the sensor, the same number of registers are configured to receive the event excitation data of each position in parallel. Through the internal timer, the event points of all positions are accumulated as grayscale values within the fixed event interval, and the grayscale image of the event frame is generated by downsampling, which is stored in the system memory through the system bus. The RGB data is resized by the hardware circuit, and the image size is adjusted to 256*256 size and stored in the system memory. First, the RGB image is stored in the external memory DDR3, and bilinear interpolation is used to resize the image. The scaling ratio is first calculated by the internal logic circuit and stored in the internal register. Then, the 8*8 logic operation units output the 8*8 pixel interpolation of the image every clock cycle and store it in the register for output.
[0048] S3. Input the event frame of uniform size and the three-channel data of the RGB image into the pre-trained feature extraction module respectively to obtain the high-level feature map of the event and the high-level feature map of RGB.
[0049] In this embodiment, the feature extraction module includes an RGB feature extraction network and an event feature extraction network; step S3 specifically includes:
[0050] Input the event frame into the event feature extraction network to obtain a high-level feature map of the event;
[0051] The three-channel data of the RGB image is input into the RGB feature extraction network to obtain the high-level feature map of RGB.
[0052] In this embodiment, the three-channel data of the RGB image is used as the input of the RGB feature extraction network, and the single channel of the event frame image is used as the input of the event feature extraction network. The feature extraction networks of the event and RGB paths both contain five convolutional layers. In order to improve the utilization rate of the edge device computing unit and improve the operation speed, the size of the feature map and the number of output channels are both powers of 2. The convolution kernel size of each convolutional layer is 3×3, and the output channels are 32, 32, 32, 32, and 64 respectively. The output feature map of each convolutional layer is sent to the maximum pooling layer through the activation function (ReLU), and the feature map is sent to the next convolutional layer after the maximum pooling operation. The output of the last convolutional layer is a high-level feature map containing detail information. The feature extraction networks of RGB images and event frame images differ only in the number of input channels of the first convolutional layer, and the rest of the structures are basically the same. The results of the fifth convolutional layer of the RGB image and the event frame are respectively output to a global average pooling layer (global average pooling layer). By calculating the average sum of the input feature map, the spatial information of the network input data can be obtained. The global average pooling layer operation is implemented through convolution, and a hardware accelerator can be used to increase the computing speed. This layer does not require parameters and can effectively avoid overfitting problems.
[0053] S4. Input the high-level feature map of the event and the high-level feature map of RGB into the pre-trained fusion classification module to obtain the final classification result of the gesture action.
[0054] In this embodiment, the fusion classification module includes a fusion layer, a fully connected layer, and a Softmax layer; step S4 specifically includes: inputting the high-level feature map of the event and the high-level feature map of RGB into the fusion layer for merging and splicing to obtain a feature map containing event and RGB information; inputting the feature map containing event and RGB information into the fully connected layer to obtain the category vector z of the gesture action; inputting the category vector z of the gesture action into the Softmax layer to obtain the probability of each category of the gesture action; and taking the category with the highest probability value in the gesture action category as the prediction result of the final gesture recognition.
[0055] In this embodiment, the fusion layer merges and concatenates the feature maps of the event path and the RGB path, and splices the output feature maps of the global average pooling layer of the two paths of the feature extraction module in the channel dimension to generate a feature map containing event and RGB information.
[0056] The fully connected layer receives the feature map containing event and RGB information output by the fusion layer and implements linear transformation through convolution. Each neuron in the fully connected layer is connected to all neurons in the previous layer, and its output is the number of categories of gesture actions.
[0057] In this embodiment, the Softmax function of the Softmax layer is specifically:
[0058]
[0059] Among them, z i is each element of the fully connected layer output vector z; k is the dimension of the fully connected layer output vector z, and e is the natural index.
[0060] In this embodiment, the Softmax layer converts the output of the network into a probability distribution, and selects the category with the highest probability among the preset 26 gesture categories as the prediction result of the final gesture recognition.
[0061] In this embodiment, the training process of the fusion classification module is as follows:
[0062] N gesture action categories are set, and high-level feature maps of multiple groups of events and high-level feature maps of RGB obtained after feature extraction are trained to establish a connection network between gesture actions and N gesture action categories; gesture actions are classified and recognized based on the connection network.
[0063] In this embodiment, the N gesture action categories are specifically 26 gesture action categories.
[0064] In this embodiment, the hardware circuit part is as follows Figure 2 As shown in the figure, it is implemented by deploying on FPGA, including processor, accelerator, memory and peripheral circuits. When the processor and accelerator adopt decoupled access / execution architecture, the instructions of "memory access" and "execution" occur simultaneously in different areas of the hardware. Therefore, three controllers are required: execution controller, load controller and storage controller. The execution controller is responsible for executing "execution" type ISA commands, such as matrix multiplication. It includes a recursive array for dot product and a transpose. The load controller is responsible for all instructions to move data from the main memory to the dedicated register or accumulator of the accelerator. The dedicated SRAM inside the accelerator is divided into a register and an accumulator. The register stores the data input by the recursive array, and the accumulator stores the partial sum and final result between the recursive array blocks during the calculation process. The width of each row of the register and accumulator SRAM is dimension × "element", where the dimension is the number of recursive array processing units, and each element represents a scalar value of an accelerator operation.
[0065] like Figure 2 and Figure 3As shown, the accelerator matrix multiplication calculation module is mainly composed of a recursive array, a transposer, and a delay register. The calculation module receives three matrices (A, B, and D) one row at a time in each cycle, and outputs the result C = A*B+D one row at a time in each cycle. In the weight fixed mode, the B value is "preloaded" into the recursive array, and the A and D values are fed through. In the output fixed mode, the D value is "preloaded" into the recursive array, and the A and B values are fed through. The input (A, B, and D) must be delayed using a shift register so that each input from one matrix reaches the correct processing unit at the right time to be multiplied and accumulated with the correct input from another matrix. The recursive array consists of two levels, the bottom layer is a spatially distributed processing unit, each processing unit can perform dot product and accumulation operations. The upper layer is a block, which consists of a rectangular array of several processing units. The processing units in each block are directly connected to each other, and the blocks are connected to each other through registers. The recursive array has two hierarchical structures, and the processing units in the configuration block can realize full pipeline calculation or parallel vector calculation of different gesture recognition algorithms. The transposer can transpose the matrix before inputting the recursive array to ensure that the input matrix is the same as the output result row data stored in the temporary register.
[0066] Peripheral device controller hardware circuit such as Figure 4 As shown, the accelerator can drive and control peripherals, including DDR, SD card, serial port, and the corresponding system bus and clock divider. The system bus adopts the AXI bus protocol, and each module is connected to the accelerator through the bus matrix. DDR is the system memory storage. The DDR module contains a DDR controller. The DDR controller acts as a bus slave, receives and responds to the host's instructions on the bus, and reads and writes the DDR storage. The SD card is the system hard disk. The SD card module controller is divided into a bus master and a slave. The host configures the DMA register to move the data in the SD and the data in the memory; the slave receives and responds to the accelerator's instructions and reads and writes the data in the SD card. The serial port is used as the system debugging control interface. The serial port module controller is a bus slave, outputs print data, triggers an interrupt when receiving user input data, and provides user data to the system accelerator.
[0067] Reference Figure 1-6 The present invention proposes a lightweight gesture recognition method integrating event streams, comprising:
[0068] A data acquisition module is used to acquire event stream data and RGB data of gesture actions collected by time-space synchronous alignment;
[0069] A data preprocessing module is used to preprocess the event stream data and RGB data of the gesture action respectively to obtain the corresponding event frame and RGB image three-channel data of uniform size;
[0070] A data extraction module is used to input the event frame of uniform size and the three-channel data of the RGB image into a pre-trained feature extraction module to obtain a high-level feature map of the event and a high-level feature map of RGB;
[0071] The gesture action classification module is used to input the high-level feature map of the event and the high-level feature map of RGB into the pre-trained fusion classification module to obtain the final classification result of the gesture action.
[0072] In the specific working process of the lightweight gesture recognition method and system for fusion event stream of this embodiment, the event stream data and RGB data of the gesture action collected by time-space synchronous alignment are preprocessed respectively to obtain corresponding event frames and RGB image three-channel data of uniform size; then the event frames and RGB image three-channel data of uniform size are input into a pre-trained feature extraction module respectively to obtain a high-level feature map of the event and a high-level feature map of RGB; then the high-level feature map of the event and the high-level feature map of RGB are input into a pre-trained fusion classification module to obtain the final classification result of the gesture action.
[0073] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and inventive concept of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.
Claims
1. A lightweight gesture recognition method integrating event streams, characterized in that: The following steps are involved: S1, obtaining event stream data and RGB data of gesture actions collected by time-space synchronous alignment; S2, pre-processing the event stream data and RGB data of the gesture action respectively to obtain corresponding event frames and RGB image three-channel data of uniform size; S3, inputting the event frame of uniform size and the three-channel data of the RGB image into a pre-trained feature extraction module respectively, to obtain a high-level feature map of the event and a high-level feature map of RGB; S4. Input the high-level feature map of the event and the high-level feature map of RGB into the pre-trained fusion classification module to obtain the final classification result of the gesture action.
2. The lightweight gesture recognition method of fusion event stream according to claim 1 is characterized in that: The pre-processing specifically includes: The event stream data of the gesture action are processed in parallel, the event stream data are equally divided into multiple groups of event points according to the timestamp within the time threshold of the event stream data, each group of event points is superimposed according to the coordinates, the background is set to a solid color, and the superimposed event points are set to different brightness according to the polarity to obtain an event frame; Bilinear interpolation is used to resize the RGB data image to obtain three-channel RGB image data.
3. The lightweight gesture recognition method of fusion event stream according to claim 1 is characterized in that: The feature extraction module includes an RGB feature extraction network and an event feature extraction network; step S3 specifically includes: Input the event frame into the event feature extraction network to obtain a high-level feature map of the event; The three-channel data of the RGB image is input into the RGB feature extraction network to obtain the high-level feature map of RGB.
4. The lightweight gesture recognition method of fusion event stream according to claim 1 is characterized in that: The fusion classification module includes a fusion layer, a fully connected layer, and a Softmax layer; step S4 specifically includes: The high-level feature map of the event and the high-level feature map of RGB are input into the fusion layer for merging and splicing to obtain a feature map containing event and RGB information; The feature map containing event and RGB information is input into the fully connected layer to obtain the category vector z of the gesture action; Input the category vector z of the gesture action into the Softmax layer to obtain the probability of each category of the gesture action; The category with the highest probability value among the gesture action categories is taken as the prediction result of the final gesture recognition.
5. The lightweight gesture recognition method of fusion event stream according to claim 4 is characterized in that: The Softmax function of the Softmax layer is specifically: Among them, z i is each element of the fully connected layer output vector z; k is the dimension of the fully connected layer output vector z, and e is the natural index.
6. The lightweight gesture recognition method of fusion event stream according to claim 1 is characterized in that: The training process of the fusion classification module is as follows: Set N gesture action categories, and train the high-level feature maps of multiple groups of events and the high-level feature maps of RGB obtained after feature extraction processing to establish a connection network between gesture actions and N gesture action categories; The gesture actions are classified and recognized according to the connection network.
7. The lightweight gesture recognition method of fusion event stream according to claim 6 is characterized in that: The N gesture action categories are specifically 26 gesture action categories.
8. A lightweight gesture recognition system integrating event streams, characterized in that: include: A data acquisition module is used to acquire event stream data and RGB data of gesture actions collected by time-space synchronous alignment; A data preprocessing module is used to preprocess the event stream data and RGB data of the gesture action respectively to obtain the corresponding event frame and RGB image three-channel data of uniform size; A data extraction module is used to input the event frame of uniform size and the three-channel data of the RGB image into a pre-trained feature extraction module to obtain a high-level feature map of the event and a high-level feature map of RGB; The gesture action classification module is used to input the high-level feature map of the event and the high-level feature map of RGB into the pre-trained fusion classification module to obtain the final classification result of the gesture action.
9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the lightweight gesture recognition method for integrating event streams according to any one of claims 1 to 7 is executed.
10. A computer device, characterized in that The device includes a memory and a processor, wherein a computer program is stored in the memory. When the processor runs the computer program stored in the memory, the processor executes a lightweight gesture recognition method for integrating event streams as described in any one of claims 1 to 7.
Citation Information
Patent Citations
Multi-modal dynamic gesture recognition method based on lightweight 3D residual network and TCN
CN112507898A
Unmanned aerial vehicle target detection method based on event camera
CN113762409A
Head posture detection method and device and computer equipment
CN114332911A
Fuzzy license plate recognition method based on event camera
CN114998879A
Gesture recognition method and electronic equipment
CN115661941A
Cited By
Potato leaf disease area positioning system based on image recognition
CN121121498A
A potato leaf disease area localization system based on image recognition
CN121121498B
