A Method and System for Gesture Recognition Based on Super-Resolution Temporal Enhancement
By combining super-resolution temporal enhancement methods with one-dimensional convolution and bidirectional long short-term memory networks, the problem of low gesture recognition accuracy in low-cost magnetic sensor arrays is solved, achieving high-precision, low-cost, and convenient magnetic signal gesture recognition.
Patent Information
- Application Number
- CN202611271576.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-08-20
- Publication Date
- 2026-11-17
AI Technical Summary
Existing magnetic field-based gesture recognition technology, under the conditions of low-cost microcontrollers and multiplexed acquisition architecture, has a limited single-channel sampling rate, which leads to the loss of transient details of gestures, low recognition accuracy, and difficulty in adapting to dynamic gesture recognition.
A super-resolution temporal enhancement method is adopted, which amplifies the temporal signal through upsampling layers and combines one-dimensional convolution and bidirectional long short-term memory network to extract features, thereby improving the accuracy and robustness of gesture recognition.
While maintaining low hardware costs, it significantly improves the accuracy and robustness of gesture recognition, resists ambient light interference, protects user privacy, provides natural and convenient interaction, has low hardware costs, consumes few interface resources, ensures stable model training, and has strong noise resistance and robustness.
Smart Images

Figure CN123115940A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of human-computer interaction technology, specifically to a magnetic signal gesture recognition method and system based on super-resolution temporal enhancement. Background Technology
[0002] With the continuous development of smart wearable devices, smart homes, in-vehicle terminals, virtual reality, and the Internet of Things (IoT), human-computer interaction is gradually shifting from traditional button and touchscreen interactions to more natural and intuitive contactless interaction methods. Gesture recognition can directly capture the user's hand gestures and express their operational intentions. It features natural interaction, convenient operation, and wide applicability, and has become an important research direction in the field of human-computer interaction.
[0003] Among existing gesture recognition technologies, the most common approach is based on visual images. This type of approach typically uses a monocular camera, binocular camera, depth camera, or infrared imaging device to capture hand images, and then determines the gesture category through image segmentation, feature extraction, or deep neural networks. For example, reference [1] discloses a gesture recognition system and method based on Faster-RCNN, which captures real-time gesture images using a monocular camera and uses a Faster-RCNN model trained on the server side to complete the recognition. This type of approach is relatively intuitive in acquiring hand movements, but the recognition effect is easily affected by factors such as ambient lighting, complex backgrounds, occlusion, and shooting angle; at the same time, image acquisition may also involve user scene privacy, and image processing and deep model inference usually require high computing and storage resources.
[0004] Another type of existing solution uses inertial measurement units, flexible sensors, or data gloves to acquire hand movement information. For example, reference [2] discloses a piano playing gesture recognition method based on inertial gloves. The inertial gloves contain multiple inertial sensors set on the back of the hand and finger joints, and the gesture classification is achieved by collecting triaxial acceleration data and triaxial angular velocity data. This type of solution can get rid of the limitation of lighting conditions to a certain extent, but it requires users to wear gloves, wristbands or other sensing devices for a long time, which not only reduces the convenience and comfort of use, but also involves problems such as power supply, connection, synchronization and wearing position consistency of multiple sensing nodes. In addition, inertial sensors usually have zero bias, drift and cumulative error, which need to be calibrated and compensated during use.
[0005] Gesture recognition technology based on magnetic field detection identifies gestures by detecting changes in the magnetic field caused by hand movements, magnetic media, or the surrounding environment. It does not rely on ambient light and does not require the acquisition of user images, and has advantages in resisting light interference and protecting user privacy. For example, Reference [3] discloses a gesture recognition device based on a triaxial magnetic sensor group, which wears multiple triaxial magnetic sensor modules on the user's fingertips, sends magnetic field data to the recognition module through wireless communication, and uses a support vector machine decision tree to achieve gesture classification. Reference [4] further discloses that six geomagnetic sensors are installed on gloves, magnetic field data is collected through a multiplexed communication module, and the collected data is directly input into a one-dimensional convolutional neural network for gesture recognition; the sampling frequency in this reference is 10Hz, and it achieved an accuracy of nearly 97% in 26 static American Sign Language letter recognition experiments. It also points out that the recognition of dynamic and complex gestures still needs further research. It can be seen that magnetic field signals can be used for gesture recognition, but existing solutions still rely heavily on wearable magnetic sensing nodes or mainly classify the original low sampling rate sequences directly, which is difficult to fully adapt to the application requirements of low-cost, non-wearable magnetic sensing arrays.
[0006] To obtain the magnetic field changes caused by gestures in a large spatial range, multiple magnetic sensors can be used to form a sensing array. Reference [5] discloses a multi-channel parallel magnetic field signal acquisition and transmission system, which uses a magnetic sensor linear array, operational amplifier, high-speed electronic switch, high sampling rate analog-to-digital converter and gigabit Ethernet to realize the acquisition and transmission of multiple magnetic field signals. This scheme is beneficial to improve the acquisition capability of complex magnetic signals, but its overall hardware link is biased towards high-performance acquisition system. For application scenarios such as smart home, mobile terminal and embedded interactive device, the system usually needs to take into account cost, power consumption, size and number of interfaces, and it is difficult to simply adopt a large-scale high-speed parallel acquisition architecture.
[0007] In low-cost embedded magnetic sensor arrays, the number of analog-to-digital conversion channels and the conversion throughput of the microcontroller are limited. Therefore, multiplexing circuits are typically used to time-division poll multiple magnetic sensor channels. For a time-division polling architecture with a single analog-to-digital converter, if the magnetic sensor array contains multiple acquisition channels, and the analog switching stabilization and analog-to-digital conversion of a single channel require a fixed time, then each channel in the array can only be accessed periodically in a time-division manner. The equivalent sampling interval and equivalent sampling rate of a single channel are expressed as follows: , ; In the formula: This is the equivalent sampling interval for a single channel; This is the equivalent sampling rate for a single channel; The number of acquisition channels; The total time consumed for stabilizing the analog switch and the analog-to-digital conversion of a single channel.
[0008] As shown in the above equation, as the number of array channels increases, the number of sampling points that a single channel can obtain per unit time decreases accordingly. For dynamic gestures such as rapid waving, short-term turning, or sudden trajectory changes, their magnetic field signals typically contain transient details such as rising edges, falling edges, peaks, and short-term oscillations. When the single-channel sampling rate is insufficient, these high-frequency details are easily smoothed, omitted, or aliased, thereby reducing the temporal characteristic differences between different gestures.
[0009] Existing threshold judgment, template matching, and traditional machine learning methods typically rely on manually extracted amplitude, slope, or statistical features, which are insufficient to fully describe the local transient changes and long-term temporal dependencies of multi-channel magnetic field signals. While convolutional neural networks can extract local waveform features, and long short-term memory networks can perform temporal modeling, the effective sampling points and detailed information available to the model are still limited by the resolution of the input signal if the original low-sampling rate sequence is directly input into the model. Although simple interpolation or resampling can increase the sequence length, it is difficult to reliably recover the transient information missing during the low-sampling acquisition process if it is not coordinated with local feature extraction, forward and backward temporal modeling, and data distribution stabilization mechanisms.
[0010] Therefore, how to enhance the detailed expression capability of multi-channel magnetic field timing signals, make full use of local waveform features and forward and backward timing dependence features, and improve the accuracy and robustness of dynamic gesture recognition under the condition of using low-cost microcontrollers and multiplexed acquisition architecture is a technical problem that magnetic field-based gesture recognition technology needs to solve.
[0011] Cited references: Reference [1]: Gao Tianhan, Yang Zhenhao. A gesture recognition system and method based on Faster-RCNN: Chinese Invention Patent Application, CN113191421A. Application No.: CN202110463348.4; Application Date: 2021-04-25; Publication Date: 2021-07-30; Applicant: Northeastern University.
[0012] Reference [2]: Lü Pin, Ye Sufen, Lai Jizhou, Qin Saiyong, Shen Minghua. Piano playing gesture recognition method based on inertial gloves: Chinese invention patent application, CN110210547A. Application number: CN201910445355.4; Application date: 2019-05-27; Publication date: 2019-09-06; Applicants: Nanjing University of Aeronautics and Astronautics, Wuxi Wucheng Internet Technology Co., Ltd.
[0013] Reference [3]: Shi Bo, He Zhongzheng, Chen Xi, Han Ruoyu. A gesture recognition device based on a triaxial magnetic sensor group: Chinese invention patent application, CN116360585A. Application number: CN202310074559.8; Application date: 2023-02-07; Publication date: 2023-06-30; Applicant: Beijing Institute of Technology.
[0014] Reference [4]: Shi B, Chen X, He Z, Sun H, Han R. Research on Gesture Recognition System Using Multiple Sensors Based on Earth's Magnetic Field and 1D Convolution Neural Network (Research on Gesture Recognition System Based on Earth's Magnetic Field and 1D Convolution Neural Network) [J]. Applied Sciences, 2023, 13(9): 5544. Received: 2023-03-14; Revised: 2023-04-02; Accepted: 2023-04-26; Published: 2023-04-29; DOI: 10.3390 / app13095544.
[0015] Reference [5]: Han Weihui, Wu Shuang, Wang Jiangtao, Zhang Qing, Yuan Yujian, Xu Zhibin, Wang Zhenhui. A system and method for acquiring and transmitting multi-channel parallel magnetic field signals: Chinese Invention Patent Application, CN104459570A. Application No.: CN201410668372.1; Application Date: 2014-11-20; Publication Date: 2015-03-25; Applicants: East China Normal University, Shanghai Huayuan Chuangxin Software Co., Ltd. Summary of the Invention
[0016] To address the technical problems of limited single-channel sampling rate, loss of transient gesture details, and low recognition accuracy caused by multiplexing and time-division acquisition in low-cost magnetic sensor arrays, this invention provides a magnetic signal gesture recognition method and system based on super-resolution temporal enhancement. This method amplifies the temporal signal through an upsampling layer and combines one-dimensional convolution with a bidirectional long short-term memory network to collaboratively extract features, thereby significantly improving the accuracy and robustness of gesture recognition while maintaining low hardware costs.
[0017] The technical solution adopted in this invention is as follows: A magnetic signal gesture recognition method based on super-resolution temporal enhancement includes the following steps: S1: Use a microcontroller to control the multiplexing circuit to select each acquisition channel of the magnetic sensor array in a time-division manner, and acquire the original magnetic field timing signal caused by the user's gesture. S2: Perform zero-field calibration and normalization on the original magnetic field time sequence signal to obtain the standard input sequence; S3: Construct a gesture recognition neural network model, which includes an input layer, a super-resolution upsampling layer, a feature extraction convolutional layer, a temporal feature learning layer, and a classification output layer arranged sequentially. The super-resolution upsampling layer is used to amplify the time dimension of the input sequence by a preset factor. S4: The standard input sequence is input into the gesture recognition neural network model, and then the time dimension is magnified by the super-resolution upsampling layer, the local waveform features are extracted by the feature extraction convolutional layer, and the bidirectional temporal dependency features are extracted by the temporal feature learning layer. Finally, the probability of each gesture category is output by the classification output layer. S5: Determine the target gesture category based on the probability of each gesture category, and output the gesture command corresponding to the target gesture category.
[0018] S1 includes the following steps: S1.1: The microcontroller outputs an address signal to the multiplexing circuit through the GPIO pin to select the target acquisition channel; S1.2: After channel switching, a preset stabilization time is applied to wait for the output signal of the selected target acquisition channel to stabilize; S1.3: Start analog-to-digital conversion, read the analog voltage value output by the target acquisition channel, and obtain the sampled value of the target acquisition channel; S1.4: Execute S1.1~S1.3 in a loop according to the preset scanning order until all acquisition channels of the magnetic sensor array are traversed, and store the sampled values of each acquisition channel as the original magnetic field timing signal in time sequence.
[0019] In S1, The preset stabilization time is 1ms; The magnetic sensor array includes 25 linear Hall sensors, and the output terminal of each linear Hall sensor is provided with a voltage divider protection circuit and an overload protection circuit. The multiplexing circuit includes three cascaded eight-channel analog switch chips. The channel selection pins of each eight-channel analog switch chip are connected to the corresponding GPIO pins of the microcontroller, and the common output terminal of each eight-channel analog switch chip is connected to the corresponding analog-to-digital conversion channel of the microcontroller. Of the 25 linear Hall sensors, 24 are connected to the input channels of three eight-channel analog switch chips, and the remaining linear Hall sensor is directly connected to the independent analog-to-digital conversion channel of the microcontroller to form a reference acquisition channel.
[0020] S2 includes: 2.1: In an environment without an external magnetic field, continuously acquire data N times through each acquisition channel, and calculate the zero-point offset of each acquisition channel: ; In the formula: For the first Zero offset of each acquisition channel; The first under the condition of no external magnetic field The first acquisition channel Sub-original sampled values; This refers to the acquisition channel number; N represents the calibration sampling sequence number; N represents the number of calibration samplings.
[0021] 2.2: During system operation, the real-time sampled values of each acquisition channel are subtracted from the corresponding zero-point offset to obtain the difference signal. ; In the formula: For the first Real-time sampled values from each acquisition channel; For the first The difference signal of each acquisition channel.
[0022] 2.3: Convert the difference signal into sensor output voltage: ; In the formula: For the first The sensor output voltage corresponding to each acquisition channel; This is the reference voltage for analog-to-digital conversion; G represents the number of bits in the analog-to-digital conversion; G represents the preamplifier gain.
[0023] 2.4: Convert the sensor output voltage into magnetic field strength: ; In the formula: The sensitivity of the Hall sensor; For the first The magnetic field strength measured by each acquisition channel.
[0024] S2 further includes: S2.5: Clean the time series of magnetic field strength measured by each acquisition channel and remove invalid data; S2.6: Truncate or fill the time sequence of magnetic field intensity corresponding to each gesture sample after cleaning to a preset time length to obtain a sample sequence of equal length; S2.7: Perform min-max normalization on the equal-length sample sequences, mapping them to the [0,1] interval to obtain the standard input sequence: ; In the formula: X is the magnetic field strength value in the equal-length sample sequence; This represents the minimum magnetic field strength value in a sequence of samples of equal length. This represents the maximum magnetic field strength value in a sequence of samples of equal length. This is the standard input sequence obtained after normalization; S2.8: Perform one-hot encoding on the gesture labels corresponding to each gesture sample to generate model training labels.
[0025] S3 includes: S3.1: The super-resolution upsampling layer uses one-dimensional interpolation upsampling to amplify the time dimension of the standard input sequence input to the input layer by a preset factor. The relationship between the sequence lengths before and after amplification is as follows: ; In the formula: The length of the standard input sequence; The time length of the amplified sequence; The preset magnification factor is, and The value is 2; S3.2: After the super-resolution upsampling layer, a batch normalization layer is set to perform distribution correction on the amplified sequence, so as to stabilize the data distribution of the amplified sequence and accelerate the convergence of the gesture recognition neural network model.
[0026] The feature extraction convolutional layer comprises a first convolutional block, a max-pooling layer, and a second convolutional block connected in sequence. The extraction of local waveform features in S4 includes: 4.1: The first convolutional block performs sliding convolution on the magnified time dimension to capture long-distance local features with a wide receptive field. The first convolutional block contains 64 one-dimensional convolutional kernels with a length of 7. 4.2: The output sequence of the first convolutional block is downsampled by the max pooling layer, compressing the sequence length to half and restoring the sequence length to the order of magnitude before amplification; 4.3: The second convolutional block performs sliding convolution on the compressed sequence to capture short-term mutation features. The second convolutional block contains 128 one-dimensional convolutional kernels with a length of 3.
[0027] The temporal feature learning layer comprises a first bidirectional long short-term memory network layer, a random deactivation layer, and a second bidirectional long short-term memory network layer connected in sequence. The extraction of bidirectional temporal dependency features in S4 includes: 4.4: The first bidirectional long short-term memory network layer performs temporal modeling on the convolutional feature sequence output by the feature extraction convolutional layer simultaneously along the forward and backward directions, and outputs a temporal feature sequence that retains all time steps. The first bidirectional long short-term memory network layer has 128 unidirectional memory units, an input random inactivation rate of 40%, and a cyclic state inactivation rate of 10%. 4.5: The time-series feature sequences are randomly deactivated by the random deactivation layer with a deactivation rate of 50% to suppress overfitting; 4.6: The second bidirectional long short-term memory network layer compresses and refines the temporal feature sequence after random deactivation, and outputs the fused feature vector of the entire sequence. The number of unidirectional memory units in the second bidirectional long short-term memory network layer is 64.
[0028] The classification output layer includes a fully connected layer and a Softmax activation function, and S5 includes: 5.1: The fully connected layer maps the bidirectional temporal dependency features extracted by the temporal feature learning layer into scores for each category, and calculates the probability of each gesture category using the Softmax activation function. ; In the formula: For the input sequence to belong to the first The probability of each gesture category; For the first Scores for each gesture category; For gesture category number; Total number of gesture categories; The sequence number is the traversal index for the gesture category; It is a natural constant; 5.2: Select the gesture category corresponding to the highest probability as the target gesture category: ; In the formula: Target gesture category; 5.3: Map the target gesture category to the corresponding gesture command, output and execute the operation corresponding to the gesture command.
[0029] A magnetic signal gesture recognition system based on super-resolution temporal enhancement, comprising: Magnetic sensing module: includes a magnetic sensor array, used to sense changes in the magnetic field caused by user gestures; Signal acquisition module: includes a microcontroller and a multiplexing circuit. The input terminal of the multiplexing circuit is connected to the magnetic sensing module, and the control terminal of the multiplexing circuit is connected to the microcontroller. The multiplexing circuit is used to control the multiplexing circuit to select each acquisition channel of the magnetic sensor array in a time-division manner to acquire the original magnetic field timing signal. The data processing module is used to perform zero-field calibration and normalization on the original magnetic field time-series signal to obtain a standard input sequence; and to input the standard input sequence into a pre-constructed gesture recognition neural network model. The gesture recognition neural network model includes an input layer, a super-resolution upsampling layer, a feature extraction convolutional layer, a temporal feature learning layer, and a classification output layer arranged sequentially. The standard input sequence is passed through the super-resolution upsampling layer to amplify the time dimension, the feature extraction convolutional layer to extract local waveform features, and the temporal feature learning layer to extract bidirectional temporal dependency features. The classification output layer outputs the probability of each gesture category and determines the target gesture category based on the probability of each gesture category. Output module: Used to output gesture commands corresponding to the target gesture category.
[0030] The present invention provides a magnetic signal gesture recognition method and system based on super-resolution temporal enhancement, which has the following advantages: 1) Resistant to ambient light interference, protecting user privacy: This invention employs a magnetic sensor array to detect changes in the magnetic field caused by user gestures, achieving gesture recognition without relying on optical devices such as cameras or infrared imaging. The recognition process is unaffected by changes in ambient lighting, background complexity, occlusion, or shooting angle. Furthermore, since it does not collect user images, it fundamentally avoids the privacy risks associated with visual solutions. This approach solves the problems of optical gesture recognition schemes being greatly affected by lighting conditions and posing privacy risks in the prior art.
[0031] 2) No need for users to wear sensing devices, making interaction natural and convenient: This invention uses an array of magnetic sensors to detect changes in the magnetic field in space for identification. Users do not need to wear data gloves, inertial wristbands, or any other sensing devices, thus avoiding the burden of powering multiple sensor nodes, connecting cables, synchronizing data, and maintaining consistent wearing positions found in wearable solutions. It also avoids the inherent zero-bias drift and cumulative error compensation problems of inertial measurement units, improving the naturalness and convenience of human-computer interaction. This achievement solves the problem of inertial glove-based solutions requiring the wearing of equipment and being inconvenient to use.
[0032] 3) Low hardware cost and minimal interface resource consumption: This invention employs a microcontroller-controlled time-division multiplexing circuit acquisition architecture, requiring only a small number of analog-to-digital conversion channels to expand into a large-scale magnetic sensor array. It eliminates the need for independent high-speed acquisition links and high-sampling-rate analog-to-digital converters for each sensor, significantly reducing system hardware cost, power consumption, and size while maintaining spatial sensing capabilities. This makes the solution suitable for resource-constrained embedded terminal scenarios. This achievement solves the problems of high-performance magnetic signal acquisition in the prior art, which relies on high-speed switches, high-sampling-rate analog-to-digital converters, high-speed transmission links, and complex hardware links.
[0033] 4) Maintains high recognition accuracy even under low sampling rate hardware conditions: To address the core issues of limited single-channel sampling rate and loss or smoothing of rapid transient details in gestures caused by multiplexing and time-division acquisition, this invention uses a super-resolution upsampling layer to amplify the temporal dimension of low-sampling-rate time-series signals, artificially increasing the temporal density of the sequence. Then, a feature extraction convolutional layer uses a combination of large and small convolutional kernels to capture long-range local features and amplified short-term abrupt changes in the amplified temporal dimension. Finally, a temporal feature learning layer extracts bidirectional temporal dependency features. These three elements work synergistically to compensate for information loss caused by insufficient hardware sampling, significantly improving the system's ability to distinguish and recognize rapid, subtle gestures.
[0034] 5) The model training is stable, and the system has strong noise resistance: This invention sets up a batch normalization layer after the super-resolution upsampling layer, which can correct the data distribution offset caused by upsampling interpolation, accelerate model convergence, and make the training process more stable. Zero-field calibration eliminates zero-point drift in each acquisition channel, and normalization unifies the data distribution. Furthermore, a random deactivation mechanism and a bidirectional long short-term memory network are combined to suppress sensor noise interference and model overfitting, giving the system stronger adaptability and operational stability to magnetic sensor noise, environmental magnetic field fluctuations, and individual gesture differences. This effect solves the problems in the prior art where simple interpolation or resampling cannot reliably recover transient information, and models that directly process low-resolution original sequences are sensitive to noise and have weak generalization ability. Attached Figure Description
[0035] The present invention will be further described below with reference to the accompanying drawings and examples; Figure 1 A flowchart of a magnetic signal gesture recognition method based on super-resolution temporal enhancement provided in an embodiment of the present invention.
[0036] Figure 2 This is a hardware architecture block diagram of a magnetic signal gesture recognition system based on super-resolution temporal enhancement, provided in an embodiment of the present invention.
[0037] Figure 3 A comparison diagram of zero-point error before and after zero-field calibration provided for an embodiment of the present invention.
[0038] Figure 4 This is a schematic diagram of the structure of the gesture recognition neural network model provided in an embodiment of the present invention.
[0039] Figure 5 A comparison chart of the recognition accuracy of the present invention's solution and a benchmark solution provided in an embodiment of the present invention.
[0040] Figure 6This is a schematic diagram of the spatiotemporal evolution of the magnetic field during the execution of a single gesture, provided in an embodiment of the present invention.
[0041] Figure 7 This is a schematic diagram of the data preprocessing process.
[0042] Figure 8 This is a schematic diagram illustrating the change in sequence length during super-resolution upsampling and convolutional pooling. Detailed Implementation
[0043] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments; it should be understood that the specific embodiments described herein are only for explaining the present invention and are not intended to limit the present invention.
[0044] Example 1: Magnetic signal gesture recognition system based on super-resolution temporal enhancement: like Figure 2 As shown, this embodiment provides a magnetic signal gesture recognition system based on super-resolution temporal enhancement, including a magnetic sensing module, a signal acquisition module, a data processing module, and an output module.
[0045] Magnetic sensing module: Composed of a magnetic sensor array of 25 linear Hall sensors. In this embodiment, the linear Hall sensor is model A1324, with a sensitivity s of 5mV / Gs and an operating voltage of 3.3V. The 25 linear Hall sensors are evenly arranged in the area to be measured to sense changes in the magnetic field caused by the user's hand gestures and convert the changes in magnetic field strength into an analog voltage signal. Each linear Hall sensor's output terminal is equipped with a voltage divider protection circuit and an overload protection circuit to prevent damage from overvoltage and ensure the stability of the acquired signal.
[0046] Signal acquisition module: includes a microcontroller and a multiplexing circuit. In this embodiment, the microcontroller uses an STM32H7 series chip, responsible for controlling the channel switching of the multiplexing circuit, reading the analog-to-digital conversion results, and performing zero-field calibration and magnetic field strength conversion. The multiplexing circuit consists of three cascaded eight-channel analog switch chips 4051. The system defines a MUX_4051_t structure to manage each eight-channel analog switch chip uniformly. The common output terminal of the three eight-channel analog switch chips is connected to the analog-to-digital conversion channels 16, 17, and 14 of the microcontroller, respectively, i.e., pins PA0, PA1, and PA2; the enable pins EN of the three eight-channel analog switch chips are all grounded, ensuring that each chip is always in an active state. The connection relationship between the channel selection pins of each eight-channel analog switch chip and the microcontroller is shown in Table 1.
[0047]
[0048] Of the 25 linear Hall sensors, 24 are connected to the 24 input channels of three eight-channel analog switch chips. The remaining linear Hall sensor is directly connected to the analog-to-digital conversion channel 15 of the microcontroller, i.e., pin PA3, to form a reference acquisition channel, which is used to realize the synchronous or time-division acquisition of magnetic field signals in different regions.
[0049] The data processing module performs zero-field calibration and normalization on the original magnetic field time-series signal to obtain a standard input sequence, and then identifies and determines the target gesture category based on the gesture recognition neural network model. In this embodiment, the zero-field calibration and magnetic field strength conversion functions of the data processing module are executed by the microcontroller in the signal acquisition module, while the pre-processing of the input model and the construction and recognition functions of the gesture recognition neural network model are executed by the host computer. Together, they constitute the data processing module.
[0050] Output Module: In this embodiment, an HC-05 Bluetooth serial port module with a baud rate configured at 115200bps is used. It is connected to the USART1 pin of the microcontroller, with PA9 as the transmitting end and PA10 as the receiving end. The output module transmits the magnetic field strength data converted by the microcontroller to the host computer in real time for recognition by the data processing module; and outputs the recognized gesture commands to the terminal device. In other embodiments, the output module can also use a wired serial port, a WiFi module, or directly drive a local actuator to output gesture commands.
[0051] Example 2: A magnetic signal gesture recognition method based on super-resolution temporal enhancement: like Figure 1 As shown, this embodiment provides a magnetic signal gesture recognition method based on super-resolution temporal enhancement, which can be executed based on the system of embodiment 1, including steps S1 to S5.
[0052] It should be noted that the design motivation for the method in this embodiment is as follows: To reduce cost and power consumption, embedded hardware typically uses multiplexing to acquire multi-channel signals, resulting in a limited sampling rate for a single channel. At low sampling rates, rapid transient changes (high-frequency features) in gesture movements are easily lost, leading to low recognition rates in traditional machine learning algorithms. Furthermore, most existing deep learning solutions directly process the original sequence, making it difficult to recover subtle features from low-resolution signals. Therefore, this method introduces a super-resolution upsampling mechanism at the model front end, combined with batch normalization, large and small convolutional kernel combinations, and bidirectional temporal modeling, to compensate for the insufficient hardware sampling rate from the software side. The specific steps are as follows.
[0053] Step S1: Data Acquisition A microcontroller is used to control the multiplexing circuit to time-division select each acquisition channel of the magnetic field sensor array, acquiring the raw magnetic field timing signal caused by the user's gesture. The acquisition process is implemented through the ReadAllSensors function, and the microcontroller controls the GPIO pin output address signal through the SetMuxChannel function to switch the channels of the eight-channel analog switch chip. Specifically: (1) For chip 0, write the binary value of the channel number to pins PB0~PB2 to select the target acquisition channel; (2) After channel switching, a preset stabilization time of 1ms is set to wait for the output signal of the selected target acquisition channel to stabilize. (3) Start the analog-to-digital conversion, read the analog voltage value on pin PA0, and obtain the sampled value of the target acquisition channel; (4) Repeat the above gating, delay, and reading process for chip 1 and chip 2 to read the analog voltage values on pin PA1 and pin PA2 respectively; (5) Finally, read the analog voltage value on the directly connected reference acquisition channel, i.e., pin PA3.
[0054] The above process is executed cyclically according to the preset scanning order. 25 acquisition channels are selected in sequence, and the sampled values of each acquisition channel are stored into an array in time sequence to obtain the original magnetic field time sequence signal.
[0055] Step S2: Preprocessing: The original magnetic field timing signal is zero-field calibrated to eliminate zero-point drift and then normalized to obtain a standard input sequence. The specific processing includes the following steps.
[0056] (1) Zero-field calibration and magnetic field strength conversion: To eliminate zero-point drift, the CalibrateZeroField function is run upon system startup: In an environment without an external magnetic field, each acquisition channel is continuously sampled N times (N is 100 in this embodiment), and the zero-point offset of each acquisition channel is calculated using the following calibration formula: ; In the formula: For the first Zero offset of each acquisition channel; The first under the condition of no external magnetic field The first acquisition channel The original sample value; N is the number of calibration samples.
[0057] During system operation, the real-time sampled values of each acquisition channel are subtracted from the corresponding zero-point offset to obtain the difference signal: ; In the formula: For the first Real-time sampled values from each acquisition channel; For the first The difference signal of each acquisition channel.
[0058] The difference signal is then converted into sensor output voltage and magnetic field strength in sequence: ; ; In the formula: For the first The sensor output voltage corresponding to each acquisition channel; The reference voltage for analog-to-digital conversion is 3.3V in this embodiment; The number of bits for the analog-to-digital conversion is 12 in this embodiment; The preamplifier gain is set to 1.5 in this embodiment; The sensitivity of the Hall sensor; For the first The magnetic field strength measured by each acquisition channel, in Gaussian (Gs).
[0059] like Figure 3 The figure shows a comparison of zero-point errors before and after zero-field calibration. Before calibration, each acquisition channel exhibited varying degrees of zero-point offset, with zero-field readings fluctuating around a non-zero baseline. After zero-field calibration, the zero-point errors of each acquisition channel were effectively suppressed, providing a stable benchmark for subsequent normalization and model recognition.
[0060] (2) such as Figure 7 As shown, the overall data preprocessing process is as follows: the original magnetic field time series signal is first zero-field calibrated to obtain the zero-point offset of each acquisition channel, and then the zero-point drift is eliminated by difference operation. Subsequently, the magnetic field strength time series is obtained by voltage conversion and magnetic field strength conversion. The magnetic field strength time series is then cleaned to remove invalid data, truncated or padded to a preset time length, and mapped to the [0,1] interval by minimum-maximum normalization, finally obtaining the standard input sequence. At the same time, the gesture labels corresponding to each gesture sample are generated by one-hot encoding to generate model training labels, which correspond one-to-one with the standard input sequence, for use in training and recognition of the gesture recognition neural network model.
[0061] Pre-mold treatment: Before inputting data into the gesture recognition neural network model, the following further steps are performed: a. Data cleaning: Clean the time series of magnetic field strength measured by each acquisition channel, and remove invalid data such as metadata rows; b. Uniform Length: The magnetic field strength time sequence corresponding to each gesture sample after cleaning is truncated or padded to a preset time length. In this embodiment, the preset time length is 200 time steps, resulting in a uniform-length sample sequence; this preset time length is the same as the time length of the standard input sequence. ; c. Normalization: Perform min-max normalization on the equal-length sample sequences, mapping them to the [0,1] interval to obtain the standard input sequence: ; In the formula: X is the magnetic field strength value in the equal-length sample sequence; This represents the minimum magnetic field strength value in a sequence of samples of equal length. This represents the maximum magnetic field strength value in a sequence of samples of equal length. This is the standard input sequence obtained after normalization.
[0062] d. Label Encoding: Perform one-hot encoding on the gesture labels corresponding to each gesture sample to generate model training labels.
[0063] Step S3: Model Building Construct a gesture recognition neural network model. For example... Figure 4 As shown in the figure, the gesture recognition neural network model built based on the Python / Keras framework in this embodiment includes an input layer, a super-resolution upsampling layer, a feature extraction convolutional layer, a temporal feature learning layer, and a classification output layer arranged in sequence. The structure and parameters of each layer are shown in Table 2.
[0064]
[0065] The working principle of each layer is as follows: Super-resolution upsampling layer: One-dimensional interpolation upsampling is used to amplify the time dimension of the standard input sequence input to the input layer by a preset factor. The relationship between the sequence lengths before and after amplification is as follows: ; In the formula: The length of the standard input sequence; The time length of the amplified sequence; In this embodiment, the magnification is preset. The value is set to 2, which expands the sequence length from 200 to 400. This step simulates the super-resolution reconstruction process, interpolating to increase data density in the time dimension, providing space for subsequent extraction of high-frequency features.
[0066] Batch normalization layer: Following the super-resolution upsampling layer, it is used to correct the distribution offset of the data after upsampling interpolation, suppress training oscillations, and accelerate network convergence.
[0067] Feature extraction convolutional layer: This consists of a first convolutional block, a max-pooling layer, and a second convolutional block connected in sequence. The first convolutional block contains 64 one-dimensional convolutional kernels with a kernel length of 7. It performs sliding convolution on 7 consecutive sampling points in the amplified sequence, outputting 64 sets of temporal feature channels. Its wide receptive field can capture low-frequency, large-range trend features. The sequence length remains unchanged after convolution, and nonlinearity is introduced to fit complex signal patterns. Since the input sequence has been amplified, the large convolutional kernels can capture minute fluctuations that might be ignored in the original sequence. The max-pooling layer has a pooling window of 2 and performs temporal downsampling on the output of the first convolutional block, compressing the sequence length to half. ; In the formula: This represents the sequence length output by the max pooling layer. After pooling, the sequence length exactly offsets the doubling of length caused by upsampling, restoring it to the original order of magnitude of 200, thus controlling computational load while preserving significant features. The number of channels in the second convolutional block is doubled to 128, containing a one-dimensional convolutional kernel with a kernel length of 3, used to extract finer high-frequency details and capture short-term abrupt changes, spikes, and pulse-like local features.
[0068] like Figure 8 The image shows the changes in sequence length and waveform details during super-resolution upsampling and convolutional pooling of the standard input sequence: Figure 8 The time length of the standard input sequence in subgraph (a) is =200, due to the limitation of single-channel sampling rate, the gesture transient region (the peak near the 88th to 93rd time steps in the figure) is supported by only a few sampling points, resulting in sparse details; Figure 8 In subgraph (b), the sequence length is expanded to [length] after 2x interpolation upsampling. =400, interpolation of transition points between adjacent original sampling points, the transient fluctuations are stretched in the time dimension, and the waveform profile is more continuous; Figure 8 In subgraph (c), the first convolutional block slides convolution on a 400-length sequence with a wide receptive field of kernel length 7 to extract long-range local features, while keeping the sequence length unchanged at 400. Figure 8 In subgraph (d), the max pooling layer downsamples the sequence with a window of 2, compressing the sequence length back to its original value. =200, significant features are preserved; Figure 8 In subgraph (e), the second convolutional block, with a kernel length of 3, further highlights short-term mutation features on the compressed sequence, and transient spikes are significantly sharpened. It can be seen that super-resolution upsampling provides denser temporal sampling support for the convolutional kernel, enabling the effective capture and enhancement of minute fluctuations that are easily missed under low sampling rate conditions.
[0069] The temporal feature learning layer comprises a first bidirectional long short-term memory (LSTM) network layer, a random deactivation layer, and a second bidirectional LSM network layer, connected sequentially. This layer performs bidirectional temporal modeling on the convolutional feature sequence output from the feature extraction convolutional layer. Convolutional operations can only observe a local, fixed window, while the temporal sequence of gestures is crucial for recognition. Therefore, a bidirectional LSM network is used to specifically model the correlation between preceding and following time steps. The first bidirectional LSM network layer has 128 unidirectional memory units to store long-term temporal information. It analyzes the temporal sequence both forward (reading from frame 1 to the last) and backward (reading from the last to frame 1), utilizing past information to influence the future and using future information to corroborate previous anomalies, making the temporal features more complete. The output of this layer retains the temporal feature sequence across all time steps and is passed to the random deactivation layer. 40% of its input units are randomly deactivated to suppress overfitting, and 10% of the loop's internal states are deactivated to weaken the memory of loop weights and prevent the model from remembering training set noise. The random deactivation layer has a deactivation rate of 50%, meaning that 50% of neurons are randomly discarded, significantly reducing the risk of overfitting in large-capacity recurrent networks. The second bidirectional long short-term memory network layer has 64 unidirectional memory units. It compresses and refines the high-level temporal logic of the temporal feature sequence after random deactivation, no longer retaining time-step data, and only outputting the fused feature vector of the entire sequence to adapt to the subsequent classification output layer.
[0070] Classification output layer: includes fully connected layers and a softmax activation function, used to map the fused feature vector to scores for each category and output the probability of each gesture category.
[0071] Step S4: Feature Mapping and Recognition During model training, the standard input sequence obtained from step S2 and its one-hot encoded label input gesture recognition neural network model are subjected to supervised training.
[0072] During recognition, the standard input sequence is input into the gesture recognition neural network model. It sequentially passes through a super-resolution upsampling layer to amplify the time dimension, a feature extraction convolutional layer to extract local waveform features, and a temporal feature learning layer to extract bidirectional temporal dependency features. Finally, the fully connected layer in the classification output layer maps the fused feature vector (i.e., bidirectional temporal dependency features) output by the temporal feature learning layer into scores for each category, and then uses the Softmax activation function to calculate the probability of each gesture category. ; In the formula: For the first Scores for each gesture category; For gesture category number; Total number of gesture categories; The sequence number is the traversal index for the gesture category; It is a natural constant; For the input sequence to belong to the first The probability of gesture category.
[0073] Step S5: Output: Take the gesture category with the highest probability as the target gesture category: ; In the formula: The target gesture category is then defined. The target gesture category is then mapped to the corresponding gesture command, which is output by the output module and the corresponding operation is executed.
[0074] Comparative experiments and effect analysis: To verify the effectiveness of the technical solution of the present invention, a benchmark solution and the solution of the present invention were compared in an experiment. The configuration differences between the two solutions are shown in Table 3.
[0075]
[0076] like Figure 5 The figure shows a comparison of the recognition accuracy of the proposed solution and the benchmark solution. The training process exhibits three stages: In the early stages of training, the convergence speed of the proposed method is slower than that of the benchmark method. This is due to the cold start problem caused by super-resolution upsampling: the sequence length is doubled, and the model in the initial stage has not yet learned how to process these interpolated data points.
[0077] During the middle of training, starting from about the 10th training round, the accuracy of the proposed solution began to climb steadily with a steep slope, and the advantages brought by super-resolution enhancement gradually became apparent: longer sequences mean that the temporal feature learning layer has more time steps to capture subtle feature changes, the model learning efficiency is significantly improved, and the disadvantages in the early stage are quickly made up for.
[0078] In the later stages of training, around the 30th training round, the accuracy of the proposed solution on the validation set stabilized at over 99%, and the curve was smoother. In contrast, the baseline solution showed obvious sawtooth fluctuations between the 20th and 40th training rounds, with deep pits appearing at the 20th and 35th training rounds. This indicates that although the baseline solution learns quickly, it is more sensitive to noise and has relatively weak generalization ability.
[0079] In addition, such as Figure 6As shown, taking gesture Z as an example, the magnetic field strength measured by each acquisition channel during its execution exhibits a specific spatial evolution pattern over time: when the gesture enters, passes through, and leaves the array area, characteristic fluctuations in magnetic field strength appear sequentially in the acquisition channels at different locations, forming a spatiotemporal fingerprint that can be distinguished by the model. The above comparative experiments and the spatiotemporal evolution characteristics of the magnetic field together demonstrate that by adding a super-resolution upsampling layer and combining it with batch normalization, large and small convolutional kernel combinations, and bidirectional temporal modeling, the proposed solution has significant advantages in extracting subtle gesture features.
Claims
1. A method for gesture recognition based on super-resolution temporal enhancement of magnetic signals, characterized in that Includes the following steps: S1: Use a microcontroller to control the multiplexing circuit to select each acquisition channel of the magnetic sensor array in a time-division manner, and acquire the original magnetic field timing signal caused by the user's gesture. S2: Perform zero-field calibration and normalization on the original magnetic field time sequence signal to obtain the standard input sequence; S3: Construct a gesture recognition neural network model, which includes an input layer, a super-resolution upsampling layer, a feature extraction convolutional layer, a temporal feature learning layer, and a classification output layer arranged sequentially. S4: The standard input sequence is input into the gesture recognition neural network model, and then the time dimension is magnified by the super-resolution upsampling layer, the local waveform features are extracted by the feature extraction convolutional layer, and the bidirectional temporal dependency features are extracted by the temporal feature learning layer. Finally, the probability of each gesture category is output by the classification output layer. S5: Determine the target gesture category based on the probability of each gesture category, and output the gesture command corresponding to the target gesture category.
2. The method of claim 1, wherein the method is based on a super-resolution timing-enhanced magnetic signal gesture recognition method. S1 includes the following steps: S1.1: The microcontroller outputs an address signal to the multiplexing circuit through the GPIO pin to select the target acquisition channel; S1.2: After channel switching, a preset stabilization time is applied to wait for the output signal of the selected target acquisition channel to stabilize; S1.3: Start analog-to-digital conversion, read the analog voltage value output by the target acquisition channel, and obtain the sampled value of the target acquisition channel; S1.4: Execute S1.1~S1.3 in a loop according to the preset scanning order until all acquisition channels of the magnetic sensor array are traversed, and store the sampled values of each acquisition channel as the original magnetic field timing signal in time sequence. 3.The magnetic signal gesture recognition method based on super-resolution timing enhancement according to claim 2, characterized in that: In S1, the magnetic sensor array includes 25 linear Hall sensors, and the output terminal of each linear Hall sensor is provided with a voltage divider protection circuit and an overload protection circuit. The multiplexing circuit includes three cascaded eight-channel analog switch chips. The channel selection pins of each eight-channel analog switch chip are connected to the corresponding GPIO pins of the microcontroller, and the common output terminal of each eight-channel analog switch chip is connected to the corresponding analog-to-digital conversion channel of the microcontroller. Of the 25 linear Hall sensors, 24 are connected to the input channels of three eight-channel analog switch chips, and the remaining linear Hall sensor is directly connected to the independent analog-to-digital conversion channel of the microcontroller to form a reference acquisition channel.
4. The method of claim 3, wherein the method further comprises: S2 includes: 2.1: In an environment without an external magnetic field, continuously acquire data N times through each acquisition channel, and calculate the zero-point offset of each acquisition channel: ; In the formula: is the zero offset of the first acquisition channel; is the zero offset of the first acquisition channel; is the first original sampling value of the first acquisition channel in the environment without an external magnetic field; is the first original sampling value of the first acquisition channel in the environment without an external magnetic field; is the first original sampling value of the first acquisition channel in the environment without an external magnetic field; is the acquisition channel number; is the calibration sampling number; and N is the calibration sampling number. 2.2: During system operation, the real-time sampled values of each acquisition channel are subtracted from the corresponding zero-point offset to obtain the difference signal. ; In the formula: is the real-time sampling value of the first acquisition channel; is the difference signal of the first acquisition channel; 2.3: Convert the difference signal into sensor output voltage: ; In the formula: is the sensor output voltage corresponding to the i-th acquisition channel; is the sensor output voltage corresponding to the i-th acquisition channel; is the reference voltage of analog-digital conversion; is the number of bits of analog-digital conversion; G is the preamplifier gain; 2.4: Convert the sensor output voltage into magnetic field strength: ; In the formula: is the sensitivity of the Hall sensor; is the magnetic field strength measured by the i th acquisition channel.
5. The method of claim 4, wherein: S2 further includes: S2.5: Clean the time series of magnetic field strength measured by each acquisition channel and remove invalid data; S2.6: Truncate or fill the time sequence of magnetic field intensity corresponding to each gesture sample after cleaning to a preset time length to obtain a sample sequence of equal length; S2.7: Perform min-max normalization on the equal-length sample sequences, mapping them to the [0,1] interval to obtain the standard input sequence: ; wherein: X is the magnetic field strength value in the equal-length sample sequence; is the minimum magnetic field strength value in the equal-length sample sequence; is the maximum magnetic field strength value in the equal-length sample sequence; is the normalized standard input sequence; S2.8: Perform one-hot encoding on the gesture labels corresponding to each gesture sample to generate model training labels.
6. The method of claim 5, wherein: S3 includes: S3.1: The super-resolution upsampling layer uses one-dimensional interpolation upsampling to amplify the time dimension of the standard input sequence input to the input layer by a preset factor. The relationship between the sequence lengths before and after amplification is as follows: ; In the formula: is the time length of the standard input sequence; is the time length of the amplified sequence; is the preset amplification multiple; S3.2: After the super-resolution upsampling layer, a batch normalization layer is set to perform distribution correction on the amplified sequence, so as to stabilize the data distribution of the amplified sequence and accelerate the convergence of the gesture recognition neural network model.
7. The method of claim 6, wherein the method further comprises: The feature extraction convolutional layer comprises a first convolutional block, a max-pooling layer, and a second convolutional block connected in sequence. The extraction of local waveform features in S4 includes: 4.1: The first convolutional block performs sliding convolution on the magnified time dimension to capture long-distance local features with a wide receptive field. The first convolutional block contains 64 one-dimensional convolutional kernels with a length of 7. 4.2: The output sequence of the first convolutional block is downsampled by the max pooling layer, compressing the sequence length to half and restoring the sequence length to the order of magnitude before amplification; 4.3: The second convolutional block performs sliding convolution on the compressed sequence to capture short-term mutation features. The second convolutional block contains 128 one-dimensional convolutional kernels with a length of 3.
8. The method of claim 7, wherein: The temporal feature learning layer comprises a first bidirectional long short-term memory network layer, a random deactivation layer, and a second bidirectional long short-term memory network layer connected in sequence. The extraction of bidirectional temporal dependency features in S4 includes: 4.4: The first bidirectional long short-term memory network layer performs temporal modeling on the convolutional feature sequence output by the feature extraction convolutional layer simultaneously along the forward and backward directions, and outputs a temporal feature sequence that retains all time steps. The first bidirectional long short-term memory network layer has 128 unidirectional memory units, an input random inactivation rate of 40%, and a cyclic state inactivation rate of 10%. 4.5: The time-series feature sequences are randomly deactivated by the random deactivation layer with a deactivation rate of 50% to suppress overfitting; 4.6: The second bidirectional long short-term memory network layer compresses and refines the temporal feature sequence after random deactivation, and outputs the fused feature vector of the entire sequence. The number of unidirectional memory units in the second bidirectional long short-term memory network layer is 64.
9. The method of claim 8, wherein: The classification output layer includes a fully connected layer and a Softmax activation function, and S5 includes: 5.1: The fully connected layer maps the bidirectional temporal dependency features extracted by the temporal feature learning layer into scores for each category, and calculates the probability of each gesture category using the Softmax activation function. ; wherein: is the probability that the input sequence belongs to the th gesture class; is the score of the th gesture class; is the gesture class index; is the total number of gesture classes; is the gesture class traversal index; is a natural constant; 5.2: Select the gesture category corresponding to the highest probability as the target gesture category: ; In the formulae: is a target gesture class; 5.3: Map the target gesture category to the corresponding gesture command, output and execute the operation corresponding to the gesture command.
10. A magnetic signal gesture recognition system based on super-resolution temporal enhancement, characterized in that, The system includes: Magnetic sensing module: includes a magnetic sensor array, used to sense changes in the magnetic field caused by user gestures; Signal acquisition module: includes a microcontroller and a multiplexing circuit. The input terminal of the multiplexing circuit is connected to the magnetic sensing module, and the control terminal of the multiplexing circuit is connected to the microcontroller. The multiplexing circuit is used to control the multiplexing circuit to select each acquisition channel of the magnetic sensor array in a time-division manner to acquire the original magnetic field timing signal. The data processing module is used to perform zero-field calibration and normalization on the original magnetic field time-series signal to obtain a standard input sequence; and to input the standard input sequence into a pre-constructed gesture recognition neural network model. The gesture recognition neural network model includes an input layer, a super-resolution upsampling layer, a feature extraction convolutional layer, a temporal feature learning layer, and a classification output layer arranged sequentially. The standard input sequence is passed through the super-resolution upsampling layer to amplify the time dimension, the feature extraction convolutional layer to extract local waveform features, and the temporal feature learning layer to extract bidirectional temporal dependency features. The classification output layer outputs the probability of each gesture category and determines the target gesture category based on the probability of each gesture category. Output module: Used to output gesture commands corresponding to the target gesture category.
Citation Information
Patent Citations
Multi-path parallel magnetic field signal collecting and transmitting system and method
CN104459570A
Piano playing gesture recognition method based on inertia gloves
CN110210547A
Faster-RCNN-based gesture recognition system and method
CN113191421A
Gesture recognition device based on three-axis magnetic sensor group
CN116360585A
