A method and system for recognizing continuous gesture behavior using wireless radio frequency signals

Abstract

The application relates to a method and system for recognizing continuous gesture behavior by using wireless radio frequency signals, which comprises: arranging a transmitting device and at least two receiving devices, each device is arranged with at least two antennas, and there is no obstruction between the devices; a gesture to be recognized is formed on a plane constituted by the transmitting device and the receiving devices, and any device needs to be worn; the disturbance of the wireless radio frequency signals caused by hand movement is used to perceive and recognize different air gestures by measuring the change of the channel state information on the communication link. The application can be compatible with the existing wireless communication devices, and no additional special sensing device or specific sensing signal needs to be installed. The application can effectively cope with the continuous gesture action with a transition section not fixed, so that the recognition of various single or continuous gesture behaviors is more natural. The application can be applied in the fields of wireless communication and sensorless perception.

Description

Technical Field

[0001] This invention relates to the field of radio communication and sensorless sensing technology, and in particular to a method and system for recognizing continuous hand gestures using radio frequency signals. Background Technology

[0002] Gesture recognition, as a key component of human-computer interaction, is crucial for enhancing user experience through its accuracy and naturalness. Especially in scenarios involving continuous gesture recognition, users expect to interact with devices smoothly and naturally, rather than being forced to pause or adopt unnatural postures between each gesture. However, current gesture recognition technologies face numerous challenges in handling continuous gestures.

[0003] Existing gesture recognition technologies can be divided into two categories: contact and non-contact. Contact gesture recognition methods require users to wear fixed sensing devices on other parts of their body. While some contact-based gesture recognition methods offer a certain degree of continuous gesture recognition capability, the requirement for users to wear additional devices can be inconvenient and limits the naturalness and comfort of gesture activities to some extent. Non-contact gesture recognition technology, unlike contact methods that require direct contact or wearing devices, provides users with a more natural interactive experience. The core advantage of this type of technology lies in its non-invasiveness, allowing users to interact with gestures naturally without any device restrictions. Non-contact recognition systems are diverse, including but not limited to technologies based on vision, sound waves, radar, and Wi-Fi signals. In particular, recognition methods based on wireless radio frequency signals such as Wi-Fi identify gesture movements by analyzing signal changes generated during daily communication. Due to the widespread use of wireless devices such as Wi-Fi routers in indoor environments, this type of technology is not only inexpensive but also widely applicable, achieving efficient gesture recognition without the need for additional specialized equipment.

[0004] While some contactless gesture recognition technologies can handle continuous gesture scenarios, these methods typically rely on dedicated hardware and complex algorithms, limiting their application in everyday environments. For example, vision-based methods (such as gesture recognition methods utilizing deep learning) can handle some continuous gesture scenarios, but they are limited by the performance of visual sensors, such as lighting conditions and field of view, and may involve privacy issues. These methods usually require large amounts of data for training and may need to be adjusted or retrained for different application environments, limiting their versatility and flexibility.

[0005] For methods based on radio frequency signals, most existing contactless gesture recognition methods are designed for single gestures. These methods typically require a clear interval between gestures or a pause after each gesture, which is impractical in scenarios with continuous gesture flows. For example, technologies such as Wi-Finger, WiGesture, and HandGest, while supporting the recognition of continuous gestures to some extent, all require the user's hand to remain still during gesture transitions, and cannot recognize continuous handwritten character behavior with unrestricted transition movements, thus having certain limitations.

[0006] A major challenge in processing continuous gestures is accurately segmenting individual gestures from a continuous flow of gestures. Because the hand movements corresponding to the transitions between different gestures in a continuous flow of gestures are varied and unpredictable, it is difficult to determine a fixed pattern feature to accurately segment and identify individual gestures from the continuous flow of gestures without a large amount of gesture training data. Therefore, existing technologies struggle to accurately identify continuous gestures without explicit gesture segmentation. Summary of the Invention

[0007] To address the aforementioned problems, the purpose of this invention is to provide a method and system for recognizing continuous hand gestures using radio frequency signals. This method and system are compatible with existing wireless communication devices and require no additional dedicated sensing equipment or specific sensing signals. Furthermore, it effectively handles continuous hand gestures with variable transition segments, thereby achieving more natural recognition of various single or continuous hand gestures.

[0008] To achieve the above objectives, the present invention adopts the following technical solution: a method for recognizing continuous hand gestures using radio frequency signals, comprising: arranging a transmitting device and at least two receiving devices, each device having at least two antennas, with no obstruction between the devices; the hand gesture to be recognized is formed on a plane formed by the transmitting device and the receiving device, without the need for wearing any equipment; utilizing the disturbance of the radio frequency signal caused by hand movement, and then measuring the changes in channel state information on the communication link to sense and recognize different air hand gestures.

[0009] Furthermore, by utilizing the disturbances to wireless radio frequency signals caused by hand movements, and then measuring changes in channel state information on the communication link, different air gestures can be sensed and identified, including:

[0010] The Channel State Information (CSI) is acquired and preprocessed to obtain a complex CSI quotient signal sequence for multiple subcarriers;

[0011] By using the complex signal sequence of CSI quotients of multiple subcarriers, a CSI quotient base signal for gesture behavior perception is obtained;

[0012] Estimate the Doppler velocity corresponding to hand movements based on the CSI base signal;

[0013] Based on the estimated Doppler velocities corresponding to hand movements, the basic motion segments corresponding to continuous hand movements are extracted:

[0014] Based on the basic action segments corresponding to continuous hand movements, the gesture behaviors in the continuous hand action flow are matched to complete the perception and recognition of different aerial gestures.

[0015] Furthermore, Channel State Information (CSI) is acquired and preprocessed, including:

[0016] The transmitter sends a wireless radio frequency signal, and channel state information (CSI) is collected on all receiving devices. Corresponding CSI data is generated between each pair of transmitter and receiver devices.

[0017] The CSI data collected from two antennas of the same receiving device are divided by each other to generate a new complex CSI quotient time series.

[0018] Interpolation is performed on the processed CSI quotient time series containing timestamps on each subcarrier to form multiple complex CSI quotient signal sequences with uniform sampling rate and time distribution.

[0019] Furthermore, using the complex signal sequence of CSI quotients from multiple subcarriers, a CSI quotient base signal for gesture behavior perception is obtained, including:

[0020] The CSI quotient complex signal sequence on each subcarrier is divided into several sequence segments according to a fixed time window;

[0021] Within each time window, select complex sequence segments of CSI quotients of at least three subcarriers. Perform rotation and translation on the complex plane on the sequence segments of the second and subsequent subcarriers so that the sequence segments coincide and align with the sequence segments of the first subcarrier on the complex plane. Treat the aligned multi-subcarrier sequences as the same sequence.

[0022] Within each new time window, rotation and translation are repeated to ensure that a complex CSI quotient signal sequence aligned with multiple subcarrier sequences is obtained in each time window. This sequence is used as the CSI quotient base signal for new gesture behavior recognition.

[0023] Furthermore, based on the CSI base signal, the Doppler velocity corresponding to the hand movement is estimated, including:

[0024] For the obtained CSI base signal sequence, calculate the vector difference between consecutive data points to obtain the complex form of the difference time series;

[0025] Calculate the phase value of the differential time series to form a dynamic phase change sequence. Directly and linearly map the phase change sequence to Doppler velocity to obtain the Doppler velocity corresponding to the hand movement.

[0026] Furthermore, the basic motion segments corresponding to continuous hand movements are extracted, including:

[0027] On each receiving device, a Doppler velocity sequence is acquired. The square root of the sum of the squares of the Doppler velocities on the two receiving devices is then taken and integrated along the time axis. The total displacement of the hand movement at the current moment is estimated from the integration result.

[0028] Divide the Doppler velocity sequences of the two pairs of transmitter and receiver ends to obtain the Doppler velocity ratio sequence;

[0029] A fixed displacement length is set. Whenever the total displacement of the hand changes by a set displacement length, the data for the corresponding time period is extracted from the ratio sequence and solved using the arctan function to estimate the direction of hand movement within that time period.

[0030] When the estimated rate of change of the hand's movement direction exceeds the first threshold during that time period, the hand movement at that moment is identified as a basic movement of the "turning" type.

[0031] If the estimated rate of change of the hand movement direction within the time period does not exceed the first threshold, but the cumulative change of direction exceeds the second threshold, then the movement at that moment is identified as a basic movement of the "circular arc" type, and the directional clock nature of the basic movement is further determined by the movement direction.

[0032] Furthermore, the matching of gesture behaviors in a continuous hand motion stream includes:

[0033] Before recognizing a single predefined gesture, the gesture is first executed once, and a combination sequence containing the basic action is recorded as a reference sequence.

[0034] The system identifies basic hand gestures in real time from the user's continuous hand gesture stream. When a basic gesture is identified, the occurrence time and gesture type information of the basic gesture are stored. If the basic gesture is the first one identified, more basic gestures are identified from the hand gesture stream. If the identified gesture is not the first one, the occurrence time and type of the gesture are recorded, and a set of sequences of different lengths consisting of the gesture and other previously recorded basic gestures are created retrospectively. These sequences are then matched one by one with the reference sequence of predefined gestures.

[0035] If a predefined gesture sequence is matched, it is further determined whether the subsequence is part of a larger gesture behavior. Up to 3 basic hand movements are added to expand the sequence. If the expanded sequence matches a larger gesture, it is identified as this gesture; otherwise, the original matching result is maintained.

[0036] If a predefined gesture is recognized, all previously stored basic action information is cleared; the next basic action recognized is considered the first recognized basic action, and the above process is repeated to achieve continuous gesture recognition.

[0037] A system for recognizing continuous hand gestures using radio frequency signals includes: a transceiver device arrangement module, which arranges a transmitter and at least two receivers, each of which has at least two antennas, with no obstructions between the devices; a gesture formation module, in which the gesture to be recognized is formed on a plane formed by the transmitter and receivers, without the need for wearing any equipment; and a gesture recognition module, which uses the disturbance of the radio frequency signal caused by hand movement to sense and recognize different air gestures by measuring changes in channel state information on the communication link.

[0038] A computer-readable storage medium storing one or more programs, the one or more programs including instructions that, when executed by a computing device, cause the computing device to perform any of the methods described above.

[0039] A computing device includes: one or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs include instructions for performing any of the methods described above.

[0040] The present invention has the following advantages due to the adoption of the above technical solutions:

[0041] 1. This invention aims to detect and identify single or continuous hand gestures performed by a user. By utilizing the effect of hand movement on wireless signals and measuring the Channel State Information (CSI) on multiple receiving devices, the basic hand movements of the user can be identified in real time.

[0042] 2. This invention effectively identifies predefined gestures from a continuous flow of actions by using backtracking search and sequence matching.

[0043] 3. This invention does not rely on dedicated sensors, but uses existing wireless communication devices (such as Wi-Fi, LTE, 4G / 5G) to collect CSI values, thereby avoiding the need for additional hardware and reducing costs.

[0044] 4. This invention does not require clock synchronization between the transmitting and receiving devices, demonstrating its wide applicability. Whether it's a single gesture or a series of gestures, this scheme can accurately recognize them. It allows users to transition freely during continuous gesture recognition, adapting to different hand movement speeds, enhancing the flexibility of recognition. This scheme can operate continuously in real time, is easy to use, and significantly expands its application areas. In experiments recognizing continuous gestures of the digits 0-9, the accuracy rate reached 89.6%, significantly outperforming existing technologies. Attached Figure Description

[0045] Figure 1 These are schematic diagrams of three different continuous handwriting gestures 2 and 3 in embodiments of the present invention;

[0046] Figure 2 This is a schematic diagram illustrating the matching of gestures '2', '3', and '5' from a continuous stream of hand movements in an embodiment of the present invention.

[0047] Figure 3 This is a flowchart of a method for recognizing continuous hand gestures using radio frequency signals in an embodiment of the present invention;

[0048] Figure 4 This is a schematic diagram of device deployment in an embodiment of the present invention;

[0049] Figure 5 This is a schematic diagram illustrating the extraction and matching of basic action sequences from consecutive handwritten '2', '3', and '5' in an embodiment of the present invention. Detailed Implementation

[0050] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the described embodiments of the present invention are within the scope of protection of the present invention.

[0051] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of exemplary embodiments according to the invention. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0052] While existing technologies have made some progress in single gesture recognition, significant technical challenges and limitations remain in the more complex and practical field of continuous gesture recognition. Currently, there is no method that can effectively recognize continuous user gestures using only existing communication equipment without the use of dedicated sensors, thus providing a more natural and seamless human-computer interaction experience. Therefore, this invention provides a method and system for recognizing continuous gesture behavior using wireless radio frequency signals. The basic principle is as follows: a signal propagates from the transmitter to the receiver along multiple paths in the environmental space. During this process, the user's hand movement corresponds to a change in the first reflection path. This change in the first reflection path corresponds to the gesture movement. As the hand moves, the length of the reflection path corresponding to the hand continuously changes. For every wavelength change in the reflection path length, the phase of the gesture signal changes by 2π. Since the phase change of the gesture signal corresponds to the Doppler velocity of the gesture relative to the transceiver, by deploying two pairs of non-parallel transceiver devices, the Doppler velocities in two different directions can be obtained from the gesture signals at the two receivers, thereby calculating the change in the hand's motion state (including direction and distance) at each moment. Based on the different hand movement states at different times, valid hand gestures are identified from a continuous flow of hand gestures.

[0053] This invention overcomes the limitations of existing technologies by employing wireless radio frequency signals to recognize continuous hand gestures. It primarily relies on changes in Channel State Information (CSI) captured during radio communication to accurately detect and recognize hand movements. This invention is compatible with existing wireless communication devices and requires no additional dedicated sensing equipment or specific sensing signals. Furthermore, it effectively handles continuous hand gestures with variable transition segments, thus achieving more natural recognition of various single or continuous hand gestures.

[0054] In natural, continuous gesture recognition scenarios, after completing one gesture, the hand needs to move to a suitable position to perform the next gesture. When the hand moves between two consecutive gestures, there are no unique features to mark the boundaries between gesture segments and transition segments. The movement pattern of the hand during the transition segment is also unpredictable. The transition between adjacent gestures may have different displacements and directions (e.g., ...). Figure 1 Writing the numbers 2 and 3 consecutively by hand results in different signal changes. Confusing gesture segments with transition segments will lead to incorrect gesture recognition results. Therefore, the key to continuous gesture recognition is to avoid the influence of diverse transition segments in continuous hand movements on the gesture recognition results.

[0055] To avoid the influence of non-fixed transitional hand movement segments between consecutive gestures on the recognition results, this invention proposes a method and system for recognizing consecutive gesture behaviors using radio frequency signals, which is a gesture recognition scheme based on fine-grained motion segmentation. Specifically, this scheme proposes a hand motion segmentation method that divides complex hand motion flows into basic basic actions based on the inherent geometric characteristics of hand motion (derived from displacement and direction information). Since each gesture consists of several basic actions, the consecutive gesture recognition problem is transformed into a sequence successive matching problem. Because the temporal granularity of the defined basic actions is sufficiently fine, the influence of transitional segments on the sequence of basic gesture actions is very limited. In this case, transitional segments will not hinder the successive matching of the basic action sequence or the consecutive gesture recognition results.

[0056] Specifically, the signal is segmented into signal segments corresponding to the aforementioned basic actions by extracting features from the radio frequency signal, and the type of basic action corresponding to each signal segment is identified. Then, the sequence of basic action types is compared with the template sequence of the gesture set to complete the recognition of multiple consecutive gestures. Figure 2 As shown, continuous radio frequency signals are segmented into basic action sequences based on signal characteristics, and three consecutive gestures (numbers 2, 3, and 5) are identified by comparing them with template sequences in a digital gesture set. This fine-grained action segmentation avoids large-scale confusion between transitional segments and gesture segments, ultimately achieving continuous gesture recognition for diverse interactive actions in natural scenes.

[0057] In one embodiment of the present invention, a method for recognizing continuous hand gestures using radio frequency signals is provided. This method does not require a dedicated sensor and is based on the sensing technology of existing communication devices. It uses radio frequency signals to detect and recognize continuous hand gestures (including hands or fingers).

[0058] In this embodiment, as Figure 3 As shown, the method includes the following steps:

[0059] 1) Arrange one transmitting device Tx and at least two receiving devices Rx, each device having at least two antennas, with no obstructions between the devices;

[0060] 2) The gesture to be recognized is formed on a plane consisting of a transmitting device and a receiving device, without the need to wear any equipment;

[0061] 3) By utilizing the disturbances caused by hand movements to the radio frequency signals, and then measuring the changes in channel state information on the communication link, different air gestures can be sensed and identified.

[0062] In this embodiment, specifically, as shown in... Figure 4As shown, a transmitting device and two receiving devices are arranged, with two antennas on each receiving device, and no obstructions between the devices. Utilizing the disturbance to the wireless radio frequency signal caused by hand movement, different continuous air gestures are sensed and identified by measuring changes in channel state information (CSI) on the communication link. In practice, the user moves their hand to perform air gestures on a plane formed by one transmitting device Tx (a WiFi router or wireless access point AP) and two receiving devices Rx (a personal computer or smartphone, etc.), without wearing any protective equipment. The sensing signals are OFDM modulated and can return channel state information (CSI). The exact location of the transmitting and receiving devices is not required for identification.

[0063] In step 3) above, the disturbance to the radio frequency signal caused by hand movement is utilized, and then the changes in channel state information on the communication link are measured to sense and identify different air gestures. This includes the following steps:

[0064] 3.1) Obtain Channel State Information (CSI) and preprocess it to obtain a complex signal sequence of CSI quotients for multiple subcarriers;

[0065] 3.2) Using the complex signal sequence of CSI quotients of multiple subcarriers, the CSI quotient base signal for gesture behavior perception is obtained;

[0066] 3.3) Estimate the Doppler velocity corresponding to the hand movement based on the CSI base signal;

[0067] 3.4) Extract the basic motion segments corresponding to continuous hand movements from the estimated Doppler velocities:

[0068] 3.5) Based on the basic action segments corresponding to continuous hand movements, match the gesture behaviors in the continuous hand action flow to complete the perception and recognition of different aerial gestures.

[0069] In step 3.1) above, obtaining Channel State Information (CSI) and performing preprocessing includes the following steps:

[0070] 3.1.1) The transmitting device Tx transmits a wireless radio frequency signal, and the channel state information (CSI) is collected on all receiving devices Rx respectively. Corresponding CSI data is generated between each pair of transmitting devices and receiving devices (Tx-Rx).

[0071] The complete CSI data stream includes CSI time series from two antennas and multiple OFDM subcarriers, which can be acquired during normal wireless data communication transmission.

[0072] 3.1.2) Perform mutual division on the CSI data collected from the two antennas of the same receiving device to generate a new complex CSI quotient time series;

[0073] 3.1.3) Interpolate the processed CSI quotient time series containing timestamps on each subcarrier to form multiple complex CSI quotient signal sequences with uniform sampling rate and time distribution.

[0074] In step 3.2) above, the CSI quotient base signal for gesture behavior perception is obtained using a complex signal sequence of CSI quotients from multiple subcarriers, including the following steps:

[0075] 3.2.1) Divide the complex CSI quotient signal sequence on each subcarrier into several sequence segments according to a fixed time window;

[0076] 3.2.2) Within each time window, select complex sequence segments of CSI quotients of at least three subcarriers. Perform rotation and translation on the complex plane on the sequence segments of the second and subsequent subcarriers so that the sequence segments coincide and align with the sequence segments of the first subcarrier on the complex plane. By splicing multiple subcarriers, the aligned multi-subcarrier sequence is regarded as the same sequence.

[0077] The rotation and translation in the complex plane is achieved by using a search algorithm to find an optimal vector and applying it to the sequence segment to make the sequence segment coincide and align with the sequence segment of the first subcarrier as much as possible in the complex plane.

[0078] 3.2.3) Repeat the rotation and translation in step 3.2.2) within each new time window to ensure that a complex CSI quotient signal sequence after multiple subcarrier sequences are aligned can be obtained in each time window. This sequence is used as the CSI quotient base signal for new gesture behavior recognition.

[0079] In step 3.3) above, estimating the Doppler velocity corresponding to the hand movement based on the CSI base signal includes the following steps:

[0080] 3.3.1) Process the obtained CSI base signal sequence, calculate the vector difference between consecutive data points in the sequence, and obtain the difference time series in complex form;

[0081] 3.3.2) Calculate the phase value of the differential time series to form a dynamic phase change sequence. Directly and linearly map the phase change sequence to the Doppler velocity to obtain the Doppler velocity corresponding to the hand movement.

[0082] In step 3.4) above, extracting the basic motion segments corresponding to continuous hand movements includes the following steps:

[0083] 3.4.1) On each receiving device, acquire the Doppler velocity sequence, synthesize the Doppler velocities on the two receiving devices, that is, take the square root of the sum of squares and integrate along the time axis, and estimate the total displacement of the hand movement at the current moment from the integration result;

[0084] 3.4.2) Divide the Doppler velocity sequences of the two pairs of transmitter and receiver ends to obtain the Doppler velocity ratio sequence;

[0085] 3.4.3) Set a fixed displacement length (e.g., 1 cm). Whenever the total displacement of the hand changes by a set displacement length, extract the data for the corresponding time period from the ratio sequence and solve it using the arctan function to estimate the direction of hand movement during that time period.

[0086] 3.4.4) When the estimated rate of change of the hand's movement direction exceeds the first threshold during the time period, the hand movement at that moment is identified as a basic movement of the "turning" type.

[0087] In this embodiment, preferably, the first threshold is π rad per centimeter.

[0088] 3.4.5) When the estimated highest rate of change of the hand movement direction within the time period does not exceed the first threshold, but the cumulative change of direction exceeds the second threshold, the action at that moment is identified as a basic action of the "circular arc" type, and the directional clockness of the basic action is further determined by the direction of movement.

[0089] In this embodiment, preferably, the second threshold is 1 / 2π rad.

[0090] In step 3.5 above, matching gesture behaviors in a continuous hand motion flow includes the following steps:

[0091] 3.5.1) Before recognizing a single predefined gesture, the gesture is first executed once, following the method in step 3.4), and the combined sequence containing the basic action is recorded as a reference sequence; this process only requires data collection once.

[0092] 3.5.2) Real-time recognition of basic hand gestures from the user's continuous hand motion stream. When a basic gesture is recognized, its occurrence time and type information are stored. If this is the first recognized basic gesture, more basic gestures are recognized from the hand motion stream. If it is not the first recognized gesture, its occurrence time and type are recorded, and a set of sequences of different lengths consisting of this gesture and other previously recorded basic gestures is created retrospectively. These sequences are then matched one by one with a reference sequence of predefined gestures.

[0093] 3.5.3) If a predefined gesture sequence is matched, it is further determined whether the subsequence is part of a larger gesture behavior. Up to 3 basic hand movements are added to the current subsequence as an extension sequence. If the extension sequence matches a larger gesture, it is recognized as this gesture; otherwise, if no larger gesture is matched after adding up to 3 additional basic movements, the previously matched gesture is considered to be the correct recognition result, i.e., the original matching result is maintained.

[0094] 3.5.4) If a predefined gesture is recognized, all previously stored basic action information is cleared. The next basic action recognized after this is considered the first recognized basic action.

[0095] 3.5.5) Repeat the process from 3.5.2) to 3.5.4) to achieve continuous gesture recognition.

[0096] The following example uses WiFi signals to recognize continuous user gestures. A router configured to operate in the 5GHz band and equipped with an omnidirectional antenna supporting the 802.11n protocol is used as the transmitting device. A mini PC with an Intel 5300 series network card and two omnidirectional receiving antennas is used as the receiving device. The receiving device runs Linux CSI tools developed by the University of Washington to obtain CSI data from the network card. Each antenna contains CSI information, including information about the 30 subcarriers generated after OFDM frequency modulation.

[0097] This example uses recognizing three consecutive digits '2-3-5' written in the air. When detecting the gesture, place the antenna on router Tx horizontally on the ground. Similarly, place the antennas of the two receiving devices Rx1 and Rx2 horizontally on the ground. Note that during placement, avoid parallel lines connecting Tx-Rx1 and Tx-Rx2; otherwise, there are no precise requirements for device placement.

[0098] CSI data is continuously collected on Rx1 and Rx2 and transmitted to a PC for centralized processing via WiFi network.

[0099] Generate a suitable amount of data communication at the WiFi router end to achieve a fixed CSI sampling rate. In this example, the sampling rate is set to 400Hz.

[0100] The user writes three consecutive numbers '2-3-5' in any way on the plane formed by the two lines Tx-Rx1 and Tx-Rx2.

[0101] CSI data preprocessing is performed, which involves first time alignment, then dividing the CSI data from two antennas on the same receiving device to obtain the phase-corrected CSI quotient. Then, the CSI sequences on multiple subcarriers are aligned and concatenated to obtain the base signal for gesture behavior perception.

[0102] Doppler velocity sequences are extracted from the base signals of gesture perception on each receiving device Rx. Then, the displacement and direction of motion characteristics of the corresponding hand movements are calculated based on the Doppler velocity sequences. Based on these displacement and direction characteristics, continuous hand movements are divided into several basic actions, and the types of different basic actions are identified according to the changing characteristics of each basic action, thus forming a sequence of basic action types, such as... Figure 5 As shown.

[0103] By using a backtracking method to match the generated basic action type sequence with the reference basic action sequence in the existing gesture set, the continuous gesture was successfully identified as the handwritten digits '2-3-5'.

[0104] In one embodiment of the present invention, a system for recognizing continuous hand gestures using radio frequency signals is provided, comprising:

[0105] The transceiver equipment layout module includes one transmitting device and at least two receiving devices, each device having at least two antennas, with no obstructions between the devices;

[0106] The gesture formation module forms the gesture to be recognized on a plane consisting of a transmitting device and a receiving device, without the need for wearing any equipment;

[0107] The gesture recognition module utilizes the disturbances caused by hand movements to wireless radio frequency signals, and then measures the changes in channel state information on the communication link to sense and recognize different air gestures.

[0108] In the above embodiments, the disturbance to the wireless radio frequency signal caused by hand movement is utilized, and then the changes in channel state information on the communication link are measured to sense and identify different air gestures, including:

[0109] The Channel State Information (CSI) is acquired and preprocessed to obtain a complex CSI quotient signal sequence for multiple subcarriers;

[0110] By using the complex signal sequence of CSI quotients of multiple subcarriers, a CSI quotient base signal for gesture behavior perception is obtained;

[0111] Estimate the Doppler velocity corresponding to hand movements based on the CSI base signal;

[0112] Based on the estimated Doppler velocities corresponding to hand movements, the basic motion segments corresponding to continuous hand movements are extracted:

[0113] Based on the basic action segments corresponding to continuous hand movements, the gesture behaviors in the continuous hand action flow are matched to complete the perception and recognition of different aerial gestures.

[0114] In the above embodiments, obtaining and preprocessing Channel State Information (CSI) includes:

[0115] The transmitter sends a wireless radio frequency signal, and channel state information (CSI) is collected on all receiving devices. Corresponding CSI data is generated between each pair of transmitter and receiver devices.

[0116] The CSI data collected from two antennas of the same receiving device are divided by each other to generate a new complex CSI quotient time series.

[0117] Interpolation is performed on the processed CSI quotient time series containing timestamps on each subcarrier to form multiple complex CSI quotient signal sequences with uniform sampling rate and time distribution.

[0118] In the above embodiments, a CSI quotient base signal for gesture behavior perception is obtained using a complex signal sequence of CSI quotients from multiple subcarriers, including:

[0119] The CSI quotient complex signal sequence on each subcarrier is divided into several sequence segments according to a fixed time window;

[0120] Within each time window, select complex sequence segments of CSI quotients of at least three subcarriers. Perform rotation and translation on the complex plane on the sequence segments of the second and subsequent subcarriers so that the sequence segments coincide and align with the sequence segments of the first subcarrier on the complex plane. Treat the aligned multi-subcarrier sequences as the same sequence.

[0121] Within each new time window, rotation and translation are repeated to ensure that a complex CSI quotient signal sequence aligned with multiple subcarrier sequences is obtained in each time window. This sequence is used as the CSI quotient base signal for new gesture behavior recognition.

[0122] In the above embodiments, estimating the Doppler velocity corresponding to hand movements based on the CSI base signal includes:

[0123] For the obtained CSI base signal sequence, calculate the vector difference between consecutive data points to obtain the complex form of the difference time series;

[0124] Calculate the phase value of the differential time series to form a dynamic phase change sequence. Directly and linearly map the phase change sequence to Doppler velocity to obtain the Doppler velocity corresponding to the hand movement.

[0125] In the above embodiments, the extraction of basic motion segments corresponding to continuous hand movements includes:

[0126] On each receiving device, a Doppler velocity sequence is acquired. The square root of the sum of the squares of the Doppler velocities on the two receiving devices is then taken and integrated along the time axis. The total displacement of the hand movement at the current moment is estimated from the integration result.

[0127] Divide the Doppler velocity sequences of the two pairs of transmitter and receiver ends to obtain the Doppler velocity ratio sequence;

[0128] A fixed displacement length is set. Whenever the total displacement of the hand changes by a set displacement length, the data for the corresponding time period is extracted from the ratio sequence and solved using the arctan function to estimate the direction of hand movement within that time period.

[0129] When the estimated rate of change of the hand's movement direction exceeds the first threshold during that time period, the hand movement at that moment is identified as a basic movement of the "turning" type.

[0130] If the estimated rate of change of the hand movement direction within the time period does not exceed the first threshold, but the cumulative change of direction exceeds the second threshold, then the movement at that moment is identified as a basic movement of the "circular arc" type, and the directional clock nature of the basic movement is further determined by the movement direction.

[0131] In the above embodiments, matching gesture behaviors in a continuous hand motion stream includes:

[0132] Before recognizing a single predefined gesture, the gesture is first executed once, and a combination sequence containing the basic action is recorded as a reference sequence.

[0133] The system identifies basic hand gestures in real time from the user's continuous hand gesture stream. When a basic gesture is identified, the occurrence time and gesture type information of the basic gesture are stored. If the basic gesture is the first one identified, more basic gestures are identified from the hand gesture stream. If the identified gesture is not the first one, the occurrence time and type of the gesture are recorded, and a set of sequences of different lengths consisting of the gesture and other previously recorded basic gestures are created retrospectively. These sequences are then matched one by one with the reference sequence of predefined gestures.

[0134] If a predefined gesture sequence is matched, it is further determined whether the subsequence is part of a larger gesture behavior. Up to 3 basic hand movements are added to expand the sequence. If the expanded sequence matches a larger gesture, it is identified as this gesture; otherwise, the original matching result is maintained.

[0135] If a predefined gesture is recognized, all previously stored basic action information is cleared; the next basic action recognized is considered the first recognized basic action, and the above process is repeated to achieve continuous gesture recognition.

[0136] The system provided in this embodiment is used to execute the above-described method embodiments. For specific processes and details, please refer to the above embodiments, which will not be repeated here.

[0137] In one embodiment of the present invention, a computing device is provided, which can be a terminal and may include: a processor, a communication interface, memory, a display screen, and an input device. The processor, communication interface, and memory communicate with each other via a communication bus. The processor provides computing and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system and computer programs, which, when executed by the processor, implement the methods described in the above embodiments. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The communication interface is used for wired or wireless communication with external terminals. Wireless communication can be achieved through Wi-Fi, a management network, NFC (Near Field Communication), or other technologies. The display screen can be a liquid crystal display or an e-ink display. The input device can be a touch layer covering the display screen, or buttons, a trackball, or a touchpad mounted on the casing of the computing device, or an external keyboard, touchpad, or mouse. The processor can call logical instructions stored in the memory.

[0138] Furthermore, the logical instructions in the aforementioned memory can be implemented as software functional units and sold or used as independent products, and can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0139] In one embodiment of the present invention, a computer program product is provided, the computer program product including a computer program stored on a non-transitory computer-readable storage medium, the computer program including program instructions, and when the program instructions are executed by a computer, the computer is able to perform the methods provided in the above-described method embodiments.

[0140] In one embodiment of the present invention, a non-transitory computer-readable storage medium is provided, which stores server instructions that cause a computer to perform the methods provided in the above embodiments.

[0141] The computer-readable storage medium provided in the above embodiments has a similar implementation principle and technical effect to the above method embodiments, and will not be described again here.

[0142] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0143] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0144] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0145] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for recognizing continuous hand gestures using radio frequency signals, characterized in that, include: Arrange one transmitting device and at least two receiving devices, each device having at least two antennas, with no obstructions between the devices; The gesture to be recognized is formed on a plane consisting of a transmitting device and a receiving device, without the need to wear any equipment; By utilizing the disturbances to radio frequency signals caused by hand movements, and then measuring changes in channel state information on the communication link, different air gestures can be sensed and identified, including: The Channel State Information (CSI) is acquired and preprocessed to obtain a complex CSI quotient signal sequence for multiple subcarriers; By using the complex signal sequence of CSI quotients of multiple subcarriers, a CSI quotient base signal for gesture behavior perception is obtained; Estimate the Doppler velocity corresponding to hand movements based on the CSI base signal; Based on the estimated Doppler velocity corresponding to the hand movement, the basic motion segments corresponding to continuous hand movements are extracted, including: acquiring a Doppler velocity sequence on each receiving device; taking the square root of the sum of squares of the Doppler velocities on two receiving devices and integrating along the time axis; estimating the total displacement of the hand movement at the current moment from the integration result; dividing the Doppler velocity sequences of the two pairs of transmitting and receiving ends to obtain a Doppler velocity ratio sequence; setting a fixed displacement length, whenever the total hand displacement changes by a set displacement length, extracting the data of the corresponding time period from the ratio sequence and solving it using the arctan function to estimate the direction of hand movement within that time period; when the estimated highest rate of change of the hand movement direction within that time period exceeds a first threshold, the hand movement at that moment is identified as a "turning" type basic motion; when the estimated highest rate of change of the hand movement direction within that time period does not exceed the first threshold, but the cumulative change of direction exceeds a second threshold, the movement at that moment is identified as a "circular arc" type basic motion, and the directional clock nature of the basic motion is further determined by the motion direction. Based on the basic action segments corresponding to continuous hand movements, the hand gestures in the continuous hand motion flow are matched to complete the perception and recognition of different aerial hand gestures, including: Before recognizing a single predefined gesture, the gesture is first executed once, and a combination sequence containing the basic action is recorded as a reference sequence. The system identifies basic hand gestures in real time from the user's continuous hand gesture stream. When a basic gesture is identified, the occurrence time and gesture type information of the basic gesture are stored. If the basic gesture is the first one identified, more basic gestures are identified from the hand gesture stream. If the identified gesture is not the first one, the occurrence time and type of the gesture are recorded, and a set of sequences of different lengths consisting of the gesture and other previously recorded basic gestures are created retrospectively. These sequences are then matched one by one with the reference sequence of predefined gestures. If a predefined gesture sequence is matched, it is further determined whether the subsequence is part of a larger gesture behavior. Up to 3 basic hand movements are added to expand the sequence. If the expanded sequence matches a larger gesture, it is identified as this gesture; otherwise, the original matching result is maintained. If a predefined gesture is recognized, all previously stored basic action information is cleared; the next basic action recognized is considered the first recognized basic action, and the above process is repeated to achieve continuous gesture recognition.

2. The method for recognizing continuous hand gestures using radio frequency signals as described in claim 1, characterized in that, Acquire Channel State Information (CSI) and perform preprocessing, including: The transmitter sends a wireless radio frequency signal, and channel state information (CSI) is collected on all receiving devices. Corresponding CSI data is generated between each pair of transmitter and receiver devices. The CSI data collected from two antennas of the same receiving device are divided by each other to generate a new complex CSI quotient time series. Interpolation is performed on the processed CSI quotient time series containing timestamps on each subcarrier to form multiple complex CSI quotient signal sequences with uniform sampling rate and time distribution.

3. The method for recognizing continuous hand gestures using radio frequency signals as described in claim 1, characterized in that, Using a complex sequence of CSI quotient signals from multiple subcarriers, a CSI quotient base signal for gesture behavior perception is obtained, including: The CSI quotient complex signal sequence on each subcarrier is divided into several sequence segments according to a fixed time window; Within each time window, select complex sequence segments of CSI quotients of at least three subcarriers. Perform rotation and translation on the complex plane on the sequence segments of the second and subsequent subcarriers so that the sequence segments coincide and align with the sequence segments of the first subcarrier on the complex plane. Treat the aligned multi-subcarrier sequences as the same sequence. Within each new time window, rotation and translation are repeated to ensure that a complex CSI quotient signal sequence aligned with multiple subcarrier sequences is obtained in each time window. This sequence is used as the CSI quotient base signal for new gesture behavior recognition.

4. The method for recognizing continuous hand gestures using radio frequency signals as described in claim 1, characterized in that, Based on the CSI base signal, estimate the Doppler velocity corresponding to hand movements, including: For the obtained CSI base signal sequence, calculate the vector difference between consecutive data points to obtain the complex form of the difference time series; Calculate the phase value of the differential time series to form a dynamic phase change sequence. Directly and linearly map the phase change sequence to Doppler velocity to obtain the Doppler velocity corresponding to the hand movement.

5. A system for recognizing continuous hand gestures using radio frequency signals, for implementing the method for recognizing continuous hand gestures using radio frequency signals as described in any one of claims 1 to 4, characterized in that, include: The transceiver equipment layout module includes one transmitting device and at least two receiving devices, each device having at least two antennas, with no obstructions between the devices; The gesture formation module forms the gesture to be recognized on a plane consisting of a transmitting device and a receiving device, without the need for wearing any equipment; The gesture recognition module utilizes the disturbances caused by hand movements to wireless radio frequency signals, and then measures the changes in channel state information on the communication link to sense and recognize different air gestures.

6. A computer-readable storage medium for storing one or more programs, characterized in that, The one or more programs include instructions that, when executed by a computing device, cause the computing device to perform any of the methods described in claims 1 to 4.

7. A computing device, characterized in that, include: One or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs including instructions for performing any of the methods described in claims 1 to 4.

Images