Gesture interaction method and device, ring and storage medium

Abstract

The invention discloses a gesture interaction method and device, a ring and a storage medium, and relates to the technical field of intelligent wearable device.The method comprises the steps that a hand motion data stream of a user is collected in real time through an inertial measurement unit of the intelligent ring, and an electromyographic signal data stream of the user is collected in real time through an electromyographic sensor of the intelligent ring; when the signal feature of the electromyographic signal data flow meets a preset activation condition and the track starting section of the hand motion data flow and the electromyographic signal data flow meet a synchronism condition in time sequence, a gesture interaction signal is triggered; and in response to the gesture interaction signal, executing an interaction operation. The identification accuracy of the user interaction intention in the office scene can be improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of intelligent wearable devices, and particularly relates to a gesture interaction method and device, a ring, and a storage medium. BACKGROUND

[0002] The existing intelligent rings and other intelligent wearable devices generally rely on inertial measurement units (IMUs) to detect hand movement trajectories in order to realize gesture interaction functions.

[0003] However, in complex office scenarios, there are a large number of unconscious daily activities of a user's hands, such as typing, organizing items, etc., and it is difficult to effectively distinguish such unintentional actions from intentional control instruction gestures by relying only on IMU movement trajectory analysis, resulting in a high false trigger rate of the interaction system, poor reliability, and an impact on user experience.

[0004] To sum up, how to improve the recognition accuracy of user interaction intentions in an office scenario has become a technical problem that needs to be solved in the field.

[0005] The above content is only used to assist in understanding the technical solutions of the present application and does not represent an admission that the above content is prior art. SUMMARY

[0006] The main purpose of the present application is to provide a gesture interaction method, device, ring, and storage medium, which aims to improve the recognition accuracy of user interaction intentions in an office scenario.

[0007] To achieve the above purpose, the present application provides a gesture interaction method applied to an intelligent ring, which comprises the following steps: real-time collection of hand movement data streams of a user by an inertial measurement unit of the intelligent ring, and real-time collection of electromyographic signal data streams of the user by electromyographic sensors of the intelligent ring; triggering of a gesture interaction signal when signal characteristics of the electromyographic signal data streams meet preset activation conditions, and a trajectory starting section of the hand movement data streams meets synchronization conditions in time sequence with the electromyographic signal data streams; performance of an interaction operation in response to the gesture interaction signal.

[0008] In an embodiment, the step of triggering a gesture interaction signal when signal characteristics of the electromyographic signal data streams meet preset activation conditions, and a trajectory starting section of the hand movement data streams meets synchronization conditions in time sequence with the electromyographic signal data streams comprises the following steps: real-time calculation of short-time energy of the electromyographic signal data streams; determining that the signal feature of the myoelectric signal data stream meets a preset activation condition and generating a trigger signal when the duration that the short-time energy exceeds the first threshold value reaches a first time period; analyzing whether a trajectory starting segment meeting a preset gesture starting feature exists in the hand movement data stream within a second time period after the trigger signal is generated; If the trajectory starting segment exists and a time difference between a starting time of the trajectory starting segment and an activation time of the myoelectric signal data stream is within a preset interval, it is determined that the trajectory starting segment of the hand movement data stream and the myoelectric signal data stream meet a synchronization condition in time sequence, and a gesture interaction signal is triggered.

[0009] In an embodiment, the step of performing an interaction operation in response to the gesture interaction signal comprises: In response to the gesture interaction signal, spatiotemporal features are extracted from the hand movement data stream and the myoelectric signal data stream respectively, and the spatiotemporal features are input into a multi-modal classification model for fusion recognition to obtain a gesture type recognition result; If the gesture type recognition result is a meta-gesture, an interaction mode corresponding to the meta-gesture is switched to, wherein the interaction mode is a voice control mode, a device control mode, a gesture input mode or a cursor simulation mode; If the gesture type recognition result is a function gesture, an interaction operation corresponding to the function gesture is performed.

[0010] In an embodiment, the step of switching to the interaction mode corresponding to the meta-gesture comprises: If the meta-gesture is a first type of gesture, the interaction mode of the smart ring is switched to the voice control mode, and the smart ring is used to process voice instructions in the voice control mode; If the meta-gesture is a second type of gesture, the interaction mode of the smart ring is switched to the device control mode, and the smart ring is used to control external office equipment in the device control mode; If the meta-gesture is a third type of gesture, the interaction mode of the smart ring is switched to the gesture input mode, and the smart ring is used to execute an interaction instruction corresponding to a user gesture in the gesture input mode; If the meta-gesture is a fourth type of gesture, the interaction mode of the smart ring is switched to the cursor simulation mode, and the smart ring is used to generate a cursor control instruction in the cursor simulation mode.

[0011] In an embodiment, the step of performing an interaction operation corresponding to the function gesture if the gesture type recognition result is the function gesture comprises: if the gesture type recognition result is a function gesture, comparing the hand movement data stream with standard gesture trajectories of a current interaction mode of the smart ring in terms of similarity, and determining a confidence degree of the gesture type recognition result according to the similarity; obtaining context information of a current operation scene of the smart ring, and adjusting the confidence degree based on the context information; if the adjusted confidence degree exceeds a preset confidence degree threshold, performing an interaction operation corresponding to the function gesture.

[0012] In an embodiment, the step of adjusting the confidence degree based on the context information comprises: if the context information indicates that the smart ring has established a communication connection with an external device, increasing the confidence degree by a first preset value; if the context information indicates that the user's hand is in a sustained high-frequency motion state, decreasing the confidence degree by a second preset value; if the context information indicates that the user's hand has changed from a static state to a motion state, increasing the confidence degree by a third preset value.

[0013] In an embodiment, after the step of performing an interaction operation in response to the gesture interaction signal, the method further comprises: when a user-initiated revocation operation instruction is detected within a preset time window after the interaction operation is performed, combining the hand movement data stream, the electromyographic signal data stream, and the interaction operation as an error sample; optimizing gesture interaction decision logic of the smart ring based on the error sample.

[0014] In addition, to achieve the above-mentioned purposes, the application further provides a gesture interaction device applied to a smart ring, which comprises: a data stream acquisition module, configured to acquire a hand movement data stream of a user in real time through an inertial measurement unit of the smart ring, and acquire an electromyographic signal data stream of the user in real time through an electromyographic sensor of the smart ring; a signal triggering module, configured to trigger a gesture interaction signal when a signal feature of the electromyographic signal data stream meets a preset activation condition, and a trajectory starting segment of the hand movement data stream meets a synchronism condition in time sequence with the electromyographic signal data stream; an interaction operation execution module, configured to perform an interaction operation in response to the gesture interaction signal.

[0015] In addition, to achieve the above object, the application further provides a smart ring, which comprises a memory, a processor and a computer program stored in the memory and executable on the processor, and the computer program is configured to implement the steps of the gesture interaction method.

[0016] In addition, to achieve the above object, the application further provides a storage medium, which is a computer readable storage medium, and the storage medium stores a computer program, and the computer program is executed by a processor to implement the steps of the gesture interaction method.

[0017] The one or more technical solutions provided by the application have at least the following technical effects: In the application, in the office process of a user, a smart ring worn by the user collects a hand movement data stream of the user in real time through an inertial measurement unit of the smart ring, and collects an electromyographic signal data stream of the user in real time through an electromyographic sensor of the smart ring, and a specific judgment condition is set, that is, when a signal feature of the electromyographic signal data stream meets a preset activation condition and a trajectory starting segment of the hand movement data stream meets a synchronization condition in time sequence with the electromyographic signal data stream, a gesture interaction signal is triggered to perform an interaction operation. This comprehensive judgment mode can use the muscle activity state reflected by the electromyographic signal data stream in combination with the gesture trajectory information of the hand movement data stream to more accurately identify the gesture interaction intention of the user, and compared with a mode of simply relying on an IMU to detect a gesture trajectory, the recognition accuracy of the interaction intention of the user in an office scene is improved, and a better interaction experience is brought to the user. BRIEF DESCRIPTION OF DRAWINGS

[0018] The accompanying drawings, which are incorporated into and form a part of the specification, illustrate an embodiment consistent with the application and, together with the description, serve to explain the principles of the application.

[0019] In order to more clearly illustrate the technical solutions in the embodiments of the application or the prior art, the accompanying drawings needed to be used in the embodiments or prior art description will be briefly introduced as follows. Obviously, for those skilled in the art, the other drawings can also be obtained based on these drawings without any creative effort.

[0020] Figure 1 A flowchart is provided for the gesture interaction method embodiment one of the application; Figure 2 A smart ring structure diagram is provided for the gesture interaction method embodiment one of the application; Figure 3 A flowchart is provided for the gesture interaction method embodiment two of the application; Figure 4 A module structure diagram of the gesture interaction device of the embodiment of the application is provided. Figure 5 A device structure schematic diagram of a hardware running environment involved in a gesture interaction method in an embodiment of the present application.

[0021] Figure 2 Brief Description of the Drawings

[0022] The object implementation, functional features and advantages of the present application will be further described with reference to the embodiments and the accompanying drawings. DETAILED DESCRIPTION

[0023] It should be understood that the specific embodiments described herein are merely intended to explain the technical solutions of the present application, and are not intended to limit the present application.

[0024] In order to better understand the technical solutions of the present application, the specific embodiments will be described in detail below with reference to the drawings and the specific embodiments.

[0025] The existing smart rings and other smart wearable devices generally rely on inertial measurement units to detect hand movement trajectories in order to realize gesture interaction functions.

[0026] However, in a complex office scenario, there are a large number of unconscious daily activities of the user's hands, such as typing, organizing items, etc. It is difficult to effectively distinguish such unintentional actions from intentional control instruction gestures by relying only on IMU movement trajectory analysis, resulting in a high false trigger rate of the interaction system, poor reliability, and affecting the user experience.

[0027] In summary, how to improve the recognition accuracy of user interaction intent in an office scenario has become a technical problem that needs to be solved in the field.

[0028] To solve the above technical problems, in an embodiment of the present application, during the user's office process, the smart ring worn by the user collects the user's hand movement data stream in real time through the inertial measurement unit of the smart ring, and collects the user's electromyographic signal data stream in real time through the electromyographic sensor of the smart ring, and sets a specific judgment condition, i.e. the signal characteristics of the electromyographic signal data stream meet the preset activation condition, and the trajectory starting segment of the hand movement data stream and the electromyographic signal data stream meet the synchronization condition in time sequence, only then trigger the gesture interaction signal to execute the interaction operation. This comprehensive judgment method can use the user's muscle activity state reflected by the electromyographic signal data stream, combined with the gesture trajectory information of the hand movement data stream, to more accurately identify the user's gesture interaction intent, compared with the method of simply relying on IMU to detect gesture trajectory, improve the recognition accuracy of user interaction intent in an office scenario, and bring better interaction experience to the user.

[0029] It should be noted that the execution subject of the embodiment can be an intelligent ring with data processing, network communication and program running functions. The following will take the intelligent ring as an example to describe the embodiment and the following embodiments.

[0030] The first embodiment of the gesture interaction method of the present application is proposed below. Referring to Figure 1 , Figure 1 The flowchart of the first embodiment of the gesture interaction method of the present application is shown in the figure.

[0031] In the embodiment, the gesture interaction method comprises steps S10-S30: Step S10, real-time collection of hand motion data stream of the user through the inertial measurement unit of the intelligent ring, and real-time collection of electromyographic signal data stream of the user through the electromyographic sensor of the intelligent ring; It should be noted that the hand motion data stream refers to the time series data continuously collected by the inertial measurement unit built in the intelligent ring. The inertial measurement unit usually contains an accelerometer and a gyroscope, which can capture the linear acceleration and angular velocity change of the user's hand in three-dimensional space wearing the intelligent ring at a high frequency. The electromyographic signal data stream is a microvolt-level electrophysiological signal collected by the electromyographic sensor built in the intelligent ring and contacting the surface of the user's skin, which reflects the bioelectric activity generated by the muscle fibers of the fingers when they contract and relax.

[0032] In the actual implementation process, the hand motion data stream and the electromyographic signal data stream are synchronously collected through the inertial measurement unit and the electromyographic sensor of the intelligent ring.

[0033] Step S20, when the signal characteristics of the electromyographic signal data stream meet the preset activation condition, and the trajectory starting segment of the hand motion data stream meets the synchronicity condition with the electromyographic signal data stream in time sequence, the gesture interaction signal is triggered; It should be noted that the signal characteristics of the electromyographic signal data stream can include the amplitude, frequency or short-time energy of the electromyographic signal and other parameters. The activation condition refers to the preset determination threshold, for example, the short-time energy of the electromyographic signal exceeds a certain set value. The trajectory starting segment of the hand motion data stream refers to the initial stage data from the hand static or random motion state to the initial stage data of the intentional and directional motion. The synchronicity condition refers to the activation time of the electromyographic signal and the starting time of the motion trajectory need to be highly close in the time axis, and the time difference should be within a preset reasonable interval, so as to ensure that the muscle force intention and the actual limb motion are for the same action.

[0034] The electromyogram signal data stream is monitored in real time, and when a signal feature of the electromyogram signal data stream meets a preset activation condition, a time difference between an electromyogram signal activation time and a trajectory starting time in the hand movement data stream is calculated. If the time difference between the electromyogram signal activation time and the trajectory starting time is within a preset interval, that is, a trajectory starting segment of the hand movement data stream and the electromyogram signal data stream meet a synchronization condition in time sequence, a gesture interaction signal is triggered, and the gesture interaction signal represents that a gesture recognition step is started.

[0035] In a feasible embodiment, the step S20 can include steps S201-S204: In step S201, the short-time energy of the electromyogram signal data stream is calculated in real time. The short-time energy of the electromyogram signal data stream is calculated in real time. The short-time energy of the electromyogram signal data stream refers to a sum of squares or absolute values of electromyogram signal amplitudes in a very short time window, for example, tens to hundreds of milliseconds, which can reflect the intensity change of the electromyogram signal in a short time.

[0036] In step S202, when a time length during which the short-time energy exceeds a first threshold value reaches a first time period, it is determined that a signal feature of the electromyogram signal data stream meets a preset activation condition, and a trigger signal is generated. The electromyogram signals continuously collected by the electromyogram sensor are frame-processed, and the short-time energy of each frame of data is calculated in real time. When a time length during which the short-time energy exceeds a first threshold value reaches a first time period, it is determined that a signal feature of the electromyogram signal data stream meets a preset activation condition, and a trigger signal is generated. The trigger signal indicates that the gesture interaction intention of the user has been preliminarily acquired. The first threshold value is an energy threshold value calibrated through experiments, which is used to filter out background noise and weak muscle tremors. The first time period is, for example, 100-200 milliseconds, so as to exclude some accidental and non-continuous muscle electrical activities and further improve the reliability of the activation determination.

[0037] In step S203, within a second time period after the trigger signal is generated, whether a trajectory starting segment meeting a preset gesture starting feature exists in the hand movement data stream is analyzed. In step S203, within a second time period after the trigger signal is generated, whether a trajectory starting segment meeting a preset gesture starting feature exists in the hand movement data stream is analyzed.

[0038] Step S204, if the trajectory starting segment exists and the time difference between the starting time of the trajectory starting segment and the activation time of the myoelectric signal data stream is within the preset interval, it is determined that the trajectory starting segment of the hand movement data stream and the myoelectric signal data stream satisfy the synchronization condition in time sequence, and a gesture interaction signal is triggered.

[0039] It should be noted that the activation time of the myoelectric signal data stream refers to the time when the short-time energy of the myoelectric signal first exceeds the first threshold, and the preset interval is usually set to a very small range, for example, 100 milliseconds, which conforms to the physiological characteristics of human neuromuscular control, that is, there is a short and relatively fixed delay from the brain sending instructions to muscle contraction to the limbs producing obvious movement.

[0040] If the trajectory starting segment exists and the time difference between the starting time of the trajectory starting segment and the activation time of the myoelectric signal data stream is within the preset interval, it is determined that the trajectory starting segment of the hand movement data stream and the myoelectric signal data stream satisfy the synchronization condition in time sequence, and a gesture interaction signal is triggered. By strictly limiting the time difference between the starting time of the trajectory starting segment and the activation time of the myoelectric signal data stream, it can be ensured that the activation of the myoelectric signal and the movement of the limbs are derived from the same neural instruction, thereby distinguishing the truly intended interaction gesture from those irrelevant muscle activities and limb movements that happen to occur at the same time, thereby improving the accuracy of triggering.

[0041] Step S30, in response to the gesture interaction signal, an interaction operation is performed.

[0042] After confirming that the gesture interaction signal is valid, an interaction operation corresponding to the user's gesture is generated and performed in response to the gesture interaction signal, for example, switching the interaction mode of the smart ring in response to the gesture interaction signal, or controlling the external device to display the slide page, adjusting the external device to play the volume, simulating the mouse click, etc. The external device is a device that establishes a communication connection through the built-in Bluetooth, radio frequency or infrared communication module of the ring, such as a computer, a projector, etc.

[0043] In addition, for the smart ring configured with a micro vibration motor, vibration feedback can also be provided while performing the interaction operation to inform the user that the instruction has been successfully received and processed.

[0044] Therefore, in the user's office process, the smart ring worn by the user collects the hand movement data stream of the user in real time through the inertial measurement unit of the smart ring, and collects the electromyographic signal data stream of the user in real time through the electromyographic sensor of the smart ring, and sets a specific judgment condition, that is, the signal characteristics of the electromyographic signal data stream satisfy the preset activation condition, and the trajectory starting segment of the hand movement data stream and the electromyographic signal data stream satisfy the synchronization condition in time sequence, and then the gesture interaction signal is triggered to execute the interaction operation. This comprehensive judgment method can use the muscle activity state reflected by the electromyographic signal data stream, combine the gesture trajectory information of the hand movement data stream, more accurately recognize the gesture interaction intention of the user, and compared with the method of simply relying on the IMU to detect the gesture trajectory, the recognition accuracy of the user's interaction intention in the office scene is improved, and a better interaction experience is brought to the user.

[0045] It is worth mentioning that the structure diagram of the smart ring provided in the embodiment is as shown in Figure 2 The smart ring adopts a modular structure design, mainly including a ring main body 01, a ring circuit board assembly 02 and various functional modules 03. The ring main body 01 constitutes the basic framework of the entire smart ring, and a magnetic element 04 is arranged inside the ring main body 01, which is used to realize physical connection and circuit conduction with the functional modules 03 through magnetic attraction. The ring circuit board assembly 02 is integrated with a flexible circuit board 05, a battery 06, a magnetic element 07, an antenna and a touch FPC 08, a wireless charging coil 09, a Pogo pin 10 and a sensor window 11. The wireless charging coil 09 is used to charge the built-in battery 06 through electromagnetic induction, and the antenna is responsible for the transmission of wireless data such as Bluetooth. The ring circuit board assembly 02 and the ring body 01 can be fixedly connected through a plurality of module fixing frames 12. The functional module 03 is a detachable and replaceable independent unit, which can be selected and replaced according to the needs of the motion scene. The magnetic element 13, the sensor contact 14 and the conduction contact 15 are also integrated inside each functional module. The magnetic element 13 is used for magnetic attraction connection with the ring main body 01.

[0046] For example, according to the actual application scene requirement, the functional modules that can be assembled on the smart ring include a miniature microphone, an infrared / laser sensor, a radio frequency chip, a Bluetooth chip, an accelerometer, a gyroscope, an electromyographic sensor, a miniature optical trackball, a miniature vibration motor and the like.

[0047] Based on the first embodiment of the present application, in the second embodiment of the present application, the same or similar contents as the above embodiment one can refer to the above introduction, and the following will not be described in detail. On this basis, as shown in Figure 3 Step S30 can include steps S301-S303: Step S301, in response to the gesture interaction signal, spatiotemporal features are extracted from the hand motion data stream and the electromyographic signal data stream respectively, and the spatiotemporal features are input into a multi-modal classification model for fusion recognition to obtain a gesture type recognition result; It should be noted that the multi-modal classification model can be a lightweight model running on the local microcontroller of the smart ring, such as a decision tree or a support vector machine, or a high-performance deep learning model running on a third-party device or cloud in communication connection with the smart ring, such as a convolutional neural network or a time recurrent neural network. The task of the model is to map the input spatiotemporal features to a specific gesture class label, such as double finger pinch or left swing, and also includes attribute classification, i.e., determining whether the gesture belongs to a meta-gesture for switching interaction modes or a functional gesture for performing specific tasks in a specific interaction mode.

[0048] After determining that the gesture interaction signal is triggered, the user's gesture intention needs to be understood. A time window of a specific length is intercepted before and after the time point of triggering the gesture interaction signal. In this time window, for the hand motion data stream, spatiotemporal features that can represent the motion trajectory and posture are extracted from the acceleration and angular velocity data collected by the inertial measurement unit, including but not limited to the direction cosine sequence of the motion trajectory in three-dimensional space, the peak value and distribution statistics of velocity and acceleration, the attitude angle change curve obtained by integrating the gyroscope data, and the motion frequency energy distribution obtained by fast Fourier transform. Similarly, in this time window, for the electromyographic signal data stream, spatiotemporal features that can reflect the muscle activity pattern and intensity are extracted from the bioelectric signals collected by the electromyographic sensor, including but not limited to the short-time average absolute value, variance, zero-crossing rate, wavelength of the signal, and the time series spectrum features extracted by the Mel frequency cepstrum coefficient or the time-frequency domain joint features obtained by wavelet transform.

[0049] The extracted spatiotemporal features are combined into a comprehensive feature vector, which is input into a pre-trained multi-modal classification model to obtain a gesture type recognition result. The structure of the model can be a deep learning model that combines convolutional neural networks and long short-term memory networks to capture both spatial features and time sequence dependencies of gestures; or a multi-modal fusion model based on attention mechanism, which dynamically weights the contribution of different sensor modalities and different time point features to the final classification.

[0050] Step S302, if the gesture type recognition result is a meta-gesture, switch to an interaction mode corresponding to the meta-gesture, wherein the interaction mode is a voice control mode, a device control mode, a gesture input mode, or a cursor simulation mode; If the gesture type recognition result is a meta-gesture, the function of this gesture is to switch the interaction mode of the smart ring. A predefined mapping table of meta-gestures and interaction modes can be queried, and based on the specific meta-gesture recognized, the interaction mode of the ring is switched to the interaction mode corresponding to the gesture. The interaction mode can be a voice control mode for processing voice commands, a device control mode for remotely controlling external office equipment, a gesture input mode for taking gestures as input instructions, and a cursor simulation mode for mapping gestures to cursor movement and control, etc.

[0051] In a feasible embodiment, step S302 can include steps S3021-S3024: Step S3021, if the meta-gesture is a first type of gesture, the interaction mode of the smart ring is switched to a voice control mode, and the smart ring is used to process voice instructions in the voice control mode; It should be noted that a mapping table of explicit gesture types and interaction modes is maintained in the storage area of the smart ring. When the multi-modal classification model outputs the recognition result, mode matching is immediately performed according to this table.

[0052] When the recognition result is classified as a first type of gesture, for example, the user quickly and continuously clicks the middle finger tip twice with the thumb, the smart ring is immediately switched to a voice control mode. In this mode, the hardware resource configuration and software logic of the ring will serve this mode: the miniature microphone in the smart ring is activated and starts high-fidelity recording, the collected audio data stream is real-time stream transferred to a third-party device paired with it through a Bluetooth connection or directly uploaded to a cloud server, the voice recognition engine in the third-party device or the cloud server converts the audio into text, and then further analyzes the text content. If the voice recognition result is a control instruction such as the next slide, a corresponding control signal is generated and sent back to the ring or directly controls the external device. If it is judged as ordinary declarative language, it is saved as a text record in the designated meeting minutes file.

[0053] Step S3022, if the meta-gesture is a second type of gesture, the interaction mode of the smart ring is switched to a device control mode, and the smart ring is used to control external office equipment in the device control mode; If the recognition result is a second type of gesture, for example, the user rotates the wrist clockwise after making a fist, the device control mode is switched to. In this mode, the ring will activate its infrared emitter, laser pointer, or enhanced Bluetooth control channel. The ring is ready to receive and interpret subsequent user gestures and convert them into standard device control commands, such as generating infrared remote control codes for turning on or off a projector, or sending control media volume, slide page turning instructions to a computer through Bluetooth HID protocol.

[0054] Step S3023, if the meta-gesture is the third type of gesture, the interaction mode of the smart ring is switched to a gesture input mode, and the smart ring is used to execute an interaction instruction corresponding to the user gesture in the gesture input mode; If the recognition result is the third type of gesture, for example, the index finger draws a hook in the air, the gesture input mode is entered, this mode focuses on directly mapping the gesture symbol made by the user into a system command, by directly calling a special gesture recognition sub-process, the hand movement data and the electromyographic signal data are compared at a higher frequency and accuracy, to accurately recognize various symbolic gestures, such as confirmation, cancellation, deletion, saving, etc., once a match is successful, an interaction instruction corresponding to the recognized user gesture is executed, and the user gesture can be a meta-gesture or a function gesture.

[0055] Step S3024, if the meta-gesture is the fourth type of gesture, the interaction mode of the smart ring is switched to a cursor simulation mode, and the smart ring is used to generate a cursor control instruction in the cursor simulation mode.

[0056] If the recognition result is the fourth type of gesture, for example, the thumb and the middle finger are pinched and kept still for a preset time length (such as 2 seconds), the cursor simulation mode is started, and a trackball sub-process is called to generate a cursor control instruction in the cursor simulation mode. Specifically, in the cursor simulation mode, the micro optical trackball built in the smart ring starts to work, and the displacement data generated by rubbing the micro optical trackball with the fingers is converted into cursor movement coordinates on the screen or a scroll wheel signal. At the same time, the electromyographic sensor is used to detect the pinching action of the fingers to simulate the single-click or double-click event of the mouse, so as to realize the complete mouse simulation function.

[0057] Therefore, the user does not need to switch the interaction function of the ring through a tedious button or voice menu, but directly wakes up the required function mode by performing different and natural gestures, so that the interaction process becomes shorter and more direct, reduces the operation steps of the user, and enables the smart ring to instantly replace a remote controller, a recording pen, an air mouse or a command inputter in a complex office scene, thereby improving the user experience.

[0058] In addition, in an embodiment, the smart ring can work in two or more interactive modes in parallel to achieve a more efficient and natural interactive experience. For example, the user can continuously narrate or issue instructions through the voice control mode while performing a function gesture in the device control mode. That is, the microphone of the smart ring as the input device of the voice control mode will always be in a high-sensitivity recording state and will analyze and process the collected audio data stream. At the same time, the inertial measurement unit and the myoelectric sensor of the smart ring focus on capturing the hand movement trajectory and force characteristics for device control. The smart ring can process two independent input streams in parallel: one sends the audio to the speech recognition engine, and the other recognizes the gesture data and generates device control instructions. This means that the user can achieve a seamless experience of speaking while operating without any mode switching, for example, stating the report content while controlling the slide progress with gestures and marking highlights or recording inspirations through voice instructions.

[0059] Therefore, the smart ring in the embodiment allows the user to freely combine multiple interactive modes according to the needs of the current task, which is more in line with human natural communication and behavior habits and facilitates quick completion of various interactive operations in an office environment.

[0060] In step S303, if the gesture type recognition result is a function gesture, an interactive operation corresponding to the function gesture is performed.

[0061] If the gesture type recognition result is a function gesture, it indicates that the user intends to perform a specific interactive operation in the currently activated interactive mode. Specifically, it is first necessary to confirm that the function gesture is compatible with the current interactive mode. For example, a left waving gesture is a valid function gesture in the device control mode, but it can be ignored in the voice control mode. If it is determined that the function gesture is compatible with the current interactive mode, an interactive operation corresponding to the function gesture is directly performed. For example, in the device control mode, a left waving gesture is recognized, and a command to play a song or advance a slide is immediately sent to the connected external device. In the cursor simulation mode, a function gesture of pinching the thumb and index finger twice is mapped to a mouse left double-click event.

[0062] In an embodiment, step S303 can include steps S3031-S3033: In step S3031, if the gesture type recognition result is a function gesture, the hand movement data stream is compared with the standard gesture trajectories of the current interactive mode of the smart ring in terms of similarity, and the confidence of the gesture type recognition result is determined according to the similarity. After determining that the gesture type recognition result is a function gesture, a motion trajectory starting from the current interaction trigger point is extracted from the current hand motion data stream, and the motion trajectory is compared with all standard gesture trajectory templates in the current interaction mode of the smart ring. For example, if the current mode is a device control mode, the standard gesture trajectory templates may include left swing, right swing, up push, down press, and the like. The comparison algorithm can use Dynamic Time Warping (DTW) and other algorithms that can overcome the difference in motion speed to calculate the similarity score between the current trajectory and each standard template. The highest score is taken as the initial confidence of the recognition result to ensure that the gesture recognition is based on the most relevant standard motion in the current interaction mode.

[0063] In step S3032, context information of the current operation scene of the smart ring is obtained, and the confidence is adjusted based on the context information. To compensate for the deficiency of simple trajectory comparison, the context information of the current operation scene of the smart ring is obtained in parallel. The context information is a set of multi-dimensional parameters, which aims to reflect the authenticity probability of the user's interaction intention. It can come from the hand motion data stream collected by the inertial measurement unit, the electromyographic signal data stream collected by the electromyographic sensor, or the Bluetooth connection state within a predetermined time range. Based on these context information, the confidence is adjusted according to the preset rules.

[0064] In a feasible embodiment, the step of "adjusting the confidence based on the context information" in step S3032 can include steps a10-a30: Step a10, if the context information indicates that the smart ring has established a communication connection with the external device, the confidence is adjusted according to a first preset value; When it is detected that the context information indicates that the smart ring has established a communication connection with the external device, the confidence is adjusted according to a first preset value (such as 10% of the current confidence). This is because a stable connection state constitutes the basis for the feasibility of interaction, which implies that the user is in an environment that is ready and expects human-computer interaction. Therefore, the probability of intentional gesture at this time is much higher than unintentional trigger, and the system should give a higher trust weight.

[0065] Step a20, if the context information indicates that the user's hand is in a sustained high-frequency motion state, the confidence is adjusted according to a second preset value; When the context information indicates that the user's hand is in a sustained high-frequency motion state, the confidence level is adjusted downward by a second preset value (e.g., 5% of the current confidence level). This is because such a context usually corresponds to the user being in the process of typing, frequently using gesture to assist in speech, or in a mobile environment, where the hand background activity noise level is high, and in such an environment, the probability of a piece of data stream that coincidentally matches some feature of a functional gesture is greatly increased.

[0066] Step a30, if the context information indicates that the user's hand is in a motion state from a stationary state, the confidence level is adjusted upward by a third preset value.

[0067] When the context information indicates that the user's hand is in a motion state from a stationary state, the confidence level is adjusted upward by a third preset value (e.g., 15% of the current confidence level). This is because the transition from a long period of inactivity to a sudden motion is one of the most typical and strongest behavioral signals that the user consciously initiates an interaction, and therefore, a greater reward of trust is given to this recognition result, ensuring that the user's explicit interaction intention can be quickly and accurately responded to.

[0068] Step S3033, if the adjusted confidence level exceeds a preset confidence threshold, an interactive operation corresponding to the functional gesture is performed.

[0069] The final confidence level value, which is a combination of the trajectory similarity and the context information, is compared with a confidence threshold that is calibrated in advance through experiments. When the adjusted confidence level exceeds this threshold, it is confirmed that this functional gesture is the user's true intention, and then an interactive operation corresponding to the functional gesture is performed. If the confidence level is insufficient, the operation is abandoned, and an error feedback can be provided through a micro-vibration motor to prompt the user that the operation is not recognized.

[0070] Thus, by combining gesture trajectory matching with context understanding based on the scene, a more reliable interactive experience is provided for the user.

[0071] In a feasible embodiment, after step S30, steps S40-S50 can also be included: Step S40, when a user-initiated undo operation instruction is detected within a preset time window after the interactive operation is performed, the hand motion data stream, the electromyographic signal data stream, and the interactive operation are combined as an error sample. After performing any one interaction operation, within a next preset time window (for example, 5 seconds), the monitoring of the user instruction is maintained, if the user initiates a cancel operation instruction in any form within the preset time window is detected, it is judged that the just performed interaction operation is likely to be a misrecognition, and the real intention of the user cannot be reflected, at this time, the complete hand motion data stream, the electromyographic signal data stream and the executed error interaction operation of this interaction are associated and bound, and are labeled as an error sample, and are stored in the storage area of the ring or the third party device.

[0072] In step S50, the gesture interaction decision logic of the smart ring is optimized based on the error sample.

[0073] The gesture interaction decision logic of the smart ring is optimized based on one or more accumulated error samples. This optimization can be multi-level, for example, for the threshold determination link, a statistical method can be used to fine-tune the threshold; for the multi-modal classification model, the error sample is added to the training set of the model, and incremental learning or regular model retraining is performed. Through this training process, the model can learn which similar sensor data patterns will lead to an error output, and when similar patterns are encountered in the future, the model will avoid or reduce the confidence, thereby gradually improving the recognition accuracy.

[0074] Thus, the user's immediate feedback is used as a supervisory signal to automatically build an optimized data set for real use scenarios, so that the ring can continuously adapt to the unique gesture habits of a specific user, correct gesture recognition bias, and provide the user with better and better use experience.

[0075] It should be noted that the above examples are only used to understand the present application and do not limit the gesture interaction method of the present application. More simple transformations based on this technical concept are within the protection scope of the present application.

[0076] The present application also provides a gesture interaction device, please refer to Figure 4 The gesture interaction device comprises: A data stream acquisition module 10 is configured to acquire a hand motion data stream of a user in real time through an inertial measurement unit of the smart ring, and acquire an electromyographic signal data stream of the user in real time through an electromyographic sensor of the smart ring. A signal triggering module 20 is configured to trigger a gesture interaction signal when a signal feature of the electromyographic signal data stream meets a preset activation condition, and a trajectory starting segment of the hand motion data stream meets a synchronization condition in time sequence with the electromyographic signal data stream. An interaction operation execution module 30 is configured to execute an interaction operation in response to the gesture interaction signal.

[0077] Optionally, the signal triggering module 20 is further configured to: calculating a short-time energy of the electromyography signal data stream in real time; determining that the signal feature of the electromyography signal data stream meets a preset activation condition and generating a trigger signal when the duration that the short-time energy exceeds the first threshold value reaches a first time period; analyzing whether a trajectory starting segment meeting a preset gesture starting feature exists in the hand movement data stream within a second time period after the trigger signal is generated; if the trajectory starting segment exists and a time difference between a starting time of the trajectory starting segment and an activation time of the electromyography signal data stream is within a preset interval, determining that the trajectory starting segment of the hand movement data stream and the electromyography signal data stream meet a synchronization condition in time sequence, and triggering a gesture interaction signal.

[0078] Optionally, the interaction operation execution module 30 is further configured to: in response to the gesture interaction signal, extracting spatiotemporal features from the hand movement data stream and the electromyography signal data stream respectively, and inputting the spatiotemporal features into a multi-modal classification model for fusion recognition to obtain a gesture type recognition result; if the gesture type recognition result is a meta gesture, switching to an interaction mode corresponding to the meta gesture, wherein the interaction mode is a voice control mode, a device control mode, a gesture input mode or a cursor simulation mode; if the gesture type recognition result is a function gesture, executing an interaction operation corresponding to the function gesture.

[0079] Optionally, the interaction operation execution module 30 is further configured to: if the meta gesture is a first type of gesture, switching the interaction mode of the smart ring to the voice control mode, and the smart ring is used for processing voice instructions in the voice control mode; if the meta gesture is a second type of gesture, switching the interaction mode of the smart ring to the device control mode, and the smart ring is used for controlling external office equipment in the device control mode; if the meta gesture is a third type of gesture, switching the interaction mode of the smart ring to the gesture input mode, and the smart ring is used for executing an interaction instruction corresponding to a user gesture in the gesture input mode; if the meta gesture is a fourth type of gesture, switching the interaction mode of the smart ring to the cursor simulation mode, and the smart ring is used for generating a cursor control instruction in the cursor simulation mode.

[0080] Optionally, the interaction operation execution module 30 is further configured to: if the gesture type recognition result is a function gesture, comparing the hand movement data stream with each standard gesture trajectory of the current interaction mode of the smart ring in terms of similarity, and determining a confidence degree of the gesture type recognition result according to the similarity; Contextual information of a current operation scenario of the smart ring is acquired, and the confidence is adjusted based on the contextual information; If the adjusted confidence exceeds a preset confidence threshold, an interactive operation corresponding to the function gesture is performed.

[0081] Optionally, the interactive operation execution module 30 is further configured to: If the contextual information indicates that the smart ring has established a communication connection with an external device, the confidence is adjusted higher according to a first preset value; If the contextual information indicates that the user's hand is in a sustained high-frequency motion state, the confidence is adjusted lower according to a second preset value; If the contextual information indicates that the user's hand is in a sustained high-frequency motion state, the confidence is adjusted lower according to a second preset value;

[0082] Optionally, the gesture interaction device further comprises an interaction logic optimization module (not shown), which is configured to: When a user-initiated cancel operation instruction is detected within a preset time window after the interactive operation is performed, the hand motion data stream, the electromyographic signal data stream, and the interactive operation are combined as an error sample; The gesture interaction decision logic of the smart ring is optimized based on the error sample.

[0083] The embodiments of the present application provide a smart ring, which comprises at least one processor, and a memory connected with the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the state prompting method in the above embodiments.

[0084] Reference will be made to the following description Figure 5 which shows a schematic diagram of a smart ring suitable for implementing the embodiments of the present application. Figure 5 The illustrated smart ring is merely an example and should not impose any limitation on the functions and use range of the embodiments of the present application.

[0085] As Figure 5As shown, the smart ring can include a processing device 1001 (e.g., a DSP processor or the like) that can perform various appropriate actions and processes according to programs stored in a read-only memory 1002 or loaded from a storage device 1003 into a random access memory 1004. Various programs and data required for the operation of the smart ring are also stored in the random access memory 1004. The processing device 1001, the read-only memory 1002, and the random access memory 1004 are connected to each other by a bus 1005. An input / output interface 1006 is also connected to the bus. Generally, the following systems can be connected to the input / output interface 1006: an input device 1007 including, for example, a microphone, an accelerometer, or the like; an output device 1008 including, for example, a speaker, a vibrator, or the like; the storage device 1003 including, for example, a tape, a hard disk, or the like; and a communication device 1009. The communication device 1009 can allow the smart ring to communicate wirelessly or by wire with other devices to exchange data. Although the smart ring is shown as having various systems, it should be understood that all of the shown systems are not required to be implemented or possessed. More or fewer systems can be alternatively implemented or possessed.

[0086] In particular, the processes described above with reference to the flowcharts can be implemented as a computer software program according to embodiments of the present disclosure. For example, embodiments of the present disclosure include a computer program product comprising a computer program carried on a computer readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network by a communication device, or installed from the storage device 1003, or installed from the read-only memory 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the methods of embodiments of the present disclosure are performed.

[0087] Compared with the prior art, the smart ring provided by embodiments of the present disclosure has the same beneficial effects as the gesture interaction method provided by the above-mentioned embodiments, and other technical features in the smart ring are the same as the features disclosed in the last embodiment method, which will not be described here.

[0088] It should be understood that parts disclosed in embodiments of the present disclosure can be realized by hardware, software, firmware, or a combination thereof. In the description of the above-described embodiments, specific features, structures, materials, or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0089] The above merely provides a specific implementation of the present application, but the protection scope of the present application is not limited thereto, and any person skilled in the art can easily think of changes or replacements within the technical scope disclosed by the present application, which should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

[0090] The embodiment of the present application provides a computer readable storage medium having computer readable program instructions (i.e. computer programs) stored thereon, the computer readable program instructions being used to execute the gesture interaction method in the above embodiment.

[0091] The computer readable storage medium provided by the embodiment of the present application may, for example, be a U disk, but is not limited to an electric, magnetic, optical, electromagnetic, infrared or semiconductor system or device, or any combination of the above. More specific examples of the computer readable storage medium can include, but are not limited to: an electric connection having one or more conductive wires, a portable computer disk, a hard disk, a random access memory (RAM), a read only memory (ROM), an erasable programmable read only memory (EPROM or flash memory), an optical fiber, a portable compact disk read only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the embodiment, the computer readable storage medium can be any tangible medium containing or storing a program, which can be used by or in combination with an instruction execution system or device. The program code contained on the computer readable storage medium can be transmitted by any suitable medium, including but not limited to: an electric wire, an optical cable, an RF (Radio Frequency), etc., or any suitable combination of the above.

[0092] The above computer readable storage medium can be contained in the smart ring; or can exist separately and not be assembled into the smart ring.

[0093] The above computer readable storage medium carries one or more programs, when the one or more programs are executed by the smart ring, the smart ring executes the above functions defined in the method of the embodiments disclosed by the present application.

[0094] Computer program code for carrying out operations of the present application can be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C++ or the like, and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider).

[0095] The computer program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other devices to cause a series of operational steps to be performed on the computer, other programmable apparatus or other devices to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide processes for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0096] The modules involved in the embodiments of the present application can be implemented in software or hardware. In some cases, the names of the modules do not limit the modules themselves.

[0097] The readable storage medium provided by the embodiments of the present application is a computer readable storage medium, and the computer readable storage medium stores computer readable program instructions (i.e., computer programs) for executing the gesture interaction method. Compared with the prior art, the computer readable storage medium provided by the embodiments of the present application has the same beneficial effects as the gesture interaction method provided by the above embodiments, and will not be described here.

[0098] The embodiment of the present application further provides a computer program product comprising a computer program, which, when executed by a processor, implements the steps of the state prompting method as described above. Compared with the prior art, the computer program product provided by the embodiment of the present application has the same beneficial effects as the state prompting method provided by the above-described embodiment, and thus is not described herein again.

[0099] The above are only some embodiments of the present application, and do not limit the patent scope of the present application. Any equivalent structural transformation made by using the content of the present application specification and drawings, or direct / indirect application in other related technical fields under the technical concept of the present application is included in the patent protection scope of the present application.

Claims

1. A gesture interaction method, characterized in that, Applied to a smart ring, the gesture interaction method includes: The smart ring uses an inertial measurement unit to collect real-time data streams of the user's hand movements and an electromyography (EMG) sensor to collect real-time data streams of the user's electromyography (EMG) signals. When the signal characteristics of the electromyography signal data stream meet the preset activation conditions, and the starting segment of the trajectory of the hand movement data stream and the electromyography signal data stream meet the synchronization condition in time, a gesture interaction signal is triggered. In response to the gesture interaction signal, perform the interaction operation.

2. The gesture interaction method as described in claim 1, characterized in that, The step of triggering a gesture interaction signal when the signal characteristics of the electromyographic signal data stream meet a preset activation condition, and the starting segment of the trajectory of the hand movement data stream and the electromyographic signal data stream satisfy a synchronization condition in time, includes: The short-time energy of the electromyographic signal data stream is calculated in real time; When the duration during which the short-term energy exceeds the first threshold reaches a first time period, it is determined that the signal characteristics of the electromyographic signal data stream meet the preset activation conditions, and a trigger signal is generated. During the second time period after the trigger signal is generated, analyze whether there is a trajectory start segment in the hand motion data stream that conforms to the preset gesture start characteristics; If the trajectory start segment exists, and the time difference between the start time of the trajectory start segment and the activation time of the electromyographic signal data stream is within a preset range, then it is determined that the trajectory start segment of the hand movement data stream and the electromyographic signal data stream satisfy the synchronization condition in time, and the gesture interaction signal is triggered.

3. The gesture interaction method as described in claim 1, characterized in that, The step of performing an interactive operation in response to the gesture interaction signal includes: In response to the gesture interaction signal, spatiotemporal features are extracted from the hand motion data stream and the electromyography signal data stream, respectively, and the spatiotemporal features are input into a multimodal classification model for fusion recognition to obtain the gesture type recognition result; If the gesture type recognition result is a meta-gesture, then switch to the interaction mode corresponding to the meta-gesture, wherein the interaction mode is a voice control mode, a device control mode, a gesture input mode, or a cursor simulation mode; If the gesture type recognition result is a functional gesture, then the interactive operation corresponding to the functional gesture is executed.

4. The gesture interaction method as described in claim 3, characterized in that, The step of switching to the interaction mode corresponding to the meta gesture includes: If the meta-gesture is a first type of gesture, then the interaction mode of the smart ring is switched to the voice control mode, and the smart ring is used to process voice commands in the voice control mode. If the meta-gesture is a second type of gesture, then the interaction mode of the smart ring is switched to the device control mode, and the smart ring is used to control external office equipment in the device control mode; If the meta-gesture is a third type of gesture, then the interaction mode of the smart ring is switched to the gesture input mode, and the smart ring is used to execute the interaction command corresponding to the user's gesture in the gesture input mode; If the meta-gesture is a fourth type of gesture, then the interaction mode of the smart ring is switched to the cursor simulation mode, and the smart ring is used to generate cursor control commands in the cursor simulation mode.

5. The gesture interaction method as described in claim 3, characterized in that, The step of performing the interactive operation corresponding to the functional gesture if the gesture type recognition result is a functional gesture includes: If the gesture type recognition result is a functional gesture, then the hand motion data stream is compared with the standard gesture trajectories of the current interaction mode of the smart ring, and the confidence level of the gesture type recognition result is determined based on the similarity. Obtain the context information of the current operating scenario of the smart ring, and adjust the confidence level based on the context information; If the adjusted confidence level exceeds the preset confidence level threshold, then the interactive operation corresponding to the functional gesture is executed.

6. The gesture interaction method as described in claim 5, characterized in that, The step of adjusting the confidence level based on the context information includes: If the context information indicates that the smart ring has established a communication connection with an external device, then the confidence level is increased according to the first preset value; If the context information indicates that the user's hand is in a continuous motion state, then the confidence level is lowered according to the second preset value; If the context information indicates that the user's hand has changed from a static state to an active state, then the confidence level is increased according to the third preset value.

7. The gesture interaction method as described in any one of claims 1 to 6, characterized in that, After the step of performing an interactive operation in response to the gesture interaction signal, the method further includes: If a user-initiated undo command is detected within a preset time window after the interactive operation is performed, the combination of the hand motion data stream, the electromyography signal data stream, and the interactive operation is used as an error sample. The gesture interaction decision logic of the smart ring is optimized based on the error samples.

8. A gesture interaction device, characterized in that, Applied to a smart ring, the gesture interaction device includes: The data stream acquisition module is used to acquire the user's hand movement data stream in real time through the inertial measurement unit of the smart ring, and to acquire the user's electromyographic signal data stream in real time through the electromyographic sensor of the smart ring. The signal triggering module is used to trigger a gesture interaction signal when the signal characteristics of the electromyographic signal data stream meet the preset activation conditions, and when the trajectory start segment of the hand movement data stream and the electromyographic signal data stream meet the synchronization condition in time. An interactive operation execution module is used to perform interactive operations in response to the gesture interaction signal.

9. A smart ring, characterized in that, The smart ring includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the gesture interaction method as described in any one of claims 1 to 7.

10. A storage medium, characterized in that, The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, it implements the steps of the gesture interaction method as described in any one of claims 1 to 7.

Images