Intelligent ring system oriented to XR interaction scene and interaction method

Abstract

The invention discloses an intelligent ring system facing an XR interaction scene and an interaction method. The system comprises an intelligent ring and a head display device, and the head display device is used for tracking the head rotation angle of a user and determining an operation target area; the intelligent ring is used for identifying the gesture type of a user based on the conductivity change information of the finger and the spatial motion data of the finger; the head-mounted display device executes a corresponding operation instruction for the target area in an application scene according to a corresponding relationship between a set gesture type and the operation instruction and a relative spatial position relationship between the head-mounted display device and the intelligent ring so as to realize an interaction process, and in the interaction process, the gesture type of the head-mounted display device is determined according to the operation instruction. And determining the legality of the user by using the trained identity authentication model. A novel head-hand interaction normal form is provided, and more natural, efficient and user-friendly man-machine interaction experience is achieved.

Description

Technical Field

[0001] The present invention relates to the field of extended reality (XR) interaction technology, and more specifically, to a smart ring system and interaction method for XR interaction scenarios. Background Art

[0002] Virtual world information interaction in head-mounted smart glasses is an essential and frequent task in XR applications. Currently available gesture interaction methods usually rely on external devices or cameras to capture hand movements, which have problems such as being easily affected by ambient light, limited recognition range, and low accuracy, and instability and errors.

[0003] In the prior art, patent application CN202310993412.9 involves the interaction between a smart ring and an electronic device, and uses finger kneading data to improve the user experience of using electronic devices. The smart ring collects finger data, and the electronic device executes operation instructions based on it. This solution focuses on triggering operations with finger data, and the interaction method is relatively simple. It does not involve a specific technical implementation plan, and the operation is not natural and convenient enough. Patent application CN202210910342.1 involves the interaction between a smart ring and a mainboard component. The smart ring integrates sensors and main control modules, collects hand trajectory and spatial motion data, transmits it to the main control module, and controls external devices. This solution improves the convenience and naturalness of operation. But it focuses on mainboard integrated design and data processing.

[0004] After analysis, the existing technologies in the field of human-computer interaction of XR smart glasses mainly include gesture recognition solutions, voice recognition and control solutions, handle / controller input solutions, eye tracking solutions and tactile gloves solutions. The gesture recognition solution uses cameras or sensors to recognize gestures to enable users to interact with the virtual environment, but it takes time to learn to use complex gestures, especially for novices or those who are not familiar with the technology, and it is easy to make mistakes in low light or dynamic environments, resulting in wrong operations, and large delays causing dizziness. The voice recognition and control solution allows users to control the virtual world through voice commands to perform operations or tasks, but in dialects and noisy environments, voice commands are easily misunderstood or unrecognizable, and the use of voice control may cause privacy issues, especially when sensitive information is involved. The handle / controller input solution uses handheld devices or handles to manipulate objects or menus in the virtual environment, but carrying a handle or controller increases the burden, limits the degree of freedom and mobility, and the operation is not natural enough, easy to fatigue, and requires learning and adapting to the handle operation, which is not intuitive and natural enough. Eye tracking solutions track user eye movements to interact, such as looking at a specific area to select or operate. However, long-term gaze can cause eye fatigue or discomfort, affecting user comfort. Frequent calibration is required, and some users, such as those with impaired vision, may not be able to use it accurately. Haptic gloves use hand movements and touch to interact with the virtual environment to enhance the user's sense of immersion and reality. However, this type of solution has high manufacturing and maintenance costs, which limits commercial applications. The sensors and circuits are cumbersome, uncomfortable and complex to wear, and prone to failure. Frequent maintenance is required, and long-term stable use is limited.

[0005] In summary, with the rapid development of electronic technology, users' demand for more convenient and intuitive device interaction methods is growing. Traditional device interaction methods usually rely on direct finger operation on the display or buttons, which may not be flexible or convenient in some scenarios, especially in virtual reality (VR) and augmented reality devices. Existing solutions try to achieve device control through technologies such as camera gesture recognition and eye tracking, but there are problems such as slow recognition speed, light sensitivity, low recognition rate, and dependence on professional equipment, which limits the effectiveness and universality of practical applications. Summary of the invention

[0006] The purpose of the present invention is to overcome the defects of the above-mentioned prior art and provide a smart ring system and interaction method for XR interaction scenarios.

[0007] According to the first aspect of the present invention, a smart ring system for XR interactive scenarios is provided. The system includes a smart ring and a head display device, wherein: the head display device is used to track the user's head rotation angle and determine the target area of ​​the operation; the smart ring is used to identify the user's gesture type based on the conductivity change information of the finger and the spatial motion data of the finger; the head display device executes the corresponding operation instruction for the target area in the application scenario according to the correspondence between the set gesture type and the operation instruction and the relative spatial position relationship between the head display device and the smart ring to realize the interaction process. During the interaction process, the head display device uses a trained identity authentication model to determine the legitimacy of the user.

[0008] According to a second aspect of the present invention, an interaction method is provided. The method comprises the following steps:

[0009] Based on the conductivity change information of the finger sensed by the smart ring and the spatial motion data of the finger, the user's gesture type is identified;

[0010] Use the head display device to track the user's head rotation angle and determine the target area of ​​the operation;

[0011] According to the correspondence between the set gesture type and the operation instruction and the relative spatial position relationship between the head display device and the smart ring, the corresponding operation instruction is executed for the target area in the application scenario to realize the interaction process, and in the interaction process, the trained identity authentication model is used to determine the legitimacy of the user.

[0012] Compared with the prior art, the advantage of the present invention is that the provided smart ring system for XR interactive scenarios utilizes the inertial measurement unit (IMU) and bio-impedance detection sensor provided on the smart ring to achieve accurate detection of finger pinching, finger rubbing, and left-right hand contact, precise calculation of spatial motion, and secure user identity authentication. In addition, the multi-source data fusion detection algorithm combines IMU technology and bio-impedance detection functions, detects weak gesture operations through current loops and motion intensity, and achieves accurate user identity authentication through individual differences in ring movements. Based on the characteristics of the ring device, it realizes the whole-machine interaction paradigm of eye-hand combined interaction and virtual input method. The present invention simplifies human-computer interaction in XR applications, solves the problems of complex operation and poor user experience of traditional XR head-mounted display devices, and allows users to perform hand interaction and secure identity authentication more intuitively and naturally without the need for external devices or handles.

[0013] Further features and advantages of the present invention will become apparent from the following detailed description of exemplary embodiments of the present invention with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate embodiments of the invention and, together with the description, serve to explain the principles of the invention.

[0015] Figure 1 is a schematic diagram of a smart ring system for XR interaction scenarios according to an embodiment of the present invention;

[0016] Figure 2 is a schematic diagram of an application scenario of a head-hand combined interaction paradigm according to an embodiment of the present invention;

[0017] Figure 3 is a schematic diagram of a scene for interacting with a virtual object outside the field of view according to an embodiment of the present invention;

[0018] Figure 4 is a schematic diagram of an implementation process of a user interacting with an object outside the field of view according to an embodiment of the present invention;

[0019] Figure 5 is a schematic diagram of bioimpedance principle of active electrical sensing according to an embodiment of the present invention;

[0020] Figure 6 is a flow chart of obtaining a finger touch signal based on bio-impedance information according to an embodiment of the present invention;

[0021] Figure 7 is a schematic diagram of a pinch-to-release gesture according to an interactive gesture set according to an embodiment of the present invention;

[0022] Figure 8 is a schematic diagram of the implementation process of an XR user handwriting tablet input scenario according to an embodiment of the present invention;

[0023] Fig. 9 4 is a schematic diagram of a process for determining a finger contact state based on an inertial measurement unit according to an embodiment of the present invention. DETAILED DESCRIPTION

[0024] Various exemplary embodiments of the present invention will now be described in detail with reference to the accompanying drawings. It should be noted that the relative arrangement of components and steps, numerical expressions and numerical values ​​set forth in these embodiments do not limit the scope of the present invention unless otherwise specifically stated.

[0025] The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the invention, its application, or uses.

[0026] Technologies, methods, and equipment known to ordinary technicians in the relevant art may not be discussed in detail, but where appropriate, the technologies, methods, and equipment should be considered as part of the specification.

[0027] In all examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not limiting. Therefore, other examples of the exemplary embodiments may have different values.

[0028] It should be noted that like reference numerals and letters refer to similar items in the following figures, and therefore, once an item is defined in one figure, it need not be further discussed in subsequent figures.

[0029] See also Figure 1 As shown, the provided smart ring system for XR interactive scenarios generally includes a smart ring and a head display device. The head display device is, for example, head-mounted smart glasses or other types. The smart ring and the head display device can communicate based on the Bluetooth protocol or other wireless transmission protocols. For example, the finger pinch touch and motion inertia data collected by the smart ring are sent to the smart glasses in the form of data packets through a specific protocol and encoding. The ring can be worn on any finger of the user, and the pinch data between the fingers refers to the data collected by the smart ring when the finger wearing the ring pinches and touches with other fingers.

[0030] The present invention designs a new head-hand interaction paradigm combining head movement + finger ring manual operation, which can be applied to virtual reality or extended reality scenarios. Figure 2 This is a demonstration of the use scenario of the head-hand combined interaction paradigm. First, the user's gaze area is tracked in real time, which can be achieved by monitoring head movement or eye movement to understand the area or object that the user is currently focusing on. Then, the user wears a finger ring equipped with a motion sensor, which enables the user to finely control the image in the gaze area through tiny gestures. Finally, the combination of head movement or eye tracking and finger ring control allows users to interact in a more intuitive and efficient way.

[0031] In general, in the present invention, the head-mounted smart glasses first track the user's head rotation angle in real time to infer the user's approximate operation intention area (or operation target area), and then use the motion sensor and bioimpedance sensor of the finger ring to perform spatial motion calculation and weak gesture operation recognition, such as moving, rotating, pinching and releasing, and then finely control the gaze area. In addition, user data can be collected during the user operation process, and the user identity model can be trained for identity authentication. Using the present invention, accurate and stable gesture recognition, kinematic-based head and hand space calculation, and construction of various user identity models can be performed.

[0032] For stable recognition of weak gestures, for example, by setting the electrode array on the inside of the ring in direct contact with the skin, the bioimpedance sensing technology of active induction can be used to sense the impedance changes in the weak current loop formed by the thumb and index finger. At the same time, combined with the features extracted from the IMU motion data signal, multi-source data fusion and judgment are performed to stably identify weak pinching or left-right hand contact gestures. Bioimpedance detection detects the conductivity changes of fingers during gesture execution to detect fingertip pinching and release. Combined with the smart head display device and the high-quality IMU motion inertial unit in the ring, it can comprehensively collect the motion data of the head and fingers, and provide spatial motion interaction information of the head and gestures.

[0033] For the relative calculation of spatial motion, the IMU inertial unit built into the smart head display and smart ring can track the spatial position and posture of the user's head and hands in real time. The accelerometer and gyroscope in the IMU collect the acceleration and angular velocity data of the head and hand movements, and analyze the speed and angle of the head and hands through steps such as deviation calibration, quaternion posture estimation, coordinate transformation and gravity compensation. Finally, the head display device and ring are used to reconstruct the movement trajectory and real-time posture of the head and hands in two-dimensional and three-dimensional space respectively.

[0034] For user identity authentication and recognition, by training the machine learning model and using the collected bioimpedance and motion characteristic data, the model analyzes the changes in bioimpedance signals and motion characteristics of each user when performing specific actions to build a user identity feature library, thereby learning and identifying the characteristic patterns of different users.

[0035] The present invention can also be used to interact with objects outside the field of view. Figure 3 It is a scene demonstration of the user interacting with virtual objects outside the field of vision. Figure 4 It is the interaction process between the user and the object outside the field of view, which generally includes the following steps:

[0036] Step S41: The user wears the smart XR glasses and the smart ring, wakes up the device and establishes a signal connection between the head display and the ring.

[0037] First, calibrate the relative spatial position of the ring relative to the head-mounted display device. Specifically, use the coordinates of the head-mounted display as the world origin and draw two targets A and B with known spatial position coordinate data in the virtual space. Guide the user to use the extension line of the finger wearing the ring to aim at the two spatial targets and pinch to confirm, record the spatial rotation coordinates of the ring at the two operation moments and the spatial position coordinate difference between the two actions. The spatial coordinate difference of the ring relative to the head-mounted display can be obtained through the calculation formula. The specific calculation formula is as follows:

[0038] Substitute the coordinates (PX1, PY1, PZ1) of the ring P1 at the time of the first click into the rotation matrix and the moving direction, and use the following formula to calculate the new coordinates P ′ , that is, the coordinates of the first target point A:

[0039]

[0040] Among them, R(θ) is the rotation angle matrix of the ring when it is clicked for the first time, and N represents the transformation matrix from the coordinate system of the headset to the coordinate system of the ring, which includes the scale factor of the coordinate transformation. Represents the translation matrix from the initial position of the ring to the target point A, a is the translation distance in the X-axis direction, b is the translation distance in the Y-axis direction, and c is the translation distance in the Z-axis direction.

[0041] Similarly, the second click calibration behavior can also obtain the same coordinate formula. By combining the two formulas, the relative position of the spatial coordinates of the ring device and the coordinates of the headset can be obtained.

[0042] It should be noted that other solutions can be used to calibrate the relative position of the head display device and the ring. For example, the ring is photographed by the head display camera, and the spatial position coordinates and rotation angle coordinates of the ring with the head display as the origin are obtained by visual recognition. For another example, the ring is used as a supporting component of the head display device, and the dormancy cabin of the ring is the head display body. When the dust cover of the dormancy cabin is opened, the ring function is activated, and the relative spatial position of the two devices is a fixed value.

[0043] Step S42: the hand of the user interacting with bare hands enters the blind spot of the camera.

[0044] In this scenario, the IMU can be used to monitor the user's head movement to infer the user's focus area. When the user faces a specific object, it can be considered that the user is focusing on the object. The spatial positioning information provided by the IMU sensor can be used to determine the user's position and orientation. Based on the user's position and orientation, the user's gaze area can be inferred.

[0045] In another embodiment, a more sophisticated eye tracking technology may be used to determine the area of ​​user attention. Specifically, the user first gazes at the area where interaction is required, and the illuminator inside the head display directs light to the eyes to produce reflections, while receiving sensor data through sensors. The sensing directions of these sensors are roughly aligned with the direction from the eyes to the target area, and the characteristics of the reflection are determined based on the sensor data to detect the approximate area of ​​the eye's gaze. In order to obtain a better prediction effect, optionally, the data of head movement and eye movement can be combined to better estimate what the user is looking at by calculating the relative movement of the head and eyes.

[0046] Step S43, using the IMU motion sensor built into the ring to continuously calculate the position of the hand.

[0047] Step S44, continuously tracking the movement of the hand in the three-dimensional space.

[0048] In the steps, the spatial position of the hand is continuously tracked without any blind spot limitation.

[0049] Step S45: fine-grained tracking of the hand posture in three-dimensional space.

[0050] Step S46, using bioimpedance detection and IMU algorithm to detect contact action.

[0051] Step S47, identifying the user's hand interaction actions.

[0052] For example, users can perform subtle gestures such as moving and rotating by wearing a ring. The ring collects path, posture, and gesture data, and transmits the recognition results to the head display device via Bluetooth. By calculating the interactive position mapped by the ring operation, the image in the gaze area can be finely controlled.

[0053] Step S48: the tactile vibration module responds with simulated tactile feedback.

[0054] For example, based on the user's interactive content, the interactive feedback information is transmitted back to the ring device through the Bluetooth wireless signal path. The ring provides simulated tactile feedback in the form of electrical stimulation and mechanical vibration stimulation.

[0055] Step S49, realizing the interaction between the user and the object outside the field of view.

[0056] In the following, the recognition of weak gestures, the definition of gesture sets and multi-function operation scenarios, the calculation of the relative spatial position of the head and hands, and the implementation scheme of user identity authentication will be specifically introduced.

[0057] 1) Stable recognition of weak gestures

[0058] In order to realize a stable and reliable new head-hand interaction paradigm, stable recognition of weak gestures is an essential subtask. Figure 5 This is a schematic diagram of the principle of bioimpedance recognition based on active induction. The ring collects the conductivity and spatial motion data of the finger through the bioimpedance sensor and IMU inertial sensor unit, and uses a multi-source data fusion algorithm to achieve stable recognition of weak gestures, allowing users to perform interactive operations through simple finger movements.

[0059] Figure 6This is a schematic diagram of the process of finger touch recognition based on bioimpedance, which mainly includes: signal transmission: start the ring device, the signal generation circuit generates a 10.7MHz AC signal, and couples the signal to the body through the transmission electrode, the isolation electrode blocks the direct coupling of the signal between the transmitting and receiving electrodes to reduce signal interference and thereby maximize the reduction of the device volume, and the receiving electrode measures the voltage gradient of the signal along the finger in real time; state detection to determine the touch mode: for example, when a touch occurs, the touch surface is connected to cause the human body to form a current change, and the current change generates a differential voltage between the receiving electrodes, which is amplified and processed as a touch signal. Depending on the touch mode (such as pinching, finger rubbing or palm touch), the current path is different, and the signal is also different. Based on the feature extraction and classification of the signal, accurate gesture recognition results are obtained.

[0060] When the user's index finger and thumb are pinched together, the thumb is rubbed on the index finger, or the left and right hands are in contact, a current loop is formed, and the finger pinching movement is roughly detected by detecting the change in finger conductivity. In order to avoid the phenomenon of false touch caused by unconscious half-fist or natural contact between the index finger and the middle finger, the present invention combines IMU data to perform multi-source data gesture discrimination. And introduce kinematic energy window threshold judgment, divide the acceleration and acceleration data collected by the IMU into windows, and then combine the active induction bioimpedance data recognition results to perform motion window threshold judgment. By adopting an adaptive threshold algorithm and making personalized adjustments based on the differences in the user's window energy data, it can effectively reduce false touches and improve the accuracy of gesture recognition.

[0061] The principle of bioimpedance detection of active induction is described as follows: the current source in the ring generates an alternating current (AC) of a stable frequency, which is injected into the organism through a flexible ring electrode, and the voltage difference between different points is measured to calculate the bioimpedance data. Bioimpedance is the resistance of the organism to the current, usually in ohms. The impedance of biological tissue changes with gestures, so gesture information is obtained by measuring the voltage response. For example, when the index finger and thumb of the user wearing the ring are pinched, a loop current is formed and returned to the ring through the thumb. When one hand touches another hand, a current loop is also detected. Although the current is small, a differential voltage is generated between the two receiving electrodes, which can be measured after amplification and used as the original signal for touch detection.

[0062] In summary, gesture detection can accurately capture gestures, including pinching and releasing movements, movement strength, and left and right hand contact.

[0063] 2) Definition of gesture set and multi-function operation scenarios.

[0064] The head-mounted smart glasses use the finger movement and pinching data collected by the smart ring to determine the direction, position and pinching gesture of the finger, and then execute the operation instructions corresponding to the gesture action accordingly, that is, use the corresponding relationship between the set gesture type (or operation action, control gesture, etc.) and the operation instruction to execute the corresponding operation. Figure 7 is an example of a control gesture.

[0065] In one embodiment, the operation gesture set includes a pinch operation, a slide operation, a single-click operation, a double-click operation, etc.

[0066] The pinch operation refers to the type of gesture when the finger wearing the ring is pinched with another finger. For example, if the ring is worn on the user's index finger, the user can pinch it with any of the thumb, middle finger, ring finger, or pinky finger. This pinching operation between two fingers is defined as a gesture type in the operation gesture set.

[0067] The sliding operation refers to the gesture type when the finger wearing the ring slides with another finger. For example, if the ring is worn on the user's index finger, the user can use the thumb to slide outward or inward on the side of the index finger. This sliding operation between two fingers is defined as a gesture type in the operation gesture set.

[0068] The single-tap operation refers to the action that after the user wears the ring device and ensures that it fits tightly with the finger, the user quickly touches another finger (such as the thumb) with the finger wearing the ring, and then immediately separates. This quick touch-and-release operation is defined as a gesture type in the operation gesture set.

[0069] Double-click operation means that the user puts on the smart ring and fits it tightly with his finger, then quickly touches another finger twice and immediately separates. This action simulates a quick double-click action similar to a mouse operation, and is also defined as a gesture type in the operation gesture set.

[0070] Through the preset correspondence, the pinch operation can be mapped to an operation instruction executed by the smart glasses device in a specific application scenario, such as a character input instruction, an application control instruction, and a mode control instruction. Therefore, in different application scenarios, the smart glasses device can map the ring operation to multiple corresponding operation instructions. These operation instructions can be used to control objects within the virtual reality application. Specifically, the smart glasses device extracts the corresponding configuration information based on the currently displayed virtual reality scene, and then selects the operation instruction that matches the pinch gesture in the configuration information corresponding to the virtual reality scene according to the user's head movement and gesture pinching action, and executes the operation instruction in the application scenario. By defining a gesture interaction set, a natural gesture set is designed based on gesture detection, thereby meeting the XR interaction requirements.

[0071] According to the unique interactive environment characteristics of XR and the characteristics of the ring, based on the mapping relationship between the set gesture type and the operation instruction, the whole machine interaction paradigm, eye-hand combined interaction and virtual input method and other interactions in various scenarios can be realized, such as virtual object grasping scenarios, handwriting tablet input scenarios, application control scenarios, etc.

[0072] In the virtual object grasping scene, the user wears smart glasses and a ring, and for example, a virtual teacup floats in the virtual reality air about two meters in front of him. The user can move freely, approach the teacup, and then use the index finger and thumb of the hand wearing the ring to pinch the virtual teacup in his hand and drag it to move at will.

[0073] Figure 8 It is the implementation process of the handwriting tablet input scenario, which generally includes: step S81, the user wears smart XR glasses and a ring; step S82, uses the finger wearing the ring to write in the palm of the hand; step S83, uses the bioimpedance sensor and IMU to assist in contact detection; step S84, accurately identifies the character pattern written by the user; step S85, uses the intelligent algorithm to perform stroke analysis; step S86, realizes handwriting recognition of a single word and continues the cycle.

[0074] In the handwriting tablet input scenario, the user wears smart glasses and a ring, and uses the finger wearing the ring to write strokes. For example, there are two writing methods. One is to use the pinching and releasing of the fingers as the start and end marks of the strokes during the writing process, and combine the pinching and releasing gestures and the waving motion detection of the ring in space to perform handwriting input recognition in real time. Another writing method is to use the finger wearing the ring as a drawing pen, and use any part of the body as a writing interface, such as the palm of the other hand, the back of the hand, the user's arm, or even the skin of the user's leg.

[0075] For application control scenarios, when there are multiple applications on the device, the user wants to run, switch or close one of them. Through different pinch gestures, the user can complete these operations on the application. For example, pinch-and-drag can be used to switch candidate applications, a "single click" action of quickly pinching and releasing can be used to select and open the current application, and a "double click" action of pinching and releasing twice quickly means going back or exiting.

[0076] It should be noted that since the smart ring worn on the finger occupies less space, the user can control the electronic device by just pinching and releasing the finger. In addition, since the amplitude of the pinching and releasing movement is small, the user's hand can be operated in the pocket of the clothes, which has the characteristics of micro gestures and can be operated normally in crowded places such as subways and shopping malls.

[0077] 3) Calculation of the relative spatial position of the head and hands.

[0078] In order to achieve a stable and reliable new head-hand interaction paradigm, stable recognition of the spatial movement of the head and hands is an essential subtask. When using traditional extended reality (XR) smart glasses, a common problem is that the method that relies on the camera for visual recognition may cause the user's hand to be in the blind spot of the camera's field of view, resulting in interruption of visual tracking. To solve this problem, the present invention uses IMU sensors integrated in smart glasses and finger rings to calculate the spatial position in real time, so that even if the camera's field of view is blind or the head-mounted glasses cannot directly recognize it, the hand position, displacement and angular rotation can still be continuously estimated.

[0079] Fig. 9 This is a flow chart of the motion posture recognition method based on IMU, which mainly includes: starting the finger ring IMU and starting to collect the finger ring motion data, such as using a gyroscope to detect the change of finger angular velocity; using an accelerometer to measure the linear acceleration of the finger, and using a magnetometer to perform finger positioning in the earth's magnetic field; fusing the collected data through a filtering algorithm to obtain more accurate posture data, for example, multiple filtering algorithms can be used to fuse the data to improve the accuracy and stability of the data, and describe the Euler angle posture, etc.; using the fused data to realize the relative position and posture calculation of the finger in three-dimensional space, such as obtaining information such as rotation, rotation and tilt of the finger; auxiliary bio-impedance detection to determine the contact state of the finger.

[0080] In the present invention, the six-axis IMU sensors in smart glasses and finger rings are used to collect acceleration and angular velocity data in human-computer interaction. The posture quaternion representation is used to describe the changes in the head and hand postures, and these signals are processed with the help of a specific motion posture analysis algorithm to build a simple motion trajectory tracking system. At the same time, various possible noises are eliminated through the algorithm to ensure accurate motion data, thereby extracting the specific head-hand combined interaction intention and interacting with the content in the virtual reality application. During the interaction process, vibration feedback is also provided through the tactile feedback module to enhance the user's interactive experience.

[0081] It should be noted that the smart ring designed by the present invention can be further provided with various functions. For example, the biometric function of the smart ring can be enhanced, such as fingerprint recognition, vein recognition or more advanced biometric recognition; the acceleration sensor can be used to detect the number of steps and activity intensity of the user, and the sleep quality can be monitored in combination with the heart rate and blood oxygen data; the temperature sensor can be used to monitor the skin temperature; the control module can calculate the number of steps and skin temperature of the user based on a preset algorithm, and transmit the data to the user's mobile phone application. Specifically, the Bluetooth module is connected to the microcontroller (MCU) of the control module, and generates data such as the number of steps of the user through algorithms such as exercise, sleep and pressure, and then transmits it to the mobile phone application.

[0082] 4) User identity authentication.

[0083] During the use of wearable devices such as XR glasses, many scenarios that require high-security authentication are involved, such as financial transactions, smart home access control, etc. The present invention applies bioimpedance recognition technology to user identity authentication, and realizes accurate identification of user identity by detecting changes in bioimpedance signals when users perform specific actions (such as pinching and releasing) while wearing smart rings. Through this design, the differences in biological tissue structures of different users, including factors such as fat layer thickness, finger thickness, skin roughness, speed of movement, and pinching and releasing strength, can be utilized to cause heterogeneity in bioimpedance measurement results. By collecting data from bioimpedance signals and IMU sensors, action characteristics can be obtained, and these heterogeneous features can be detected and distinguished in combination with machine learning algorithms as unique features to identify users, thereby performing effective user identity authentication.

[0084] Specifically, by training the machine learning model, using the collected bioimpedance and motion characteristic data, the model is able to learn and identify the characteristic patterns of different users. During the training process, the model first enhances the input data after feature generation and concatenates it twice, and then divides each generated data feature into two parts: motion feature and impedance feature, each of which is processed by a separate CNN branch using a 1D convolutional layer. A triplet loss (Triplet MarginLoss) is used to distinguish between real and disguised comparisons. Subsequently, it is processed by a multi-layer perceptron (MLP) and supervised contrast pre-training is used to enhance robustness to noise. By analyzing the changes in the bioimpedance signal and the motion characteristics of each user when performing a specific action, a user identity feature library is constructed. The entire model is divided into two parts: an offline pre-training phase and an online fine-tuning phase.

[0085] In the offline pre-training phase, a large number of bioimpedance and IMU data sets are collected during the use of the user to train the baseline model. This phase uses a large amount of data and computing resources to learn the common motion features and basic identity recognition features in the data. These common features can be reused in the fine-tuning tasks in the online phase to help the model quickly adapt when faced with new user data. During the fine-tuning process in the online phase, the device user enters a small amount of personal registration data when the device is turned on for the first time. At this time, some layers of the model after pre-training are frozen, and only the last few layers are trained. In this way, only a small amount of user data and computing resources are needed to make the model transition from common features to features that are more suitable for specific tasks, thereby obtaining a higher identity authentication accuracy for specific users. In general, the combination of pre-training and fine-tuning can maximize the use of existing data and computing resources, improve the generalization ability of the model and its performance on specific tasks, especially when data is limited.

[0086] In actual applications, when a new user uses the device for the first time, they only need to wear the ring for 2-3 minutes to enter registration data to complete the fine-tuning of the recognition model. In later use, when the user wears the ring and performs pinch-and-release actions, the system will use the trained machine learning model to compare the real-time data with the feature library to determine the user's identity.

[0087] Correspondingly, using the provided smart ring system, the present invention also provides an interaction method, which includes: identifying the user's gesture type based on the conductivity change information of the finger and the spatial motion data of the finger sensed by the smart ring; using a head display device to track the user's head rotation angle and determine the target area of ​​the operation; according to the correspondence between the set gesture type and the operation instruction and the relative spatial position relationship between the head display device and the smart ring, executing the corresponding operation instruction for the target area in the application scenario to realize the interaction process, and in the interaction process, using a trained identity authentication model to determine the legitimacy of the user.

[0088] In order to further verify the effect of the present invention, experimental verification was carried out.

[0089] 1) Verify gesture recognition performance

[0090] The present invention combines bioimpedance sensors and IMU sensors for gesture recognition. Compared with existing devices, it is found that single IMU sensors are insensitive to weak gestures and have low recognition accuracy; single bioimpedance sensors are prone to false touches. By integrating the advantages of IMU and bioimpedance sensors, the present invention shows high accuracy and stability in gesture recognition and can effectively recognize various gestures.

[0091] 2) Verify the spatial motion tracking effect

[0092] The present invention calculates the spatial motion by calculating the IMU sensor data in the head-mounted smart glasses and the smart ring, and continuously tracks the hand position at different viewing angles outside the field of view. Based on the universality and stability of the IMU sensor, the present invention has a small position and speed tracking error under different conditions and has a high tracking accuracy.

[0093] 3) Verify user authentication function

[0094] The present invention uses the uniqueness of gestures and bioimpedance signals to authenticate user identities. Gestures and bioimpedance signals are individual-specific, and there are significant differences between gestures and bioimpedance signals of different users. By extracting features and performing pattern recognition on gestures and bioimpedance signals through machine learning algorithms, different users can be effectively distinguished, achieving high accuracy and low misrecognition rate.

[0095] In summary, compared with the prior art, the present invention has the following beneficial effects:

[0096] 1) Compared with camera gesture recognition technology and eye tracking technology, the present invention is based on bioimpedance and IMU sensing, with bioelectricity as the measurement object, and only needs to be combined with simple signal processing calculations, with high sensitivity, good real-time performance, and high data signal-to-noise ratio, which improves the accuracy and directionality of the operation. The bioimpedance detection function reduces the dependence on the handle and controller, making the interaction more accurate, and no other external equipment is required except the ring, making the operation more natural and convenient, and avoiding eye discomfort caused by eye movement interaction.

[0097] 2) For XR glasses users, compared with the technology that only has handle / controller input or voice recognition control, the present invention designs an innovative head-hand combined interaction method, combining the definition of interaction gesture set, mapping of application operations and character input method, to meet the different needs of users in virtual environments through diversified input.

[0098] 3) Compared with existing recognition technologies such as fingerprints and irises, the authentication of the present invention occurs during normal use and does not require the user to perform deliberate operations, making authentication more natural and suitable for scenarios such as financial transactions and smart home access control that require high-security authentication.

[0099] 4) Compared with technologies such as tactile gloves that require large equipment, the present invention does not require large equipment or complex installation, reduces equipment costs and usage thresholds, improves scalability and portability, and makes it easier for users to accept and use.

[0100] 5) The present invention utilizes the IMU and bio-impedance contact detection functions of the smart ring to realize weak contact gestures, spatial motion calculation and user identity authentication functions, introduces a new gesture interaction and user authentication method for the XR virtual world, and realizes intuitive, natural, accurate and diversified XR head-mounted display device human-computer interaction and security authentication.

[0101] 6) The present invention adopts bio-impedance and IMU-assisted contact detection to ensure accurate determination of hand contact and contact between the first hand and the second hand, which helps to determine the user's operating intention and improves the accuracy of the input method and user experience.

[0102] 7) The present invention defines a gesture operation set and its mapping relationship on the smart head display, and develops and defines an operation gesture set based on the functional characteristics of the ring, such as pinch, pinch and double-click. And designs a mapping relationship between the gesture set and the background operations in the virtual reality application, such as switching, selecting, exiting, etc.

[0103] In summary, the present invention has innovated a new hand interaction paradigm through a wearable ring device, and designed a gesture recognition scheme combining multi-source data, which achieves accurate tracking of hand positions, and realizes accurate authentication of user identities through differences in user movements. The virtual keyboard layout is friendly and intuitive, reducing the learning cost and gesture memory requirements. User identity authentication based on biological individual specificity improves device security. The smart ring replaces complex tactile gloves, reducing manufacturing costs, and combines IMU and bio-impedance detection functions. Through the integration of IMU hand position and posture tracking, bio-impedance detection, eye-hand interaction innovation, and virtual input method design, a more natural, efficient, and user-friendly human-computer interaction experience is achieved.

[0104] The present invention may be a system, a method and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for causing a processor to implement various aspects of the present invention.

[0105] Computer readable storage medium can be a tangible device that can hold and store instructions used by an instruction execution device. Computer readable storage medium can be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. More specific examples (non-exhaustive list) of computer readable storage medium include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, for example, a punch card or a convex structure in a groove on which instructions are stored, and any suitable combination thereof. The computer readable storage medium used here is not interpreted as a transient signal itself, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagated by a waveguide or other transmission medium (for example, a light pulse by an optical fiber cable), or an electrical signal transmitted by a wire.

[0106] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in the computer-readable storage medium in each computing / processing device.

[0107] The computer program instructions for performing the operation of the present invention may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages, such as Smalltalk, C++, Python, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. Computer-readable program instructions may be executed entirely on a user's computer, partially on a user's computer, as an independent software package, partially on a user's computer, partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., using an Internet service provider to connect via the Internet). In some embodiments, an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), may be personalized by utilizing the state information of the computer-readable program instructions, and the electronic circuit may execute the computer-readable program instructions, thereby realizing various aspects of the present invention.

[0108] Various aspects of the present invention are described herein with reference to the flow charts and / or block diagrams of the methods, devices (systems) and computer program products according to embodiments of the present invention. It should be understood that each box of the flow chart and / or block diagram and the combination of each box in the flow chart and / or block diagram can be implemented by computer-readable program instructions.

[0109] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processor of the computer or other programmable data processing device, a device that implements the functions / actions specified in one or more boxes in the flowchart and / or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0110] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operating steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0111] The flow charts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the systems, methods and computer program products according to multiple embodiments of the present invention. In this regard, each box in the flow chart or block diagram can represent a part of a module, a program segment or an instruction, and a part of the module, a program segment or an instruction contains one or more executable instructions for realizing the specified logical function. In some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented by a dedicated hardware-based system that performs the specified function or action, or can be implemented by a combination of dedicated hardware and computer instructions. It is well known to those skilled in the art that it is equivalent to implement it by hardware, implement it by software, and implement it by combining software and hardware.

[0112] Embodiments of the present invention have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or technical improvements in the marketplace, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein. The scope of the present invention is defined by the appended claims.

Claims

1. A smart ring system for XR interaction scenarios, comprising: Smart ring and head-mounted display device, including: The head display device is used to track the rotation angle of the user's head and determine the target area of ​​the operation; The smart ring is used to identify the user's gesture type based on the conductivity change information of the finger and the spatial motion data of the finger; The head-mounted display device executes corresponding operation instructions for the target area in the application scenario according to the correspondence between the set gesture type and the operation instruction and the relative spatial position relationship between the head-mounted display device and the smart ring to realize the interaction process, and during the interaction process, the legitimacy of the user is determined by using the trained identity authentication model.

2. The system according to claim 1, characterized in that The relative spatial position relationship between the head display device and the smart ring is determined according to the following steps: Taking the coordinates of the head display device as the world origin, drawing a first target point and a second target point with known spatial position coordinate data in a virtual space; The user is guided to use the extension line of the finger wearing the smart ring to aim at the first target point and the second target point, and confirm through the finger pinching operation, so as to obtain the spatial rotation coordinate difference of the smart ring at the time of the two pinching operations and the spatial position coordinate difference between the two finger pinching operations, and then obtain the relative spatial position relationship between the head display device and the smart ring.

3. The system according to claim 1, characterized in that The smart ring is provided with a bioimpedance sensor and a first inertial measurement unit, the bioimpedance sensor is used to sense the conductivity change information of the finger, the first inertial measurement unit is used to obtain the spatial motion data of the finger, and the head display device is provided with a second inertial measurement unit. The spatial positioning information provided by the second inertial measurement unit is used to determine the user's position and orientation, and then infer the target area of ​​the user's gaze.

4. The system according to claim 1, characterized in that The application scenarios include a handwriting tablet input scenario, an application control scenario, and a virtual object grasping scenario. In the handwriting tablet input scenario, the finger wearing the smart ring is used as a drawing pen, and a set body part is used as a writing interface. In the application control scenario, the target application is switched, selected, or returned. The operation instructions include character input instructions, application control instructions, and mode control instructions.

5. The system according to claim 1, characterized in that The identity authentication model is trained according to the following steps: Constructing a training set, wherein the training set reflects the relationship between bioimpedance information, motion characteristic data and characteristic patterns of different users; Using the training set to perform offline pre-training on the identity authentication model to learn common user feature patterns; The pre-trained identity authentication model is fine-tuned online, and the identity authentication model is adjusted to adapt to a specific user by setting a certain amount of personal registration data, so as to construct an identity feature library of the user.

6. The system according to claim 1, characterized in that The target area is determined by calculating the relative movement of the user's head and eyes.

7. The system according to claim 1, characterized in that The head display device is a smart glasses, and there is a wireless connection between the smart ring and the smart glasses.

8. The system according to claim 3, characterized in that The bioimpedance sensor is an array electrode.

9. An interactive method based on the system according to any one of claims 1 to 8, comprising the following steps: Based on the conductivity change information of the finger sensed by the smart ring and the spatial motion data of the finger, identifying the gesture type of the user; Using the head display device to track the user's head rotation angle and determine the target area of ​​the operation; According to the correspondence between the set gesture type and the operation instruction and the relative spatial position relationship between the head display device and the smart ring, the corresponding operation instruction is executed for the target area in the application scenario to realize the interaction process, and in the interaction process, the trained identity authentication model is used to determine the legitimacy of the user.

10. A computer-readable storage medium having a computer program stored thereon, wherein: When the computer program is executed by a processor, the steps of the interaction method according to claim 9 are implemented.

Images