A virtual scene interaction system and method based on an inertial measurement unit

Abstract

The application discloses a virtual scene interaction system and method based on an inertial measurement unit, and the system comprises a preprocessing module, a feature extraction module, a feature fusion module and a multi-task recognition module; the preprocessing module is used for collecting original ten-axis data vectors by using a micro inertial measurement unit and preprocessing the ten-axis data vectors; the feature extraction module is used for inputting the ten-axis data vectors into a statistical feature encoder and a time feature encoder respectively to extract space embedding vectors and time sequence embedding vectors; the feature fusion module is used for obtaining finger feature fusion vectors and action feature fusion vectors according to the space embedding vectors, the time sequence embedding vectors and the ten-axis data vectors; the multi-task recognition module is used for recognizing target fingers and action types according to the finger feature fusion vectors and the action feature fusion vectors, and obtaining joint confidence of the target fingers and the action types; and a mapping strategy and a key generation module is used for selecting a mapping strategy, outputting corresponding target characters according to the joint confidence and the mapping strategy, and displaying the target characters; thereby, text input without complex aiming and low fatigue is realized.

Description

Technical Field

[0001] This invention relates to the field of virtual scene interaction technology, and in particular to a virtual scene interaction system and method based on an inertial measurement unit. Background Technology

[0002] With the widespread adoption of virtual reality / augmented reality devices, text input has become an important yet challenging interactive task. Traditional handheld controllers are bulky, and users easily forget their placement after switching between multiple scenarios, increasing the burden of use. Camera-based gesture recognition or hand-tracking-based aiming selection requires high cognitive and posture control costs, leading to user fatigue. While ring or thumb-worn devices offer high comfort, current technologies generally rely on complex vision systems to track finger movements to identify contact or accurate positioning between fingers. This process involves acquiring high-precision images of finger movements with a camera, followed by complex steps such as coordinate system transformation and image processing, resulting in high computational resources and making it impossible to achieve efficient, low-cognitive-burden text input while maintaining comfort. Summary of the Invention

[0003] In view of the above-mentioned prior art, the present invention provides a virtual scene interaction system and method based on an inertial measurement unit, which mainly solves the technical problems existing in the background art.

[0004] To achieve the above objectives, the technical solution of this invention is implemented as follows:

[0005] In a first aspect, the present invention provides a virtual scene interaction system based on an inertial measurement unit, comprising:

[0006] The preprocessing module is used to collect raw ten-axis data vectors from the micro inertial measurement unit and preprocess them to obtain ten-axis data vectors.

[0007] The feature extraction module is used to input the ten-axis data vector into the statistical feature encoder and the temporal feature encoder respectively to extract features and obtain spatial embedding vector and temporal embedding vector;

[0008] The feature fusion module is used to input the spatial embedding vector, the temporal embedding vector and the ten-axis data vector into the context-aware gating network to obtain the finger feature fusion vector and the action feature fusion vector.

[0009] The multi-task recognition module is used to input the finger feature fusion vector into the finger classification head to identify the target finger, and input the action feature fusion vector into the action classification head to identify the action type, thereby obtaining the joint confidence of the target finger and the action type;

[0010] The mapping strategy and key generation module is used to map the identified target finger and action type to the virtual keyboard according to the joint confidence and the selected mapping strategy, output the corresponding target characters and display them; the mapping strategy includes a spatially compatible mapping strategy and an ergonomically optimal mapping strategy.

[0011] Optionally, the statistical feature encoder includes a physical feature extraction layer, a fully connected layer, an activation function, a batch normalization layer, and a Dropout layer;

[0012] The physical feature extraction layer is used to extract statistical feature scalars based on the ten-axis data vector;

[0013] The fully connected layer and the activation function are used to establish a cross-user finger mapping baseline;

[0014] The batch normalization layer is used to automatically adjust the sensor reference plane drift by standardizing the feature distribution;

[0015] The Dropout layer is used to output hidden layer feature representations and output spatial embedding vectors.

[0016] Optionally, the step of inputting the ten-axis data vector into the temporal feature encoder for feature extraction to obtain the temporal embedding vector includes:

[0017] The ten-axis data vector includes a gravity-free linear acceleration vector, a filtered three-axis angular velocity vector, and an attitude quaternion vector.

[0018] The temporal feature encoder includes a physical feature extraction layer and a temporal convolutional network;

[0019] The temporal convolutional network includes causal convolutional layers, dilated convolutional layers, and residual connection layers;

[0020] The physical feature extraction layer is used to obtain an acceleration vector sequence and a short-time frequency domain capability spectrum sequence based on the linear acceleration vector;

[0021] The causal convolutional layer is used to extract high-frequency pulse features from the acceleration vector sequence;

[0022] The dilated convolutional layer is used to expand the receptive field;

[0023] The residual connection layer is used to transmit the original acceleration vector sequence information and finally output the temporal embedding vector.

[0024] Optionally, the context-aware gating network includes: a context-aware gating unit and a task decoupling gating unit;

[0025] The context-aware gating unit is used to perform temporal pooling on the ten-axis data vector to obtain the original physical context vector, and to concatenate the spatial embedding vector, the temporal embedding vector and the original physical context vector to obtain the gating input vector;

[0026] The task decoupling gating unit is used to process the gating input vector, calculate the finger task weight and the action task weight, and calculate the finger feature fusion vector and the action feature fusion vector based on the finger task weight and the action task weight.

[0027] Optionally, the finger classification head includes a fully connected layer and a Softmax normalization layer, which are used to process the finger feature fusion vector to obtain a three-dimensional probability vector, and output the target finger recognition result based on the three-dimensional probability vector;

[0028] The action classification head includes a fully connected layer and a Softmax normalization layer, which are used to process the action feature fusion vector to obtain a five-dimensional probability vector, and output the action type recognition result based on the five-dimensional probability vector.

[0029] Optionally, the ergonomic optimal mapping strategy specifically includes:

[0030] The fatigue cost of each movement is calculated based on the finger dexterity factor and the movement difficulty factor.

[0031] A preset standard English character frequency table is provided, which divides characters into a high-frequency core character set, a mid-frequency commonly used character set, and a low-frequency edge character set.

[0032] Obtain the user's historical input logs, calculate the user's character distribution frequency based on the historical input logs, update the standard English character frequency table based on the character distribution frequency, and obtain the current character frequency table;

[0033] A greedy algorithm is used to traverse all available finger-action combinations to generate a character recommendation mapping table;

[0034] The character with the highest confidence in the character recommendation mapping table is output and displayed as the target character.

[0035] Optionally, the step of using a greedy algorithm to traverse all available finger-action combinations and generate a character recommendation mapping table includes:

[0036] Step S1: Traverse all available "finger-action" combinations, sort them in ascending order according to the fatigue value, and generate the preferred action sequence;

[0037] Step S2: Arrange the twenty-six letters in descending order according to the current character frequency table to generate a character frequency sequence;

[0038] Step S3: Generate a character recommendation mapping table by mapping the preferred action sequence and the character frequency sequence.

[0039] Optionally, the spatially compatible mapping strategy specifically includes:

[0040] The virtual keyboard is divided into a left mapping area and a right mapping area. Both the left and right mapping areas include three sub-areas, and each sub-area includes a certain number of characters.

[0041] Based on the number of characters in the sub-region, the valid actions of the target finger in the sub-region are defined, the corresponding characters are determined based on the valid actions, and a target finger-action-character mapping rule table is generated. Based on the target finger-action-character mapping rule table, after the target finger performs an action in the sub-region, the target character corresponding to the action is output and displayed.

[0042] Optionally, defining the effective actions of the target finger in the sub-region based on the number of characters in the sub-region includes:

[0043] If the number of characters in the sub-region is five, the center character in the sub-region is used as the reference key, and the reference key is used for the target finger to perform a click action; the adjacent keys of the center character are used for the target finger to perform a short swipe action; the distal keys of the center character are used for the target finger to perform a long swipe action.

[0044] If the number of characters in the sub-region is four, the center point of the middle two characters in the sub-region is used as the reference key. The adjacent keys of the center point are used for the target finger to perform a short slide operation; the keys at the far end of the center point are used for the target finger to perform a long slide operation.

[0045] If the number of characters in the sub-region is three, the center character in the sub-region is used as the reference key, and the reference key is used for the target finger to perform a click action; the adjacent keys of the center character are used for the target finger to perform a long swipe action.

[0046] When the number of characters in the sub-region is two, the center point of the two characters in the sub-region is used as the reference key, and the two characters are used for the target finger to perform a long sliding operation.

[0047] When the number of characters in the sub-region is one, the characters in the sub-region are used by the target finger to perform a click action.

[0048] Secondly, the present invention provides a virtual scene interaction method based on an inertial measurement unit, comprising the following steps:

[0049] The raw ten-axis data vector is acquired based on a miniature inertial measurement unit, and the raw ten-axis data vector is preprocessed to obtain a ten-axis data vector including acceleration vector, angular velocity vector and attitude quaternion vector;

[0050] The ten-axis data vector is input into the statistical feature encoder and the temporal feature encoder respectively for feature extraction to obtain the spatial embedding vector and the temporal embedding vector.

[0051] Based on the spatial embedding vector, the temporal embedding vector, and the ten-axis data vector, a finger feature fusion vector and a motion feature fusion vector are obtained.

[0052] Based on the finger feature fusion vector and the action feature fusion vector, the target finger and action type are identified;

[0053] The user selects a mapping strategy, which maps the target finger and action type based on the recognition, outputs the corresponding target character, and displays it.

[0054] The beneficial effects of this invention are as follows: This invention provides a virtual scene interaction system based on an inertial measurement unit. Through the setting of a preprocessing module, a feature extraction module, a feature fusion module, a multi-task recognition module, and a mapping strategy and key generation module, it can process the raw ten-axis data obtained from the micro inertial measurement unit, and finally identify the target finger and action type in contact with the thumb. Based on the target finger, action type, and mapping strategy, it outputs accurate characters and displays them. The operation is simple.

[0055] This invention provides a virtual scene interaction system based on inertial measurement units (IMUs). It eliminates the need for cameras to capture finger movement images and convert them into spatial mappings in virtual reality to determine the coordinates between fingers and the corresponding keyboard. Instead, it uses miniature IMUs worn on the thumbs to identify contact and sliding operations between the thumb and index / middle / ring fingers based on vibration response characteristics and inertial signals. Combined with keyboard column partitioning and finger-gesture mapping, this system breaks free from the reliance of virtual keyboard technology on absolute spatial coordinates and visual positioning. In other words, it uses the device to detect the distance between the finger and the target position to confirm contact and input, thus achieving text input without complex aiming and with low fatigue.

[0056] This invention provides a virtual scene interaction method based on an inertial measurement unit (IMU). By establishing a mapping relationship between "finger-action-corresponding target character," the input character no longer depends on the coordinates of the hovering finger, but rather on which finger the thumb contacts and what micro-gesture (click or swipe) is performed after contact. Utilizing the natural proprioceptive tactile feedback inherent when the thumb makes physical contact with other fingers, it provides users with a clear physical confirmation of "input start," solving the pain point of traditional air gestures lacking physical feedback. By using the vibration when the finger contacts and the displacement integral in one dimension for input mapping, the complexity of confirming the position in two-dimensional space is reduced, making the method provided by this technical solution simpler. Attached Figure Description

[0057] Figure 1 This is a schematic diagram of the structure of the virtual scene interaction system based on an inertial measurement unit provided in an embodiment of the present invention;

[0058] Figure 2 This is a schematic diagram showing the wearing state of the miniature inertial measurement unit provided in an embodiment of the present invention;

[0059] Figure 3 This is a schematic diagram of the structure of the miniature inertial measurement unit provided in an embodiment of the present invention;

[0060] Figure 4 This is a schematic diagram of a virtual keyboard for a space-compatible mapping strategy provided in an embodiment of the present invention;

[0061] Figure 5 This is a flowchart illustrating the virtual scene interaction method based on an inertial measurement unit provided in an embodiment of the present invention.

[0062] Explanation of icon numbers:

[0063] 1. Miniature inertial measurement unit; 11. Upper shell; 12. PCB body; 13. Lower shell; 14. IMU; 15. Bluetooth chip; 16. Battery; 17. Type-C charging port. Detailed Implementation

[0064] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the invention. In the following description, the expression "some embodiments" refers to a subset of all possible embodiments; however, it should be understood that "some embodiments" can be the same subset or different subsets of all possible embodiments and can be combined with each other without conflict.

[0065] Example 1

[0066] Please refer to the attached document. Figure 1-3 This application provides a virtual scene interaction system based on an inertial measurement unit, comprising:

[0067] The preprocessing module is used to collect raw ten-axis data vectors from the micro inertial measurement unit and preprocess them to obtain ten-axis data vectors.

[0068] The feature extraction module is used to input the ten-axis data vector into the statistical feature encoder and the temporal feature encoder respectively to extract features and obtain spatial embedding vector and temporal embedding vector;

[0069] The feature fusion module is used to input the spatial embedding vector, the temporal embedding vector and the ten-axis data vector into the context-aware gating network to obtain the finger feature fusion vector and the action feature fusion vector.

[0070] The multi-task recognition module is used to input the finger feature fusion vector into the finger classification head to identify the target finger, and input the action feature fusion vector into the action classification head to identify the action type, thereby obtaining the joint confidence of the target finger and the action type;

[0071] The mapping strategy and key generation module is used to map the identified target finger and action type to the virtual keyboard according to the joint confidence and the selected mapping strategy, output the corresponding target characters and display them; the mapping strategy includes a spatially compatible mapping strategy and an ergonomically optimal mapping strategy.

[0072] Specifically, two miniature inertial measurement units (MMUs) 1 are provided, worn on the dorsal side of the proximal phalanx of the left and right thumbs respectively. The left and right MMUs 1 are used to collect the raw ten-axis data vectors of the left and right hands respectively. Each MMU 1 includes an upper shell 11, a lower shell 13, and a PCB body 12 between the upper and lower shells 11 and 13. The PCB body 12 houses an IMU 14, a Type-C charging port 17, a Bluetooth chip 15 for communication, a battery 16 providing power to the MMU 1, and a main processor. The IMU 14 includes a three-axis accelerometer for collecting acceleration data and a three-axis gyroscope for collecting angular velocity data. The IMU 14 is located on the upper part of the PCB body 12 and close to the center of the thumb skin contact surface. The Bluetooth chip 15 is located below the IMU 14. (Type-C...) The charging port 17 is located at the bottom edge of the PCB body 12 and is exposed to the outside through the opening of the lower shell 13; the upper shell 11 and the lower shell 13 are fixed to both sides of the PCB body 12 by fasteners to form a sealed cavity, and the fasteners are buckles or screws; the upper shell 11 or the lower shell 13 is connected to the thumb by an adjustable wearing strap, the wearing strap is an elastic fabric or a synthetic material with Velcro, and the micro inertial measurement unit 1 is attached to the thumb by the wearing strap;

[0073] The sensor coordinate system is set as follows: X-axis: parallel to the wearing loop, i.e., the lateral direction around the finger; Y-axis: the longitudinal direction pointing towards the fingertip; Z-axis: perpendicular to the back of the finger upwards, i.e., the normal direction; the miniature inertial measurement unit 1 outputs raw ten-axis data vectors in real time at the sampling frequency. The raw ten-axis data vectors include three-axis acceleration vectors. Three-axis angular velocity vector and quaternion vector ;

[0074] Next, a low-pass filter is applied to the triaxial acceleration and triaxial angular velocity vectors in the original ten-axis data vector to obtain the filtered acceleration and angular velocity vectors. A rotation matrix is ​​then constructed using the quaternion vectors in the original ten-axis data vector, with the standard gravity vector in the Earth coordinate system being... By mapping the standard gravity vector to the current sensor coordinate system, the gravity components are obtained. The total acceleration is calculated from the filtered triaxial acceleration vector. Based on the total acceleration, the linear acceleration vector generated solely by the thumb's movement is obtained. ;

[0075]

[0076]

[0077] in, For gravity, For the gravitational component, It is a linear acceleration vector. This is the total acceleration;

[0078] Next, the preprocessed ten-axis data vector is input into the statistical feature encoder and the temporal feature encoder for feature extraction. The statistical feature encoder calculates the mapping relationship between the spatial pose of the thumb and the target finger (index, middle, or ring finger) to obtain a spatial embedding vector. The temporal feature encoder captures thumb motion features to obtain a temporal embedding vector. The spatial and temporal embedding vectors are then input into the feature fusion module for processing, yielding a finger feature fusion vector and a motion feature fusion vector. The multi-task recognition module then uses the finger feature fusion vector and motion feature fusion vector to perform the final recognition. The feature fusion vector determines the target finger and action type. The target finger includes the index finger, middle finger, or ring finger, that is, determining whether the finger in contact with the thumb is the index finger, middle finger, or ring finger. The action type includes click, short inner swipe, short outer swipe, long inner swipe, and long outer swipe. The action type refers to whether the contact state between the thumb and the target finger is a click, a short inner swipe, a short outer swipe, a long inner swipe, or a long outer swipe. After determining the target finger and action type, the mapping strategy and key generation module maps the target finger to the action type according to the mapping strategy selected by the user, outputs the corresponding target character, and displays it.

[0079] The virtual scene interaction system based on an inertial measurement unit provided by this invention may further include a feedback module for prompting the user after valid output of a target character is recognized. Specifically, this can be achieved through either intrinsic feedback or external feedback. In intrinsic feedback, a tactile actuator (e.g., LRA, ERM, or piezoelectric element) can be integrated into the micro inertial measurement unit 1. After valid input is recognized, the actuator is directly driven to emit a short pulse tactile feedback (e.g., 20–50 ms, intensity of which can be linearly adjusted according to confidence level) to prompt the user that the target character has been recognized. In external feedback, no actuator is built-in; that is, the micro inertial measurement unit 1 does not integrate an actuator. Instead, after valid input is recognized, a feedback trigger message is sent to a paired external device via a communication module, and the external device (mobile phone, watch, AR glasses) generates tactile, acoustic, or visual cues.

[0080] It should be noted that existing virtual keyboard mapping text input technologies generally require the participation of complex vision systems (such as cameras) to identify the contact or positioning between fingers. Specifically, the visual positioning is converted into spatial mapping in virtual reality to determine the coordinate position between the finger and the corresponding keyboard, thereby outputting the corresponding target character. In contrast, this technical solution does not require the use of a vision system to collect hand image information, nor does it require determining the spatial coordinate position of the hand. Instead, it uses the collected raw ten-axis data vector for processing to identify the target finger (whether the finger in contact with the thumb is the index, middle, or ring finger) and the action type (the contact state of the target finger in contact with the thumb is a click, etc.). (Inner short swipe, outer short swipe, inner long swipe, or outer long swipe); based on the target finger and the type of action, the corresponding target character can be output; that is to say, the output target character of this technical solution does not depend on the coordinates of the hovering finger, but on which target finger the thumb contacts and what type of action behavior is performed after contact; that is: based on the micro inertial measurement unit 1, blind typing can be realized without knowing the coordinates of the hand in space, and without the need for a camera to be watching the hand. The user can even put their hand behind their back, and as long as the thumb can touch the index finger and slide, they can type. This is a coding based on touch and relative motion, rather than spatial positioning based on vision, thereby realizing text input without complex aiming and with low fatigue.

[0081] As an optional implementation, the statistical feature encoder includes a physical feature extraction layer, a fully connected layer, an activation function, a batch normalization layer, and a Dropout layer.

[0082] The physical feature extraction layer is used to extract statistical feature scalars based on the ten-axis data vector;

[0083] The fully connected layer and the activation function are used to establish a cross-user finger mapping baseline;

[0084] The batch normalization layer is used to automatically adjust the sensor reference plane drift by standardizing the feature distribution;

[0085] The Dropout layer is used to output hidden layer feature representations and output spatial embedding vectors.

[0086] Specifically, by setting a fixed time window (containing N sampling points), a ten-axis data vector within the fixed time window is received, and a ten-axis temporal tensor of shape (N, 10) is constructed from the ten-axis data vector. For any sampling time within the window Its data vector It includes the following 10 physical components:

[0087]

[0088] in, Let x, y, and z represent the linear acceleration components along the x, y, and z axes after gravity is removed at the i-th sampling time. These represent the filtered x, y, and z axis angular velocity components at the i-th sampling time. Let represent the attitude quaternion components at the i-th sampling time;

[0089] The signal energy value is extracted from the ten-axis data vector through the physical feature extraction layer. ), waveform kurtosis (K), waveform skewness (S), inter-axis correlation coefficient ( ), variance of resultant angular velocity ( The six statistical characteristic scalars are: ), rotational energy ratio (R);

[0090] The signal energy value is calculated based on the linear acceleration vector, and the calculation formula is:

[0091]

[0092]

[0093] Where i represents the i-th sampling point within the time window; N represents the total number of sampling points within the time window; Let X, Y, and Z represent the linear acceleration components of the X, Y, and Z axes after gravity removal at the i-th sampling time, respectively. This represents the magnitude of the resultant acceleration at the i-th sampling time. This represents the calculated signal energy value.

[0094] The waveform kurtosis value is calculated based on the x-axis acceleration component sequence in the linear acceleration vector. The calculation formula is as follows:

[0095]

[0096] in, This represents the arithmetic mean of the X-axis linear acceleration components within the time window; The variance of the X-axis linear acceleration component is represented by ; N is the total number of sampling points; and K is the calculated waveform kurtosis value, used to characterize the impact features.

[0097] The waveform skewness value is calculated based on the x-axis acceleration component sequence in the linear acceleration vector, using the following formula:

[0098]

[0099] in, The standard deviation (i.e., the square root of the variance) of the linear acceleration component of the X-axis. The calculated waveform skewness value is used to characterize the sliding directionality.

[0100] The interaxial correlation coefficient is calculated based on the x-axis acceleration component sequence and the z-axis acceleration component sequence of the linear acceleration vector. The calculation formula is as follows:

[0101]

[0102] in, Let x be the arithmetic mean of the x-axis linear acceleration component sequence. Let be the arithmetic mean of the z-axis linear acceleration component sequence. This represents the standard deviation of the Z-axis linear acceleration component; The Pearson correlation coefficient, or inter-axis correlation coefficient, is used to calculate the linear acceleration sequences of the X and Z axes.

[0103] The variance of the resultant angular velocity is calculated based on the x-axis angular velocity component sequence, y-axis angular velocity component sequence, and z-axis angular velocity component sequence in the three-axis angular velocity vector. The calculation formula is as follows:

[0104]

[0105]

[0106] in, This is the sequence of x-axis angular velocity components in the three-axis angular velocity vector. This is the sequence of y-axis angular velocity components in the three-axis angular velocity vector. This is the sequence of z-axis angular velocity components in the three-axis angular velocity vector. For the resultant angular velocity modulus, The mean of the resultant angular velocity, The variance of the resultant angular velocity;

[0107] The rotational energy ratio (R) is calculated based on the x-axis angular velocity component sequence, y-axis angular velocity component sequence, and z-axis angular velocity component sequence in the three-axis angular velocity vector. The calculation formula is as follows:

[0108]

[0109]

[0110] Where R is the rotational energy ratio. The root mean square of the acceleration, The root mean square of the angular velocity. It is a constant;

[0111] The original pose quaternion vector is concatenated with six statistical feature scalars to construct a ten-dimensional static feature vector, which is: Next, a fully connected layer and activation function are used to project the quaternion vector into a high-dimensional semantic space to decouple the complex nonlinear relationship between "hand size" and "target finger" and establish a cross-user finger identification mapping benchmark. Then, a batch normalization layer is used to standardize the feature distribution and automatically compensate for the sensor reference plane drift caused by different wearing tightness and finger thickness. When the thumb searches for the target finger, it is often accompanied by a slight tremor in a suspended state. Then, a Dropout layer is used to focus on the steady-state features at the moment of contact between the thumb and the target finger to improve the stability of subsequent target finger identification. After the Dropout layer, a high-dimensional spatial position embedding vector is finally output, with an embedding dimension of D, where D is 32 or 64.

[0112] Feature extraction in the statistical feature encoder yields the spatial embedding vector, expressed as follows:

[0113]

[0114] in, is the trainable weight matrix of the fully connected layer (D is the embedding dimension); b is the bias vector; The input is a ten-dimensional static feature vector (containing quaternion pose and statistical feature scalar) to the statistical feature encoder. This is a batch normalization operation used to standardize the feature distribution; It is a non-linear activation function; This is the intermediate feature representation after processing by the activation function. After further processing through a Dropout layer (random deactivation), the final output is a spatial embedding vector. .

[0115] As an optional implementation, the step of inputting the ten-axis data vector into a temporal feature encoder for feature extraction to obtain a temporal embedding vector includes:

[0116] The ten-axis data vector includes a gravity-free linear acceleration vector, a filtered three-axis angular velocity vector, and an attitude quaternion vector.

[0117] The temporal feature encoder includes a physical feature extraction layer and a temporal convolutional network;

[0118] The temporal convolutional network includes causal convolutional layers, dilated convolutional layers, and residual connection layers;

[0119] The physical feature extraction layer is used to obtain an acceleration vector sequence and a short-time frequency domain capability spectrum sequence based on the linear acceleration vector;

[0120] The causal convolutional layer is used to extract high-frequency pulse features from the acceleration vector sequence;

[0121] The dilated convolutional layer is used to expand the receptive field;

[0122] The residual connection layer is used to transmit the original acceleration vector sequence information and finally output the temporal embedding vector.

[0123] Specifically, the acceleration vector sequence and short-time frequency domain energy spectrum are extracted through the physical feature extraction layer in the temporal feature encoder. The linear acceleration vector, angular velocity vector, extracted acceleration vector sequence, and short-time frequency domain energy spectrum are concatenated along the channel dimension to construct a ten-dimensional dynamic feature vector for each moment. ;

[0124] in, Let x, y, and z represent the linear acceleration components along the x, y, and z axes after gravity is removed at the i-th sampling time. These represent the filtered x, y, and z axis angular velocity components at the i-th sampling time. These represent the x, y, and z-axis acceleration component sequences extracted at the i-th sampling time. This represents the short-time frequency domain energy spectrum extracted at the i-th sampling time.

[0125] Stack the feature vectors from all moments within the time window to form a shape of... Multidimensional dynamic input tensor The data is then fed into a Temporal Convolutional Network (TCN). Inside the network, joint feature extraction is performed on all channels (including acceleration, angular velocity, and derived features), with a focus on capturing the linear acceleration channel. ) and acceleration channel ( The high-frequency pulse features in the image are then passed through dilated convolutional layers and residual connection layers. The residual structure directly transmits the original acceleration vector sequence information to the deep network to prevent long-distance sliding features from attenuating during deep feature extraction, thus improving the accuracy of subsequent action recognition. The expression is:

[0126]

[0127] in, For the first The input feature tensor of the layer (when) At that time, the input is the multidimensional dynamic feature tensor constructed above. ), For dilation convolution operators, For the first The kernel weights of the layer To modify the activation function of the linear unit, For the first The output features of a layer (i.e., the input of the next layer).

[0128] The temporal feature encoder ultimately outputs a high-dimensional temporal embedding vector. The temporal embedding vector encodes the type and intensity information of the action behavior, and its dimension is aligned with the output of the statistical feature encoder.

[0129] As an optional implementation, the context-aware gating network includes: a context-aware gating unit and a task decoupling gating unit;

[0130] The context-aware gating unit is used to perform temporal pooling on the ten-axis data vector to obtain the original physical context vector, and to concatenate the spatial embedding vector, the temporal embedding vector and the original physical context vector to obtain the gating input vector;

[0131] The task decoupling gating unit is used to process the gating input vector, calculate the finger task weight and the action task weight, and calculate the finger feature fusion vector and the action feature fusion vector based on the finger task weight and the action task weight.

[0132] Specifically, the ten-axis data vector within the time window is subjected to temporal pooling to obtain the original physical context vector. The spatial embedding vector, temporal embedding vector, and original physical context vector are then concatenated to obtain the gated input vector, expressed as:

[0133]

[0134]

[0135] in, For the original physical context vector, This is a time-domain average pooling operation used to extract macroscopic environmental features within a time window; Input data for the ten axes; This represents the average acceleration across the three axes. This represents the average angular velocity across the three axes. The mean of the attitude quaternions; This is the concatenated gating input vector; For spatial embedding vectors; This is a temporal embedding vector.

[0136] The task decoupling gating unit includes a finger task gating subunit and an action task gating subunit. The optimal target finger combination weights are calculated based on the finger task gating subunit, and the target finger feature fusion vector is calculated based on these weights. The calculation formula is as follows:

[0137] ;

[0138]

[0139] in, Finger task weights for the target finger recognition task ( For spatial flow weights, (for time-series flow weights) The trainable weight matrix for the target finger task gating network; The gating input vector; For bias vectors, The target finger feature fusion vector.

[0140] The optimal action behavior combination weights are calculated based on the action task gating subunit, and the action feature fusion vector is calculated based on the action behavior combination weights. The calculation formula is as follows:

[0141]

[0142]

[0143] in, Action task weights for action recognition tasks ( For spatial flow weights, (for time-series flow weights) For the trainable weight matrix of the action task gating network; For the gated input vector, It is the bias vector; This is the weighted fusion vector of action features.

[0144] It should be noted that in the finger task gating subunit, Spatial flow weights The temporal stream weights correspond to the trust weights of the statistical feature encoder and the temporal feature encoder, respectively. However, in the finger task gating subunit, the statistical feature encoder tends to be given a higher weight. This is because the gating subunit focuses more on the quaternion pose and inter-axis correlation to ensure the accuracy of finger spatial positioning; in the motion task gating subunit, Spatial flow weights The weights for the temporal stream also correspond to the trust weights for the statistical feature encoder and the temporal feature encoder, respectively. However, in the action task gating subunit, the temporal feature encoder tends to be given a higher weight. In particular, when a high-frequency Jerk signal is detected in the original physical context vector, it significantly amplifies dynamic flow features to capture subtle click or swipe textures.

[0145] As an optional implementation, the multi-task recognition module includes a finger classification head and an action classification head;

[0146] The finger classification head includes a fully connected layer and a Softmax normalization layer, which are used to process the finger feature fusion vector to obtain a three-dimensional probability vector, and output the target finger recognition result based on the three-dimensional probability vector;

[0147] The action classification head includes a fully connected layer and a Softmax normalization layer, which are used to process the action feature fusion vector to obtain a five-dimensional probability vector, and output the action type recognition result based on the five-dimensional probability vector.

[0148] Specifically, the target finger feature fusion vector is input into the finger classification head. Specifically, the target finger feature fusion vector passes through a fully connected layer and a Softmax normalization layer, ultimately outputting a three-dimensional probability vector, corresponding to the confidence scores of the index, middle, and ring fingers respectively. The index with the highest confidence score is used as the prediction result to obtain the Target_Finger_ID, expressed as:

[0149]

[0150]

[0151] in, The linear output of the fully connected layer for finger classification; The weight matrix for the finger classification head; The input is a fused vector of finger features; It is the bias vector; Let be the probability distribution of the target finger after Softmax normalization; These represent the confidence levels for the target finger being the index finger, middle finger, and ring finger, respectively.

[0152] Set global confidence threshold ,like This indicates that the current gesture feature is ambiguous (it may be in an intermediate state of finger switching), and outputs "None" without triggering any key mapping;

[0153] The action feature fusion vector is input into the action classification head. Specifically, the action feature fusion vector passes through a fully connected layer and a Softmax normalization layer, ultimately outputting a five-dimensional probability vector, corresponding to the confidence scores of click, inner short swipe, outer short swipe, inner long swipe, and outer long swipe, respectively. The index with the highest confidence score is used as the prediction result to obtain Action_Type, expressed as:

[0154]

[0155] in, Linear output of the fully connected layer for the action classification head; The trainable weight matrix for the action classification head; The input is a motion feature fusion vector; It is the bias vector; This represents the probability distribution of action types after Softmax normalization. This represents the probability of a click action. The probability of the inner short sliding action. The probability of the behavior being an external short slide. Let be the probability of the inner length sliding action. The probability of the behavior being a sliding motion of the outer length;

[0156] Different target fingers (index finger / middle finger / ring finger) correspond to different subsets of effective actions (e.g., some fingers are configured in full-function mode, supporting all swipes; while some fingers are configured in simplified mode, supporting only swipes or clicks in specific directions). The prediction results from the finger classification head are used to generate corresponding action mask vectors. This mask is applied to the output layer of the action classification head, forcibly zeroing the probability of undefined or unsupported action categories for that finger, and then re-normalizing the probabilities. This ensures that decisions are made only within the effective action space of the current finger, completely eliminating the possibility of misjudgments of illegal action combinations. Finally, a structured tuple containing discretized decision results and confidence levels is output, expressed as:

[0157]

[0158] in, For structured tuples, Target_Finger_ID is taken as... The index with the highest probability (e.g., Index_Finger); Action_Type is taken as... The index with the highest probability (e.g., Short_Slide_In); Confidence_Score is the joint confidence score.

[0159] The tuple will be directly transmitted as input to the next-level mapping strategy and key generation module, with a preset joint confidence threshold. If Target_Finger_ID is None or If the event is invalid, it will be discarded before transmission and will not enter the mapping process; if the event is valid, the mapping strategy and key generation module will convert the tuple into a specific keyboard key according to the preset mapping strategy (space-compatible mapping strategy or ergonomic optimal mapping strategy) and output the corresponding character for display.

[0160] As an optional implementation, the ergonomic optimal mapping strategy specifically includes:

[0161] The fatigue cost of each movement is calculated based on the finger dexterity factor and the movement difficulty factor.

[0162] A preset standard English character frequency table is provided, which divides characters into a high-frequency core character set, a mid-frequency commonly used character set, and a low-frequency edge character set.

[0163] Obtain the user's historical input logs, calculate the user's character distribution frequency based on the historical input logs, update the standard English character frequency table based on the character distribution frequency, and obtain the current character frequency table;

[0164] A greedy algorithm is used to traverse all available finger-action combinations to generate a character recommendation mapping table;

[0165] The character with the highest confidence level in the character recommendation mapping table is output and displayed as the target character.

[0166] Specifically, the core idea of ​​the ergonomic optimal mapping strategy is to construct a mechanical model that can quantify "input fatigue" and to reverse-match the "linguistic frequency" of character input with the "biomechanical cost" of hand movements in order to minimize global input energy consumption.

[0167] Specifically, for each available physical action (No. The first finger For each type of movement, a fatigue cost is defined, which is determined by the product of a finger dexterity factor and a movement difficulty factor: ;

[0168] in, The value of fatigue. As a factor for finger dexterity, This is the difficulty factor for the action;

[0169] The finger dexterity factor is set based on finger independence and the strength of the flexor and extensor muscles. For example, the index finger is 1.0 (baseline value, most flexible); the middle finger is 1.2 (slightly affected); and the ring finger is 1.6 (poor independence, high operating resistance).

[0170] The difficulty factor is based on the length of the movement and the amount of muscle work required, explicitly distinguishing the biomechanical differences between finger adduction and abduction. For example, click 1.0 (shortest path, only requires slight flexion, the easiest); short inward slide 1.3 (thumb naturally adducts, following muscle texture, less effort); short outward slide 1.5 (thumb needs to actively abduct against tendon tension, slightly more effort); long inward slide 2.4 (longer range, but easier direction); long outward slide 3.0 (long range and requires a large abduction, highest fatigue).

[0171] Preload a standard English character frequency table obtained from statistics of a large-scale general corpus as the initial state. The standard English character frequency table divides characters into a high-frequency core character set, a mid-frequency common character set, and a low-frequency edge character set.

[0172] Among them, the cumulative frequency of characters in the high-frequency core character set accounts for about 65%, covering vowels and the most commonly used consonants; including: 'e', ​​'t', 'a', 'o', 'i', 'n', 's', 'h'; the character mapping strategy in the high-frequency core character set is to prioritize mapping to low-cost actions (such as index / middle finger clicks, short inner swipes).

[0173] The cumulative frequency of characters in the mid-frequency commonly used character set accounts for about 30%, and most of them are word-forming auxiliary consonants; including: 'r', 'd', 'l', 'c', 'u', 'm', 'w', 'f', 'g', 'y'; the character mapping strategy in the mid-frequency commonly used character set is to map to actions with medium cost (such as a short swipe with the index finger or a click with the ring finger).

[0174] Characters in the low-frequency edge character set account for approximately 5% of the total frequency and appear only in specific words; these include: 'p', 'b', 'v', 'k', 'j', 'x', 'q', and 'z'. The character mapping strategy for the low-frequency edge character set is to map them to high-cost actions (such as long swipes with the ring finger or long swipes with the outside), sacrificing the comfort of a small amount of input in exchange for the compactness of the overall layout.

[0175] Because different users use words in different scenarios (such as coding or daily chatting), their word usage varies. Therefore, we obtain users' historical input logs and use a sliding window to calculate the historical frequency distribution of each character in real time based on these logs. Then, based on the frequency distribution of each character in the standard English character frequency table and the historical frequency distribution, we use a weighted moving average algorithm to dynamically update and generate the current character frequency of each character. Based on the current character frequency of each character, we generate a current character frequency table. The formula for calculating the current character frequency is:

[0176]

[0177] in, The current effective frequency of the calculated character k; This is a personalization factor (with a value range of [0, 1], used to adjust the weight of user habits). For preset universal frequencies in the standard English character frequency table; The cumulative number of times the user has entered the character 'k' within the most recent time window (e.g., the last 10,000 keystrokes); This represents the total number of keystrokes for all characters within the given time window.

[0178] It should be noted that the update of the standard English character frequency table based on the user's historical input log is not real-time, but triggered periodically. For example, when the accumulated new input reaches a preset threshold, or when a plateau is detected in the user's input efficiency, the updated current character frequency will be used as the input weight for the next generation of the character frequency table, thereby generating a key layout that is more in line with the user's current habits.

[0179] Next, a greedy algorithm is used to solve a combinatorial optimization problem. A double-sequence sorting and matching algorithm is used to obtain the global optimum. The "high-frequency characters" in the current character frequency table are assigned to the "low-cost actions" corresponding to the low fatigue cost, ensuring that the global input fatigue is minimized. A character recommendation mapping table is generated, specifically as follows:

[0180] Step S1: Traverse all available "finger-action" combinations, sort them in ascending order according to the fatigue value, and generate the preferred action sequence, expressed as:

[0181]

[0182] in, The optimal sequence of actions is arranged in ascending order of fatigue cost; This represents the 'finger-action' combination at the m-th position in the sequence; The fatigue value corresponding to this combination; The cost of the next combination (i.e., the sequence where the cost of the previous action is less than or equal to the cost of the next action); N is the total number of available action combinations;

[0183] For example, the preferred sequence of actions generated is: [index finger - click (1.0)], [middle finger - click (1.2)], [index finger - short inner swipe (1.3)] ..., [ring finger - long inner swipe (3.84)], [ring finger - long outer swipe (4.8)];

[0184] Step S2: Based on the current character frequency table, sort the 26 letters in descending order to generate a character frequency sequence. The expression is:

[0185]

[0186] in, This is a sequence of character frequencies arranged in descending order of current frequency. This represents the character at the m-th position in the sequence; This represents the current frequency of effectiveness for that character. The frequency of the next character (i.e., the sequence satisfies that the frequency of the previous character is greater than or equal to the frequency of the next character);

[0187] For example, the generated character frequency sequence is ['e', 't', 'a', 'o', 'i', 'n' ... 'z'];

[0188] Step S3: Map the action selection sequence and character frequency sequence accordingly, specifically: The m-th element in the array is directly mapped to The m-th element in: Generate a character recommendation mapping table;

[0189] For example, the resulting recommendation mapping table is shown in Table 1:

[0190] Table 1 Recommendation Mapping Table

[0191]

[0192] According to the aforementioned recommendation mapping table, for example, when a user clicks with their right index finger, the character 'e' is output; when the user clicks with their left index finger, the character 't' is output; when the user clicks with their right middle finger, the character 'a' is output; when the user clicks with their left middle finger, the character 'o' is output; when the user makes a short inward swipe with their right index finger, the character 'i' is output; when the user makes a short inward swipe with their left index finger, the character 'n' is output; and so on. When the target finger of the target hand performs the corresponding action according to the aforementioned recommendation mapping table, the corresponding character is output. The user performs the actions sequentially to obtain multiple characters, thus obtaining the corresponding text, which is then output and displayed.

[0193] In another embodiment, the user is allowed to fine-tune the finger dexterity factor and the movement difficulty factor according to their own physiological condition. For example, the finger dexterity factor can be adjusted as follows: if the user's ring finger has been injured and has extremely poor dexterity, the finger dexterity factor can be adjusted. From version 1.6 to 5.0, the algorithm will automatically "remove" the characters on that finger to other fingers when generating the character frequency table next time, retaining only extremely low-frequency characters (such as 'z', 'q').

[0194] Action difficulty factor adjustment: Users can mark specific actions as "disliked". For example, if a user dislikes "long swipe", the action difficulty factor of long swipe actions will be increased, forcing high-frequency characters to concentrate on "click" or "short swipe".

[0195] It should be noted that frequent changes to the keyboard layout can disrupt users' muscle memory; therefore, a "non-intrusive" update strategy has been adopted.

[0196] The latest personal frequency table is periodically used in the background. Perform double verification: use the currently used mapping table. Based on the latest character frequencies, calculate the current expected fatigue level; and regenerate the theoretically optimal mapping table. And calculate the theoretical expected fatigue level; calculate the optimized rate of return based on the current expected fatigue level and the theoretical expected fatigue level, using the following formula:

[0197]

[0198]

[0199]

[0200] in, The expected fatigue level of the current layout under new user habits; The latest statistical frequency of the character; This represents the currently used mapping relationship (i.e., the action corresponding to the character); This is the fatigue cost associated with the action; The expected fatigue level of the theoretically optimal layout under new habits; This represents the theoretically optimal mapping relationship generated based on the new frequency. To optimize the rate of return.

[0201] Preset optimized yield threshold Only when When the current layout is determined to be no longer suitable for the user's new input habits, an update process is triggered.

[0202] As an optional implementation method, please refer to the appendix. Figure 4 The space-compatible mapping strategy specifically includes:

[0203] The virtual keyboard is divided into a left mapping area and a right mapping area. Both the left and right mapping areas include three sub-areas, and each sub-area includes a certain number of characters.

[0204] Based on the number of characters in the sub-region, the valid actions of the target finger in the sub-region are defined, the corresponding characters are determined based on the valid actions, and a target finger-action-character mapping rule table is generated. Based on the target finger-action-character mapping rule table, after the target finger performs an action in the sub-region, the target character corresponding to the action is output and displayed.

[0205] The step of defining the effective actions of the target finger in the sub-region based on the number of characters in the sub-region includes:

[0206] If the number of characters in the sub-region is five, the center character in the sub-region is used as the reference key, and the reference key is used for the target finger to perform a click action; the adjacent keys of the center character are used for the target finger to perform a short swipe action; the distal keys of the center character are used for the target finger to perform a long swipe action.

[0207] If the number of characters in the sub-region is four, the center point of the middle two characters in the sub-region is used as the reference key. The adjacent keys of the center point are used for the target finger to perform a short slide operation; the keys at the far end of the center point are used for the target finger to perform a long slide operation.

[0208] If the number of characters in the sub-region is three, the center character in the sub-region is used as the reference key, and the reference key is used for the target finger to perform a click action; the adjacent keys of the center character are used for the target finger to perform a long swipe action.

[0209] When the number of characters in the sub-region is two, the center point of the two characters in the sub-region is used as the reference key, and the two characters are used for the target finger to perform a long sliding operation.

[0210] When the number of characters in the sub-region is one, the characters in the sub-region are used by the target finger to perform a click action.

[0211] Specifically, the standard keyboard layout is logically broken down and mapped to three action states of the target fingers (index, middle, and ring fingers) of the left and right hands. The mapping positions are consistent with the relative positions of the real keyboard (e.g., R above F, V below F). The virtual keyboard is divided into a left mapping area and a right mapping area. The left mapping area is assigned to the three fingers of the left hand, and the right mapping area is assigned to the three fingers of the right hand. Since different fingers have different dexterity, different fingers are assigned to different numbers of keys. Then, based on the total number of logical keys assigned to each finger... ( The system automatically matches the optimal action combination scheme. In this invention, both the left and right mapping areas include three sub-regions, each containing a certain number of characters. In the left mapping area, the three sub-regions from top to bottom include QWERT, ASDFG, and ZXCV; in the right mapping area, the three sub-regions from top to bottom include YUIOP, HJKL, and BNM. The mapping principle is to prioritize retaining actions with high distinguishability (such as long swipes) and eliminate actions prone to accidental touches (such as clicks under high density) to achieve a dynamic balance between accidental touch rate and input efficiency.

[0212] Specifically, clicking the base key maps to the middle row of the actual keyboard; a short swipe to the left maps to the adjacent left key, corresponding to the adjacent left position on the actual keyboard; a short swipe to the right maps to the adjacent right key, corresponding to the adjacent right position on the actual keyboard; a long swipe to the left maps to the far left extended key, corresponding to the left key that requires a large finger movement; and a long swipe to the right maps to the far right extended key, corresponding to the right key that requires a large finger movement.

[0213] The specific mapping rule parsing table is shown in Table 2:

[0214] Table 2 Mapping Rule Parsing Table

[0215]

[0216] Specifically: the left hand is responsible for QWERT, ASDFG, and ZXCV, and the right hand is responsible for YUIOP, HJKL, and BNM. The mapping tables for the target fingers of the left and right hands are shown in Table 3 and Table 4, respectively.

[0217] Table 3 Mapping table of target fingers for the left hand

[0218]

[0219] Table 4 Mapping table of target fingers for the right hand

[0220]

[0221] It should be noted that in the mapping area of ​​the right ring finger, only click and inner long swipe and outer long swipe actions are available. This is because the ring finger has the worst dexterity and it is difficult to accurately control the 'short swipe'. Although the physical distance between (N) and (B / M) is close, in order to prevent accidental touches and reduce the difficulty of operation, the algorithm forces it to be mapped as a long swipe. This is a special design based on ergonomics. Therefore, for the right ring finger, the 'short swipe' is discarded, and only the 'click' (N) and the 'long swipe' (B / M) with the highest distinguishability are retained.

[0222] The mapping area for the left ring finger differs from that of the right ring finger. This is because the left ring finger mapping area controls four characters. When the number of characters is even, a neutral point (click position) is left in the air, and 'short swipe' and 'long swipe' are retained, making it easier for the fingers to operate.

[0223] According to the mapping table above, for example, when the user uses the index finger of their left hand to click, the character 'E' is output; when the user uses the index finger of their left hand to make a short inward swipe, the character 'W' is output; when the user uses the index finger of their left hand to make a short outward swipe, the character 'R' is output; when the user uses the index finger of their left hand to make a long inward swipe, the character 'Q' is output; when the user uses the index finger of their left hand to make a long outward swipe, the character 'T' is output; and so on. When the target finger of the target hand performs the corresponding action according to the mapping table above, the corresponding character is output. The user performs the actions in sequence to obtain multiple characters, thereby obtaining the corresponding text, which is then output and displayed.

[0224] Example 2

[0225] Please refer to the attached document. Figure 5 This invention provides a virtual scene interaction method based on an inertial measurement unit, comprising the following steps:

[0226] The raw ten-axis data vector is acquired based on a miniature inertial measurement unit, and the raw ten-axis data vector is preprocessed to obtain a ten-axis data vector including acceleration vector, angular velocity vector and attitude quaternion vector;

[0227] The ten-axis data vector is input into the statistical feature encoder and the temporal feature encoder respectively for feature extraction to obtain the spatial embedding vector and the temporal embedding vector.

[0228] Based on the fusion of the spatial embedding vector, the temporal embedding vector, and the ten-axis data vector, a finger feature fusion vector and a motion feature fusion vector are obtained.

[0229] Based on the finger feature fusion vector and the action feature fusion vector, the target finger and action type are identified;

[0230] The user selects a mapping strategy, which maps the target finger and action type based on the recognition, outputs the corresponding target character, and displays it.

[0231] The above are merely specific embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. The scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A virtual scene interaction system based on an inertial measurement unit, characterized in that, include: The preprocessing module is used to collect raw ten-axis data vectors from the micro inertial measurement unit and preprocess them to obtain ten-axis data vectors. The ten-axis data vectors include a three-axis linear acceleration vector after gravity removal, a filtered three-axis angular velocity vector, and an attitude quaternion vector. Two micro inertial measurement units are provided and worn on the dorsal side of the proximal phalanx of the left and right thumbs, respectively. The feature extraction module is used to input the ten-axis data vector into the statistical feature encoder and the temporal feature encoder respectively to extract features and obtain a spatial embedding vector. and temporal embedding vector ; The feature fusion module is used to input the spatial embedding vector, the temporal embedding vector and the ten-axis data vector into the context-aware gating network to obtain the finger feature fusion vector and the action feature fusion vector. The context-aware gating network includes a context-aware gating unit and a task decoupling gating unit; the context-aware gating unit is used to perform temporal pooling processing on the ten-axis data vector to obtain the original physical context vector. The spatial embedding vector, the temporal embedding vector, and the original physical context vector are concatenated to obtain the gated input vector. The task decoupling gating unit processes the gated input vector, calculates the finger task weight and the action task weight, and calculates the finger feature fusion vector and the action feature fusion vector based on the finger task weight and the action task weight. The calculation formula is as follows: ； in, For the finger task weights of the target finger recognition task, For spatial flow weights, For time-series flow weights; The trainable weight matrix for the target finger task gating network; The gating input vector; For bias vectors, The target finger feature fusion vector; in, For action recognition tasks, the action task weights are... For spatial flow weights, For time-series flow weights; For the trainable weight matrix of the action task gating network; For the gated input vector, It is the bias vector; This is the weighted fusion vector of action features; The multi-task recognition module is used to input the finger feature fusion vector into the finger classification head to identify the target finger, and input the action feature fusion vector into the action classification head to identify the action type, thereby obtaining the joint confidence of the target finger and the action type; The mapping strategy and key generation module is used to map the identified target finger and action type to the virtual keyboard according to the joint confidence and the selected mapping strategy, output the corresponding target characters and display them; the mapping strategy includes a spatially compatible mapping strategy and an ergonomically optimal mapping strategy. The optimal ergonomic mapping strategy specifically includes: The fatigue cost of each movement is calculated based on the finger dexterity factor and the movement difficulty factor. A preset standard English character frequency table is provided, which divides characters into a high-frequency core character set, a mid-frequency commonly used character set, and a low-frequency edge character set. Obtain the user's historical input logs, calculate the user's character distribution frequency based on the historical input logs, update the standard English character frequency table based on the character distribution frequency, and obtain the current character frequency table; A greedy algorithm is used to traverse all available finger-action combinations to generate a character recommendation mapping table; The character with the highest confidence level in the character recommendation mapping table is output and displayed as the target character. The space-compatible mapping strategy specifically includes: The virtual keyboard is divided into a left mapping area and a right mapping area. Both the left and right mapping areas include three sub-areas, and each sub-area includes a certain number of characters. Based on the number of characters in the sub-region, the valid actions of the target finger in the sub-region are defined, the corresponding characters are determined based on the valid actions, and a target finger-action-character mapping rule table is generated. Based on the target finger-action-character mapping rule table, after the target finger performs an action in the sub-region, the target character corresponding to the action is output and displayed.

2. The virtual scene interaction system based on an inertial measurement unit according to claim 1, characterized in that, The finger classification head includes a fully connected layer and a Softmax normalization layer, which are used to process the finger feature fusion vector to obtain a three-dimensional probability vector, and output the target finger recognition result based on the three-dimensional probability vector; the three-dimensional probability vectors correspond to the confidence levels of the target finger currently being contacted by the thumb as the index finger, middle finger, and ring finger, respectively. The action classification head includes a fully connected layer and a Softmax normalization layer, which are used to process the action feature fusion vector to obtain a five-dimensional probability vector. The action type recognition result is output based on the five-dimensional probability vector. The five-dimensional probability vector corresponds to the confidence scores of the action behaviors performed between the thumb and the target finger: click, short inner swipe, short outer swipe, long inner swipe, and long outer swipe.

3. The virtual scene interaction system based on an inertial measurement unit according to claim 1, characterized in that, The method employs a greedy algorithm to traverse all available finger-action combinations and generate a character recommendation mapping table, including: Step S1: Traverse all available "finger-action" combinations, sort them in ascending order according to the fatigue value, and generate the preferred action sequence; Step S2: Arrange the twenty-six letters in descending order according to the current character frequency table to generate a character frequency sequence; Step S3: Generate a character recommendation mapping table by mapping the preferred action sequence and the character frequency sequence.

4. A virtual scene interaction system based on an inertial measurement unit according to claim 1, characterized in that, The step of defining the effective actions of the target finger in the sub-region based on the number of characters in the sub-region includes: If the number of characters in the sub-region is five, the center character in the sub-region is used as the reference key, and the reference key is used for the target finger to perform a click action; the adjacent keys of the center character are used for the target finger to perform a short swipe action; the distal keys of the center character are used for the target finger to perform a long swipe action. If the number of characters in the sub-region is four, the center point of the middle two characters in the sub-region is used as the reference key. The adjacent keys of the center point are used for the target finger to perform a short slide operation; the keys at the far end of the center point are used for the target finger to perform a long slide operation. If the number of characters in the sub-region is three, the center character in the sub-region is used as the reference key, and the reference key is used for the target finger to perform a click action; the adjacent keys of the center character are used for the target finger to perform a long swipe action. When the number of characters in the sub-region is two, the center point of the two characters in the sub-region is used as the reference key, and the two characters are used for the target finger to perform a long sliding operation. When the number of characters in the sub-region is one, the characters in the sub-region are used by the target finger to perform a click action.

5. A virtual scene interaction method based on an inertial measurement unit, the method being used to implement a virtual scene interaction system based on an inertial measurement unit as described in any one of claims 1-4, characterized in that, Includes the following steps: The raw ten-axis data vector is acquired based on a miniature inertial measurement unit, and the raw ten-axis data vector is preprocessed to obtain a ten-axis data vector including acceleration vector, angular velocity vector and attitude quaternion vector; The ten-axis data vector is input into the statistical feature encoder and the temporal feature encoder respectively for feature extraction to obtain the spatial embedding vector and the temporal embedding vector. Based on the spatial embedding vector, the temporal embedding vector, and the ten-axis data vector, a finger feature fusion vector and a motion feature fusion vector are obtained. Based on the finger feature fusion vector and the action feature fusion vector, the target finger and action type are identified; The user selects a mapping strategy, which maps the target finger and action type based on the recognition, outputs the corresponding target character, and displays it.

Images