Smart phone back handwritten letter input method integrating IMU and UWB

Abstract

The invention discloses an IMU (Inertial Measurement Unit) and UWB (Ultra Wideband) integrated smart phone back handwritten letter input method, which comprises the following steps: acquiring IMU sequence data and UWB ranging data in two devices through API (Application Program Interface) interfaces of a smart phone and a smart watch; aligning the data in time and space, and acquiring relative motion characteristics of the two devices; analyzing the influence of different features in the relative motion features of the two devices on the recognition accuracy of the handwritten letters, selecting a required feature combination according to the influence, and carrying out normalization processing; and constructing a handwritten letter recognition learning model based on LSTM and TCN, and recognizing handwritten letters input on the back of the smart phone according to real-time IMU sequence data and UWB ranging data of the smart phone and the smart watch. According to the invention, a novel interaction mode of data input at the back of the mobile phone is realized, so that a man-machine interaction expansion mode is realized.

Description

Technical Field

[0001] The present invention relates to the field of human-computer interaction, and in particular to a method for inputting handwritten letters on the back of a smart phone by integrating IMU and UWB. Background Art

[0002] Human-Computer Interaction (HCI) is an interdisciplinary subject that studies the information transmission and collaborative work between people and computer systems. Its purpose is to improve the user's operating efficiency, comfort and experience by optimizing technical design. It integrates knowledge from computer science, psychology, design, cognitive science and other fields. Its core goal is to make technology more in line with human natural behavior patterns and lower the threshold for use.

[0003] Common ways to implement human-computer interaction include: 1. Graphical user interface, based on visual elements (windows, icons, menus, buttons, etc.), users directly operate through devices such as mice and touch screens. For example, the touch interaction of smartphones and the visual operation of desktop systems reduce the difficulty of using technology. 2. Voice interaction, using natural language processing (NLP) technology to achieve voice dialogue between people and machines, such as smart speakers and in-car voice assistants. Users control devices through voice commands, which is suitable for multi-task scenarios or barrier-free needs. 3. Gesture and somatosensory interaction, capturing user movements through cameras and sensors, such as VR handles, Microsoft Kinect or gesture recognition screens. Users can interact with the virtual environment by waving their hands and moving their bodies to enhance immersion. 4. Virtual and augmented reality (VR / AR), VR builds a virtual space through head-mounted display devices, and users can explore freely; AR superimposes digital information on the real world, and is applied to education, medical and other fields to expand the interaction dimension. 5. Wearable devices, smart watches, AR glasses and other devices achieve seamless interaction through biosensors (heart rate, motion monitoring) or environmental perception, emphasizing real-time and convenience. 6. Brain-computer interface (BCI), a cutting-edge technology that directly controls external devices by interpreting EEG signals, helping people with disabilities or achieving more efficient command transmission. It is still in the experimental stage but has great potential.

[0004] The most common scenario of human-computer interaction is the interaction between people and smartphones. The interaction between people and smartphones can be implemented in a variety of ways. Almost all of the common human-computer interaction methods introduced above can be applied to the interaction between people and mobile phones. The diverse interaction methods make the interaction between people and mobile phones very flexible and the human-computer interaction experience is also very good. However, the various interaction methods of the above applications are often face-to-face, that is, positive interaction.

[0005] If it is possible to realize the interaction between people and the back of the phone, it will certainly greatly enrich the use scenarios of human-computer interaction and bring more human-computer interaction methods. In addition, human-computer interaction on the back of the phone has the following benefits: 1. Enriching operation types: For example, the same swipe-up operation can be designed into different functions on the front and back of the phone according to different application requirements, and can also be applied to shortcut keys. 2. Protect personal privacy: When operating on the front of the phone in public places, others may guess what you input based on the input gestures, but there will be no such problem when operating on the back. 3. Increase the visible area: Placing the key input on the back can effectively reduce the reduction in visible area caused by the occlusion of the keyboard and hand. 4. Improve one-handed operation: For users of large-screen mobile phones, the touch area on the back is easier to reach when operating with one hand (such as the index finger naturally placed on the back of the phone), reducing the fatigue of overstretching the thumb.

[0006] There are also some studies on the human-computer interaction on the back of the phone, such as waking up some shortcut functions of the phone through some specific gestures (such as tapping the back of the phone with knuckles, etc.). However, this interaction method is too simple and the content that can be interacted is very limited. Summary of the invention

[0007] The present invention aims to solve one of the technical problems existing in the related art at least to a certain extent.

[0008] The purpose of the present invention is to provide a method for inputting handwritten letters on the back of a smartphone by integrating IMU and UWB, so as to realize handwriting recognition and input on the back of a smartphone, expand a new human-computer interaction mode, and enhance the interactive experience.

[0009] In order to achieve the above-mentioned purpose, the present invention provides a method for inputting handwritten letters on the back of a smartphone by integrating IMU and UWB, comprising: S1. Build a communication framework between smartphones and smart watches, and obtain IMU sequence data and UWB ranging data in the two devices through the API interfaces of smartphones and smart watches; S2, preprocessing the IMU sequence data and UWB ranging data in the two devices to align the data in time and space, and obtain the relative motion characteristics of the two devices; S3, analyzing the influence of different features in the relative motion features of the two devices on the recognition accuracy of handwritten letters, and then selecting the required feature combination based on the analysis, and performing normalization processing; S4. Build a handwritten letter recognition learning model based on LSTM and TCN, and use the feature combination extracted from IMU sequence data and UWB ranging data for training; S5. Load the trained handwritten letter recognition learning model into a smartphone and / or smart watch, and recognize the handwritten letters input on the back of the smartphone based on the real-time IMU sequence data and UWB ranging data of the smartphone and the smart watch.

[0010] A further preferred technical solution of the present invention is that, in step S2, the IMU sequence data and the UWB ranging data in the two devices are preprocessed to align the data in time and space, and obtain the relative motion characteristics of the two devices. The specific method is: S21, truncating the IMU sequence data of the smart phone and the smart watch at the beginning and the end, and interpolating the UWB ranging data, so that the time of the two IMU sequence data and the UWB ranging data is aligned; S22, setting a global coordinate system, and mapping the acquired posture information of the smart phone and the smart watch to the coordinate system; S23. Construct a relative motion feature set of the two devices, including UWB ranging data, various relative motion features of the smartphone and the smart watch in their respective local coordinate systems, and various relative motion features of the smart watch in the smartphone coordinate system.

[0011] Preferably, in step S21, the method for truncating the IMU sequence data of the smart phone and the smart watch is: The IMU sequence data of the smartphone and smartwatch are cut, and only the data between the previous timestamp of the first timestamp of the UWB ranging data and the next timestamp of the last timestamp of the UWB ranging data are retained; In step S21, the method for aligning the time of two IMU sequence data is as follows: Assume that the update frequency of the IMU serial data of the smartphone and smartwatch is , when the time interval between the timestamp of the smartphone’s IMU serial data and the timestamp of the smartwatch’s IMU serial data is less than When , the middle moment of the two timestamps is taken as the alignment moment, and the corresponding data are the IMU sequence data of the smartphone and smartwatch close to the alignment moment.

[0012] Preferably, in step S21, the specific method for interpolating the UWB ranging data to align the time of the two IMU sequence data with the UWB ranging data is: Assume that the update frequency of UWB ranging data is , taking the timestamp of the IMU serial data of the smart watch as a reference, determining the timestamp of the IMU serial data of the smart watch close to a certain timestamp of the UWB ranging data, and combining the two with the IMU serial data of the corresponding timestamp of the smart phone to form a timestamp data group; At this time, some of the IMU sequence data with time stamps do not have corresponding UWB ranging data. The UWB ranging data of two adjacent IMU sequence data with corresponding IMU sequence data are averaged and linearly interpolated to fill the missing UWB ranging data.

[0013] Preferably, step S22 sets a global coordinate system, and corresponds the acquired posture information of the smart phone and the smart watch to the coordinate system, and the specific method is: First, set up a global coordinate system with the X-axis pointing to the geographic North Pole, the Z-axis pointing vertically upward, and the Y-axis obtained by the right-hand rule; The rotation matrix corresponding to the smart watch is inverted and then multiplied by the rotation matrix corresponding to the smartphone to obtain the rotation matrix from the smart watch coordinate system to the smartphone coordinate system, expressed as:

[0014] in, To directly obtain the rotation matrix from the global coordinate system to the smartphone coordinate system, To directly obtain the rotation matrix from the global coordinate system to the smartwatch coordinate system, is the calculated rotation matrix from the smartwatch coordinate system to the smartphone coordinate system; T is the transpose operation of the matrix, which is equivalent to taking the inverse matrix.

[0015] Preferably, in the relative motion feature set of the two devices constructed in step S23, some features are directly obtained from the IMU sequence data and the UWB ranging data, and some features are obtained by calculating the rotation matrix from the smart watch coordinate system to the smartphone coordinate system in combination with the data groups of each timestamp.

[0016] Preferably, in step S3, the influence of different features of the relative motion features of the two devices on the recognition accuracy of handwritten letters is analyzed; the specific method is: S31. Conduct a preliminary experiment to train different models for different features. During the training, 26 letters are used as labels, 26 labels are encoded using one-hot encoding, categorical cross entropy is used as the loss function, and early stopping method is used for training; S32. Select the required features based on the performance of different models trained with different features in the preliminary experiment on the validation set and the training set before overfitting. In the process of early stopping method training data, when a new minimum loss value appears, if there is no new minimum loss smaller than the previous one within 3 rounds of training, it is considered to be overfitting.

[0017] Preferably, in step S3, the specific method for selecting the required feature combination and performing normalization processing is: S33, selecting the UWB distance, the acceleration of the smart watch in the smart phone coordinate system, and the rotation matrix from the smart watch to the smart phone as a feature combination; S34, fill the sequence length of each sample with 0.0 to 500; S35. Use MinMaxScaler of Python's sklearn.preprocessing module to normalize the data. The formula is as follows:

[0018] in, X is the current value of a feature, is the minimum value of the feature in the current sample, is the maximum value of the feature in the current sample; is a user-defined range to restrict data to, is its maximum value, is its minimum value, and the default range is (0,1).

[0019] Preferably, the handwritten letter recognition learning model based on LSTM and TCN constructed in step S4 has a network architecture including: The Masking layer is used to mask the 0.0 filled in the sample; assuming the sample is , whose shape is (batch_size, timesteps, features), the Masking layer generates a mask matrix M by checking whether the value of each time step is equal to the specified specific value. The shape of the matrix is ​​(batch_size, timesteps), and its elements are:

[0020] in, i is the sample index in the batch, j is the time step index; Bidirectional TCN network is used for data convolution; the size of the convolution kernel is set to 6, the number of filters used in the convolution layer is 128, and each layer uses skip residual connection; there is no causal relationship when setting convolution, that is, the data of the previous timestamp is not used to predict the data of the later timestamp; the expansion list is set to [1, 2, 4, 8]; the data of all timestamps is set to be returned; finally, the loss ratio of the input data of the previous layer is set to 0.1; A unidirectional LSTM network is used to transform the data sequence into 128 dimensions and output the data of the last timestamp; There are two fully connected layers, where the number of units in the first fully connected layer is 64, the activation function is ReLU, and the calculation formula is:

[0021] in, is the input value after linear transformation, which is the 1×128 data output by the unidirectional LSTM network, multiplied by a 128×64 weight matrix, and then added with a 1×64 bias term; The number of units in the second fully connected layer is 26, the activation function is Softmax, and the calculation formula is:

[0022] in, The first column vector is obtained by multiplying the 1×64 data output by the first fully connected layer by a 64×26 weight matrix and adding a 1×26 bias term. i Row data, K is the total number of categories, is the index, The index corresponding to j Row data.

[0023] On the other hand, the present invention provides a non-transitory computer-readable storage medium having computer instructions stored thereon, which enable a computer to execute the above-mentioned method for inputting handwritten letters on the back of a smartphone that integrates IMU and UWB.

[0024] Another aspect of the present invention provides an electronic device, comprising: a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus, and the processor calls the logic instructions in the memory to execute the above-mentioned method of handwritten letter input on the back of a smartphone that integrates IMU and UWB.

[0025] On the other hand, the present invention provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer executes the above-mentioned method for inputting handwritten letters on the back of a smartphone that integrates IMU and UWB.

[0026] Beneficial effect: The method for inputting handwritten letters on the back of a smartphone integrating IMU and UWB of the present invention extracts the IMU and UWB sensor data from commercial mobile phones and wearable watch devices, and fuses the data of the two, integrates the LSTM and TCN network modules in deep learning, and realizes a new interactive mode of data input on the back of the mobile phone, thereby realizing a way to expand human-computer interaction, which can effectively avoid the hand or finger blocking part of the screen when handwriting on the front, resulting in discontinuous browsing, and the left and right fingers occupying each other's space when multiple sliding or pinyin input is required. In addition, it solves the problem that the watch input keyboard is relatively small relative to the fingers and is inconvenient to input. Finally, it can play a role when the front screen of the mobile phone is in poor contact or the fingers are stained and it is inconvenient to wipe after handwriting on the front.

[0027] The present invention realizes a lightweight dynamic handwritten letter recognition. Compared with the traditional handwriting recognition (HWR), which needs to rely on computers to analyze and process images of handwritten text, the present invention does not need to recognize images, nor does it use cameras to obtain user privacy. It dynamically receives and processes data to realize dynamic handwriting recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 This is an overall flow chart of the method for inputting handwritten letters on the back of a smartphone integrating IMU and UWB according to the present invention; Figure 2 is the confusion matrix of the model on the training set when the training is stopped in Example 1 of the present invention; Figure 3 It is the confusion matrix of the model in Example 1 of the present invention on the test set. DETAILED DESCRIPTION

[0029] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings in the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments, and they should not be understood as limitations on the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention. In the description of the present invention, it should be understood that the terms used are only for descriptive purposes and cannot be understood as indicating or implying relative importance.

[0030] Before describing the specific embodiments of the present invention in detail, some technical terms involved in the present invention need to be explained.

[0031] IMU, short for Inertial Measurement Unit, is an electronic device that measures and reports the three basic linear motions (acceleration) and three basic angular motions (angular velocity) of an object.

[0032] UWB ranging (ultra-wideband ranging) is a high-precision distance measurement method based on ultra-wideband wireless communication technology. Its core principle is to determine the distance by calculating the flight time (TOF) of the signal between two devices multiplied by the speed of light. For example, device A sends a ranging request and records the sending timestamp (T1), device B receives it and records the timestamp (T2) and replies, and device A then records the timestamp of the received response (T3). By calculating the time difference (such as T3-T1-T2-T1), the round-trip time of the signal is obtained, and then the one-way flight time is calculated.

[0033] An API (Application Programming Interface) is a set of predefined rules, protocols, and tools for interaction between different software systems or components. It defines how to request services, pass data, and obtain results, while hiding the underlying implementation details so that developers can call functions without having to worry about the internal logic.

[0034] LSTM (Long Short-Term Memory) is a deep learning model commonly used to process sequence data. Compared with traditional RNN (recurrent neural network), LSTM introduces three gates (input gate, forget gate, output gate) and a cell state. These mechanisms enable LSTM to better handle long-term dependencies in sequences.

[0035] TCN (Temporal Convolutional Network) is a deep learning model specifically designed for processing sequence data. It combines the parallel processing capabilities of convolutional neural networks (CNN) and the long-term dependency modeling capabilities of recurrent neural networks (RNN), making it a powerful tool in sequence modeling tasks.

[0036] Handwriting Recognition (HWR) refers to the technology that converts handwritten text into machine-readable text or data by analyzing and processing the image of the handwritten text.

[0037] Combine the following Figure 1-Figure 3 The present invention describes a method for inputting handwritten letters on the back of a smart phone by integrating IMU and UWB.

[0038] Embodiment 1: This embodiment provides a method for inputting handwritten letters on the back of a smartphone by integrating IMU and UWB. The overall process of the method is as follows: Figure 1 As shown, including: S1. Build a communication framework between smartphones and smart watches, and obtain IMU sequence data and UWB ranging data in the two devices through the API interfaces of smartphones and smart watches; S2, preprocessing the IMU sequence data and UWB ranging data in the two devices to align the data in time and space, and obtain the relative motion characteristics of the two devices; S3, analyzing the influence of different features in the relative motion features of the two devices on the recognition accuracy of handwritten letters, and then selecting the required feature combination based on the analysis, and performing normalization processing; S4. Build a handwritten letter recognition learning model based on LSTM and TCN, and use the feature combination extracted from IMU sequence data and UWB ranging data for training; S5. Load the trained handwritten letter recognition learning model into a smartphone and / or smart watch, and recognize the handwritten letters input on the back of the smartphone based on the real-time IMU sequence data and UWB ranging data of the smartphone and the smart watch.

[0039] Each step is described in detail below.

[0040] In step S1, the Nearby Interaction and CoreMotion frameworks are used to obtain the UWB ranging data of the smartphone and smartwatch, as well as the IMU sequence data of the smartphone and smartwatch. The specific method is as follows: S11. Create a WCSession object for each smartphone and smart watch. Use the activate() method of each object to activate the connection between the smartphone and the smart watch. Use the isReachable property to determine whether the connection is successful.

[0041] S12. Create a NISession object for each smartphone and smartwatch, obtain their corresponding discoveryTokens, and use the sendMessage method of the WCSession object to send the discoveryToken of the NISession to each other. After successful reception, configure NIConfiguration and set the value of isCameraAssistanceEnabled to false. Use the run method of the NISession object to start UWB ranging and obtain the distance through the distance object of NINearbyObject. The principle of obtaining the distance is TOF, and the calculation formula is as follows:

[0042] d is the calculated distance, is the timestamp of the transmitted pulse, is the timestamp of receiving the reflected pulse, and c is the speed of light.

[0043] S13. Create a CMDeviceMotion object for each smartphone and smartwatch, set the deviceMotionUpdateInterval property to 0.02 seconds, and use the startDeviceMotionUpdates method to obtain the acceleration of the three axes, the angular velocity of the three axes, the Euler angle corresponding to the posture information, and the rotation matrix (set the global frame to xTrueNorthZVertical).

[0044] S14. Write the data into phone_uwb.txt, phone_imu.txt, watch_uwb.txt, and watch_imu.txt.

[0045] Step S2 pre-processes the IMU sequence data and UWB ranging data in the two devices to align the data in time and space and obtain the relative motion characteristics of the two devices. The specific method is: S21. Since the IMU sequence data is collected before the UWB ranging data is collected and ends after the UWB ranging data is obtained, it is necessary to cut the IMU sequence data of the smartphone and the smartwatch, and only retain the data between the previous timestamp of the first timestamp of the UWB ranging data and the next timestamp of the last timestamp of the UWB ranging data; Interpolate the UWB ranging data to align the time of the two IMU sequence data with the UWB ranging data. The method is as follows: take the timestamp of the IMU sequence data of the watch as the benchmark, and take one of the timestamps A as an example. Since the update frequency of the IMU of the mobile phone watch is 0.02 seconds, if the timestamp of the IMU sequence data of a mobile phone is B, and the time interval between A and B is less than 0.01 seconds, then take the middle moment of A and B, and the corresponding data comes from the data of the mobile phone watches corresponding to A and B. For the UWB ranging data, take the timestamp of the IMU sequence data of the watch as the benchmark, see which two adjacent watch IMU timestamps a certain timestamp of the UWB ranging data falls between, and determine which timestamp is closer, and combine the two with the IMU sequence data of the corresponding timestamp of the smartphone to form a timestamp data group.

[0046] Since the update frequency of UWB ranging data is about 0.15 seconds and the update frequency of the watch IMU sequence is 0.02 seconds, some IMU sequence data do not have corresponding UWB ranging data to match. In this way, two adjacent UWB ranging data with corresponding IMU sequence data are used for average linear interpolation to fill in the missing UWB ranging data.

[0047] S22, setting a global coordinate system, and mapping the acquired posture information of the smart phone and the smart watch to the coordinate system, the specific method is: First, set up a global coordinate system with the X-axis pointing to the geographic North Pole, the Z-axis pointing vertically upward, and the Y-axis obtained by the right-hand rule; The rotation matrix corresponding to the smart watch is inverted and then multiplied by the rotation matrix corresponding to the smartphone to obtain the rotation matrix from the smart watch coordinate system to the smartphone coordinate system, expressed as:

[0048] in, To directly obtain the rotation matrix from the global coordinate system to the smartphone coordinate system, To directly obtain the rotation matrix from the global coordinate system to the smartwatch coordinate system, is the calculated rotation matrix from the smartwatch coordinate system to the smartphone coordinate system; T is the transpose operation of the matrix, which is equivalent to taking the inverse matrix.

[0049] S23. Construct a relative motion feature set of the two devices, including UWB ranging data, various relative motion features of the smartphone and smartwatch in their own local coordinate systems, and various relative motion features of the smartwatch in the smartphone coordinate system. In the relative motion feature set of the two devices constructed here, some features are directly obtained from the IMU sequence data and UWB ranging data, and some features are obtained through the data groups of each time stamp combined with the rotation matrix calculation from the smartwatch coordinate system to the smartphone coordinate system, for example: Multiply the acceleration of the smartwatch by the previous rotation matrix to get the acceleration of the smartwatch in the smartphone coordinate system.

[0050] Subtract the acceleration of the smartwatch in the smartphone coordinate system from the acceleration of the smartphone in its own coordinate system to get the acceleration of the smartwatch relative to the smartphone. It can be expressed as:

[0051]

[0052] In the above formula is the acceleration of the smartphone in its own coordinate system, is the calculated acceleration of the smartwatch in the smartphone coordinate system, is the acceleration of the smartphone in its own coordinate system, and a is the calculated acceleration of the smartwatch in the smartphone coordinate system.

[0053] The acceleration in the local coordinate system of the watch and the acceleration in the coordinate system of the mobile phone are integrated to obtain the velocity, and the initial velocity is set to 0. The three-dimensional velocity is normalized to obtain the direction of motion. It can be expressed as:

[0054]

[0055] The specific method of step S3 is: S31. Conduct a preliminary experiment to train different models for different features. During the training, 26 letters are used as labels, 26 labels are encoded using one-hot encoding, categorical cross entropy is used as the loss function, and early stopping method is used for training; S32. Select the required features based on the performance of different models trained with different features in the preliminary experiment on the validation set and the training set before overfitting. In the process of early stopping method training data, when a new minimum loss value appears, if there is no new minimum loss smaller than the previous one within 3 rounds of training, it is considered to be overfitting.

[0056] S33, selecting the UWB distance, the acceleration of the smart watch in the smart phone coordinate system, and the rotation matrix from the smart watch to the smart phone as a feature combination; S34, fill the sequence length of each sample with 0.0 to 500; S35. Use MinMaxScaler of Python's sklearn.preprocessing module to normalize the data. The formula is as follows:

[0057] in, X is the current value of a feature, is the minimum value of the feature in the current sample, is the maximum value of the feature in the current sample; is a user-defined range to restrict data to, is its maximum value, is its minimum value, and the default range is (0,1).

[0058] The network architecture of the handwritten letter recognition learning model based on LSTM and TCN constructed in step S4 includes: The Masking layer uses the Masking layer in tensorflow's keras to mask the 0.0 filled in the sample; the Masking layer detects a specific value (usually 0) in the input tensor and marks it as "invalid" or "masked". These marks are passed to subsequent model layers to tell them which time steps should be ignored.

[0059] Assume that the sample is , whose shape is (batch_size, timesteps, features), the Masking layer generates a mask matrix M by checking whether the value of each time step is equal to the specified specific value. The shape of the matrix is ​​(batch_size, timesteps), and its elements are:

[0060] in, i is the sample index in the batch, j is the time step index; Bidirectional TCN network is used for data convolution; the size of the convolution kernel is set to 6, and the number of filters used in the convolution layer is 128, where the number of filters is equivalent to the number of units in LSTM; each layer uses skip residual connection; there is no causal relationship when setting convolution, that is, the data of the subsequent timestamp is not predicted by the data of the previous timestamp; the expansion list is set to [1, 2, 4, 8], that is, in the process of convolution, the timestamp data used are adjacent, or separated by 1, 3, or 7; all timestamp data are set to be returned; finally, the loss ratio of the input data of the previous layer is set to 0.1; The unidirectional LSTM network is used to transform the data sequence into 128 dimensions and output the data of the last timestamp. The principle of LSTM is expressed as follows: cell state hidden state cell input input gate forget gate output gate Weight vector , , , and It is related to the relationship between the unit input, input gate, forget gate, and output gate. , , , and It is related to the relationship between unit input, input gate, forget gate, and output gate. , , , is the bias term. and is the activation function (usually tanh) for the cell input and hidden state. is used to normalize or compress the unit state. The activation function of all gates is sigmoid. LSTM effectively controls the flow of information by introducing a gating mechanism. The forget gate controls the discarding of information, the input gate controls the addition of new information, and the output gate controls the output of hidden states. In this way, LSTM can retain important information for a long time when processing sequence data, avoiding the gradient vanishing problem of standard RNN.

[0061] There are two fully connected layers, where the number of units in the first fully connected layer is 64, the activation function is ReLU, and the calculation formula is:

[0062] in, is the input value after linear transformation, which is the 1×128 data output by the unidirectional LSTM network, multiplied by a 128×64 weight matrix, and then added with a 1×64 bias term; the output of ReLU is zero or the input itself, which is usually used in deep networks, especially in convolutional neural networks (CNN).

[0063] The number of units in the second fully connected layer is 26, the activation function is Softmax, and the calculation formula is:

[0064] in, The first column vector is obtained by multiplying the 1×64 data output by the first fully connected layer by a 64×26 weight matrix and adding a 1×26 bias term. i Row data, K is the total number of categories, which is 26. is the index, The data of the jth row corresponding to the index; Softmax maps each output to between [0,1], and the sum of all outputs is 1, so that they can be interpreted as probabilities.

[0065] In order to verify the method of this embodiment, the following experiment is performed: The smartphone used is iPhone12, equipped with the ios18.2 system; the smart watch used is iWatch seris9, equipped with the watchOS 11.2 system.

[0066] Put the index finger, middle finger and ring finger of your left hand together and hold the phone against the back. Use the thumb of your left hand to tap the screen to start the UWB connection or tap to end it after writing letters. Use the middle finger of your right hand to write letters in the area between the index finger of your left hand and the lower frame of the phone camera.

[0067] The experiments were run using keras 2.14.0. For the entire model, the learning rate was set to 0.0001, and Adam was used as the optimizer. The early stopping method was used to train the data. When a new minimum loss value appeared, if a new minimum loss smaller than the previous one did not appear within 3 rounds of training, it was considered to be overfitting.

[0068] There are 1040 training data samples, 40 samples for each of the 26 letters. Set the eight directions that the person faces when writing: east, south, west, north, southeast, southwest, northwest, and northeast. The angle between the phone and the horizontal plane is 0°, 30°, 45°, 75°, and 90° in each direction. The training data was overfitted in the 93rd round of training, and 100 samples were used for each round of training. The final accuracy on the training set was 98.8%. Figure 2 As shown, A, D, N, T, Y, and Z each have one sample that has not been successfully trained, and G and Q have three samples that have not been successfully trained.

[0069] There are 208 test samples, 8 samples for each of the 26 letters. Set the angles to 30° east by south, 60° east by south, 30° south by west, 60° south by west, 30° west by north, 60° west by north, 30° north by east, and 60° north by east. The corresponding angles of the phone to the horizontal plane are 0°, 90°, 30°, 45°, 45°, 30°, 75°, and 0°. The specific predictions are as follows: Figure 3 As shown, the accuracy on the test set is 85.57%.

[0070] In addition to this experiment, the present invention conducted input tests on a variety of different smartphones and smart watches. The results showed that the present method can achieve effective uppercase letter input recognition, and the fast recognition rate can meet real-time interaction needs.

[0071] Embodiment 2: This embodiment provides a non-transitory computer-readable storage medium, on which computer instructions are stored, the computer instructions causing the computer to execute a method for inputting handwritten letters on the back of a smartphone by integrating IMU and UWB, the method comprising the following steps: S1. Build a communication framework between smartphones and smart watches, and obtain IMU sequence data and UWB ranging data in the two devices through the API interfaces of smartphones and smart watches; S2, preprocessing the IMU sequence data and UWB ranging data in the two devices to align the data in time and space, and obtain the relative motion characteristics of the two devices; S3, analyzing the influence of different features in the relative motion features of the two devices on the recognition accuracy of handwritten letters, and then selecting the required feature combination based on the analysis, and performing normalization processing; S4. Build a handwritten letter recognition learning model based on LSTM and TCN, and use the feature combination extracted from IMU sequence data and UWB ranging data for training; S5. Load the trained handwritten letter recognition learning model into a smartphone and / or smart watch, and recognize the handwritten letters input on the back of the smartphone based on the real-time IMU sequence data and UWB ranging data of the smartphone and the smart watch.

[0072] Embodiment 3: This embodiment provides an electronic device, which may include: a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus. The processor may call the logic instructions in the memory to execute a method for inputting handwritten letters on the back of a smartphone that integrates IMU and UWB, and the method includes the following steps: S1. Build a communication framework between smartphones and smart watches, and obtain IMU sequence data and UWB ranging data in the two devices through the API interfaces of smartphones and smart watches; S2, preprocessing the IMU sequence data and UWB ranging data in the two devices to align the data in time and space, and obtain the relative motion characteristics of the two devices; S3, analyzing the influence of different features in the relative motion features of the two devices on the recognition accuracy of handwritten letters, and then selecting the required feature combination based on the analysis, and performing normalization processing; S4. Build a handwritten letter recognition learning model based on LSTM and TCN, and use the feature combination extracted from IMU sequence data and UWB ranging data for training; S5. Load the trained handwritten letter recognition learning model into a smartphone and / or smart watch, and recognize the handwritten letters input on the back of the smartphone based on the real-time IMU sequence data and UWB ranging data of the smartphone and the smart watch.

[0073] In addition, the logic instructions in the above-mentioned memory can be implemented in the form of software functional units and can be stored in a computer-readable storage medium when sold or used as an independent product. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc., which can store program code.

[0074] Embodiment 4: This embodiment provides a computer program product, which includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute a method for inputting handwritten letters on the back of a smartphone by integrating IMU and UWB. The method includes the following steps: S1. Build a communication framework between smartphones and smart watches, and obtain IMU sequence data and UWB ranging data in the two devices through the API interfaces of smartphones and smart watches; S2, preprocessing the IMU sequence data and UWB ranging data in the two devices to align the data in time and space, and obtain the relative motion characteristics of the two devices; S3, analyzing the influence of different features in the relative motion features of the two devices on the recognition accuracy of handwritten letters, and then selecting the required feature combination based on the analysis, and performing normalization processing; S4. Build a handwritten letter recognition learning model based on LSTM and TCN, and use the feature combination extracted from IMU sequence data and UWB ranging data for training; S5. Load the trained handwritten letter recognition learning model into a smartphone and / or smart watch, and recognize the handwritten letters input on the back of the smartphone based on the real-time IMU sequence data and UWB ranging data of the smartphone and the smart watch.

[0075] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0076] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0077] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for inputting handwritten letters on the back of a smartphone integrating IMU and UWB, characterized in that: include: S1. Build a communication framework between smartphones and smart watches, and obtain IMU sequence data and UWB ranging data in the two devices through the API interfaces of smartphones and smart watches; S2, preprocessing the IMU sequence data and UWB ranging data in the two devices to align the data in time and space, and obtain the relative motion characteristics of the two devices; S3, analyzing the influence of different features in the relative motion features of the two devices on the recognition accuracy of handwritten letters, and then selecting the required feature combination based on the analysis, and performing normalization processing; S4. Build a handwritten letter recognition learning model based on LSTM and TCN, and use the feature combination extracted from IMU sequence data and UWB ranging data for training; S5. Load the trained handwritten letter recognition learning model into a smartphone and / or smart watch, and recognize the handwritten letters input on the back of the smartphone based on the real-time IMU sequence data and UWB ranging data of the smartphone and the smart watch.

2. The method for inputting handwritten letters on the back of a smartphone integrating IMU and UWB according to claim 1, characterized in that: Step S2 pre-processes the IMU sequence data and UWB ranging data in the two devices to align the data in time and space and obtain the relative motion characteristics of the two devices. The specific method is: S21, truncating the IMU sequence data of the smart phone and the smart watch at the beginning and the end, and interpolating the UWB ranging data, so that the time of the two IMU sequence data and the UWB ranging data is aligned; S22, setting a global coordinate system, and mapping the acquired posture information of the smart phone and the smart watch to the coordinate system; S23. Construct a relative motion feature set of the two devices, including UWB ranging data, various relative motion features of the smartphone and the smart watch in their respective local coordinate systems, and various relative motion features of the smart watch in the smartphone coordinate system.

3. The method for inputting handwritten letters on the back of a smart phone integrating IMU and UWB according to claim 2, characterized in that: In step S21, the method for truncating the IMU sequence data of the smart phone and the smart watch is as follows: The IMU sequence data of the smartphone and smartwatch are cut, and only the data between the previous timestamp of the first timestamp of the UWB ranging data and the next timestamp of the last timestamp of the UWB ranging data are retained; In step S21, the method for aligning the time of two IMU sequence data is as follows: Assume that the update frequency of the IMU serial data of the smartphone and smartwatch is , when the time interval between the timestamp of the smartphone’s IMU serial data and the timestamp of the smartwatch’s IMU serial data is less than When , the middle moment of the two timestamps is taken as the alignment moment, and the corresponding data are the IMU sequence data of the smartphone and smartwatch close to the alignment moment.

4. The method for inputting handwritten letters on the back of a smart phone integrating IMU and UWB according to claim 3 is characterized in that: In step S21, the specific method of interpolating the UWB ranging data to align the time of the two IMU sequence data with the UWB ranging data is: Assume that the update frequency of UWB ranging data is , taking the timestamp of the IMU serial data of the smart watch as a reference, determining the timestamp of the IMU serial data of the smart watch close to a certain timestamp of the UWB ranging data, and combining the two with the IMU serial data of the corresponding timestamp of the smart phone to form a timestamp data group; At this time, some of the IMU sequence data with time stamps do not have corresponding UWB ranging data. The UWB ranging data of two adjacent IMU sequence data with corresponding IMU sequence data are averaged and linearly interpolated to fill the missing UWB ranging data.

5. The method for inputting handwritten letters on the back of a smart phone integrating IMU and UWB according to claim 4, characterized in that: Step S22 sets a global coordinate system, and corresponds the acquired posture information of the smart phone and the smart watch to the coordinate system. The specific method is: First, set up a global coordinate system with the X-axis pointing to the geographic North Pole, the Z-axis pointing vertically upward, and the Y-axis obtained by the right-hand rule; The rotation matrix corresponding to the smart watch is inverted and then multiplied by the rotation matrix corresponding to the smartphone to obtain the rotation matrix from the smart watch coordinate system to the smartphone coordinate system, expressed as: ； in, To directly obtain the rotation matrix from the global coordinate system to the smartphone coordinate system, To directly obtain the rotation matrix from the global coordinate system to the smartwatch coordinate system, is the calculated rotation matrix from the smartwatch coordinate system to the smartphone coordinate system; T is the transpose operation of the matrix, which is equivalent to taking the inverse matrix.

6. The method for inputting handwritten letters on the back of a smart phone integrating IMU and UWB according to claim 5, characterized in that: In the relative motion feature set of the two devices constructed in step S23, some features are directly obtained from the IMU sequence data and the UWB ranging data, and some features are obtained by calculating the rotation matrix from the smart watch coordinate system to the smartphone coordinate system in combination with the data groups of each timestamp.

7. The method for inputting handwritten letters on the back of a smart phone integrating IMU and UWB according to claim 1, characterized in that: In step S3, the influence of different features of the relative motion features of the two devices on the recognition accuracy of handwritten letters is analyzed; the specific method is: S31. Conduct a preliminary experiment to train different models for different features. During the training, 26 letters are used as labels, 26 labels are encoded using one-hot encoding, categorical cross entropy is used as the loss function, and early stopping method is used for training; S32. Select the required features based on the performance of different models trained with different features in the preliminary experiment on the validation set and the training set before overfitting. In the process of early stopping method training data, when a new minimum loss value appears, if there is no new minimum loss smaller than the previous one within 3 rounds of training, it is considered to be overfitting.

8. The method for inputting handwritten letters on the back of a smart phone integrating IMU and UWB according to claim 7, characterized in that: In step S3, the specific method for selecting the required feature combination and performing normalization processing is: S33, selecting the UWB distance, the acceleration of the smart watch in the smart phone coordinate system, and the rotation matrix from the smart watch to the smart phone as a feature combination; S34, fill the sequence length of each sample with 0.0 to 500; S35. Use MinMaxScaler of Python's sklearn.preprocessing module to normalize the data. The formula is as follows: ； in, X is the current value of a feature, is the minimum value of the feature in the current sample, is the maximum value of the feature in the current sample; is a user-defined range to restrict data to, is its maximum value, is its minimum value.

9. The method for inputting handwritten letters on the back of a smart phone integrating IMU and UWB according to claim 8, characterized in that: The handwritten letter recognition learning model based on LSTM and TCN constructed in step S4 has a network architecture including: The Masking layer is used to mask the 0.0 filled in the sample; assuming the sample is , whose shape is (batch_size, timesteps, features), the Masking layer generates a mask matrix M by checking whether the value of each time step is equal to the specified specific value. The shape of the matrix is ​​(batch_size, timesteps), and its elements are: ； in, i is the sample index in the batch, j is the time step index; Bidirectional TCN network is used for data convolution; the size of the convolution kernel is set to 6, the number of filters used in the convolution layer is 128, and each layer uses skip residual connection; there is no causal relationship when setting convolution, that is, the data of the previous timestamp is not used to predict the data of the later timestamp; the expansion list is set to [1, 2, 4, 8]; the data of all timestamps is set to be returned; finally, the loss ratio of the input data of the previous layer is set to 0.1; A unidirectional LSTM network is used to transform the data sequence into 128 dimensions and output the data of the last timestamp; There are two fully connected layers, where the number of units in the first fully connected layer is 64, the activation function is ReLU, and the calculation formula is: ； in, is the input value after linear transformation, which is the 1×128 data output by the unidirectional LSTM network, multiplied by a 128×64 weight matrix, and then added with a 1×64 bias term; The number of units in the second fully connected layer is 26, the activation function is Softmax, and the calculation formula is: ； in, The first column vector is obtained by multiplying the 1×64 data output by the first fully connected layer by a 64×26 weight matrix and adding a 1×26 bias term. i Row data, K is the total number of categories, is the index, The index corresponding to j Row data.

10. A computer program product, characterized in that The computer program product includes a computer program stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer executes the method for inputting handwritten letters on the back of a smartphone integrating IMU and UWB as described in any one of claims 1 to 9.

Images