Method and device for realizing touchpad function by using intelligent ring and electronic equipment
Through the smart ring and white box model, the problem of low accuracy of finger motion information in mobile office scenarios is solved, more efficient input device performance is achieved, and portability and accuracy are enhanced.
Patent Information
- Application Number
- CN202411295644.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-14
- Publication Date
- 2025-01-07
AI Technical Summary
In mobile office scenarios, existing input devices cannot accurately judge finger motion information, resulting in low portability and accuracy of input devices.
Through the intelligent ring with inertial sensors and neural networks, a white box model is established to map the angle between the preset finger and the operating plane and the displacement relationship between the finger tips, and accurately predict the displacement of the fingertips.
It improves the accuracy of finger motion information, reduces the demand for data volume, enhances the interpretability of the algorithm, and achieves more efficient input device performance.
Smart Images
Figure CN119271107A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of mobile office technology, and more specifically, to a method, device and electronic device for implementing a touchpad function using a smart ring. Background Art
[0002] At present, the need for mobile office is increasing. In the scenario of mobile office, the portability of input devices and the accuracy of reflecting the user's finger movement information are required to be high. However, the algorithms used in the relevant input devices in the related technologies cannot determine the correspondence between features and results, resulting in low accuracy in inferring the user's finger movement information. Summary of the invention
[0003] To solve the above problems, the purpose of the embodiments of the present application is to provide a method, device and electronic device for realizing a touchpad function using a smart ring.
[0004] In a first aspect, an embodiment of the present application provides a method for realizing a touchpad function by using a smart ring, wherein the smart ring comprises: an inertial sensor, and the smart ring is worn at a preset position of a finger; comprising:
[0005] Establishing a white box model with a motion mapping relationship between the angle between the preset finger position and the operation plane and the displacement of the fingertip;
[0006] The angle between the inertial sensor and the operating plane is regressed and predicted by a neural network to obtain the angle between the second finger joint and the operating plane and the angle between the first finger joint and the operating plane;
[0007] Obtaining the angle between the inertial sensor and the operating plane collected by the inertial sensor;
[0008] The displacement of the fingertip is determined according to the angle between the inertial sensor and the operation plane, the angle between the second finger joint and the operation plane, the angle between the first finger joint and the operation plane, and the white box model.
[0009] In a second aspect, the embodiment of the present application further provides a device for realizing a touchpad function using a smart ring, including:
[0010] A model building module, used to build a white box model having a motion mapping relationship between the angle between the preset finger position and the operation plane and the displacement of the fingertip;
[0011] A data acquisition module is used to acquire the angle between the inertial sensor and the operating plane collected by the inertial sensor; the angle between the inertial sensor and the operating plane is regressed and predicted by a neural network to obtain the angle between the second finger joint and the operating plane and the angle between the first finger joint and the operating plane;
[0012] The displacement determination module is used to determine the displacement of the fingertip according to the angle between the inertial sensor and the operation plane, the angle between the second finger joint and the operation plane, the angle between the first finger joint and the operation plane, and the white box model.
[0013] In a third aspect, an embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the method in the first aspect are executed.
[0014] In a fourth aspect, an embodiment of the present application further provides an electronic device, comprising a memory, a processor, and one or more programs, wherein the one or more programs are stored in the memory and configured so that the processor executes the steps of the method of the first aspect described above.
[0015] In the solution provided in the first aspect of the embodiment of the present application, a white box model having a motion mapping relationship between the angle between a preset finger position and an operation plane and the displacement of the fingertip is established; the angle between the inertial sensor and the operation plane collected by the inertial sensor is obtained; the angle between the inertial sensor and the operation plane is regressed and predicted by a neural network to obtain the angle between the second finger joint and the operation plane and the angle between the first finger joint and the operation plane; the displacement of the fingertip is determined according to the angle between the inertial sensor and the operation plane, the angle between the second finger joint and the operation plane, the angle between the first finger joint and the operation plane and the white box model; compared with the method in the related art in which the pure black box algorithm used in the related input device cannot accurately determine the correspondence between features and results, by establishing a white box model of the motion mapping relationship between the motion information of the preset finger position and the motion information of the fingertip, the correspondence between the angle between the inertial sensor and the operation plane and the displacement of the fingertip can be accurately obtained without using a pure black box algorithm, and the interpretability of the overall algorithm is enhanced by the white box model while the demand for data is smaller.
[0016] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, preferred embodiments are specifically cited below and described in detail with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS
[0017] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0018] Figure 1A flow chart of a method for realizing a touchpad function by using a smart ring provided in Embodiment 1 of the present application is shown;
[0019] Figure 2 A schematic diagram of the structure of a device for realizing a touchpad function by using a smart ring provided in Example 2 of the present application is shown;
[0020] Figure 3 A schematic diagram of the structure of an electronic device provided in Example 3 of the present application is shown. DETAILED DESCRIPTION
[0021] The method for realizing a touchpad function by using a smart ring provided by the present invention is performed by a smart ring or other smart devices. If the execution subject is other smart devices, the smart ring transmits the data collected by the inertial sensor to the other smart devices. The algorithm provided by the present invention includes a neural network and a white box model.
[0022] Example 1
[0023] See also Figure 1 The embodiment shown is a flow chart of a method for realizing a touchpad function by using a smart ring. The smart ring includes: an inertial sensor, which may be a six-axis IMU sensor.
[0024] The present embodiment provides a method for realizing a touchpad function by using a smart ring, including the following specific steps:
[0025] Step 100: Establish a white box model having a motion mapping relationship between the angle between the preset finger position and the operation plane and the displacement of the fingertip.
[0026] In the above step 100, the preset position refers to the position of the finger where the smart ring is worn, and the preset position includes but is not limited to: the root of the index finger or the root of the middle finger. For the convenience of description, the preset finger position mentioned in this embodiment refers to the root of the index finger.
[0027] The angle between the preset position of the finger and the operation plane refers to the angle between the preset position of the finger and the operation plane when the finger moves on the operation plane.
[0028] The operating plane refers to a hard surface, which can be but is not limited to: the surface of a book or a desktop.
[0029] The above-mentioned displacement of the finger tip refers to the displacement of the finger tip when the finger moves on the operation plane.
[0030] When a person uses a touchpad, the movement trajectory of the fingertip is mapped to the movement trajectory of the cursor on the screen of the mobile device. Therefore, the calculated displacement of the fingertip can accurately correspond to the movement trajectory of the cursor.
[0031] The displacement of the fingertip is reflected in the missing motion information of the base of the index finger. In the current end-to-end method, only the situations covered by the existing data can be processed correctly. If you want to achieve accurate results, a large amount of data is required. The white box modeling method summarizes some rules of the displacement of the fingertip reflected in the angle between the base of the index finger and the operating plane. The motion mapping relationship between the angle between the base of the index finger and the operating plane and the displacement of the fingertip indicated by this part of the law does not need to be constructed through data, thereby reducing the demand for data volume. Therefore, establishing the motion mapping relationship between the angle between the preset part of the finger and the operating plane and the displacement of the fingertip through a white box model can reduce the demand for data volume.
[0032] Specifically, step 100 may include the following steps (1) to (3):
[0033] Step (1): obtaining the motion restriction relationship between different finger joints from multiple sets of touchpad interaction data; wherein each set of touchpad interaction data in the multiple sets of touchpad interaction data includes: the angle between the preset finger position and the operation plane, and the motion position and displacement of the fingertip obtained by the touchpad while the preset finger position moves.
[0034] Step (2): adding the motion restriction relationship as a constraint condition of the neural network to narrow the prediction range of the neural network; regressing and predicting the angle between the preset finger position and the operation plane in each set of touchpad interaction data through the neural network to obtain the angle between the second knuckle of the finger and the operation plane and the angle between the first knuckle of the finger and the operation plane.
[0035] Step (3): adjusting the parameters of the white box model according to the angle between the preset finger position and the operation plane, the angle between the second knuckle of the finger and the operation plane, the angle between the first knuckle of the finger and the operation plane, the movement position of the finger tip and the displacement of the finger tip in each set of touchpad interaction data, and obtaining a white box model having a motion mapping relationship between the angle between the preset finger position and the operation plane and the displacement of the finger tip.
[0036] In the above step (1), the fingers from the finger tip to the finger base are: the first knuckle, the second knuckle and the third knuckle. The different knuckles refer to the first knuckle, the second knuckle and the third knuckle.
[0037] The angle between the preset position of the finger and the operation plane is the angle between the third knuckle of the finger and the operation plane.
[0038] The specific process of regressing and predicting the angle between the preset finger position and the operation plane through a neural network to obtain the angle between the second knuckle and the operation plane and the angle between the first knuckle and the operation plane is prior art and will not be repeated here.
[0039] The motion restriction relationship refers to the partial rules of the motion information of the index finger root reflected by the fingertip motion summarized in the neural network modeling method, including the motion restriction relationship of the angles between adjacent knuckles in different knuckles. Specifically, the restriction relationship of the angles between adjacent knuckles in different knuckles refers to: the motion restriction relationship of the angle between the first knuckle and the second knuckle and the motion restriction relationship of the angle between the second knuckle and the third knuckle. When using the touchpad interaction mode, each group of touchpad interaction data in multiple groups of touchpad interaction data is observed and counted to obtain the motion restriction relationship of the angles between adjacent knuckles in different knuckles. Taking the index finger as an example to illustrate the motion restriction relationship between adjacent knuckles in different knuckles, the motion restriction relationship of the angles between adjacent knuckles in different knuckles of the index finger is: the angle between the first knuckle and the second knuckle of the index finger is generally not less than 150 degrees, and the angle between the second knuckle and the third knuckle is generally not less than 90 degrees.
[0040] In the above step (2), adding the motion restriction relationship as a constraint condition of the neural network to the neural network means regressing the angles between different knuckles in the motion restriction relationship through a neural network, and constraining the angle relationship between different knuckles by adding a restriction term for the angle in the loss function. For example, the two angles of 150 degrees and 90 degrees mentioned above are regressed through a neural network, and the restriction term for the angle is added to the loss function to constrain the angle between the second knuckle and the third knuckle of the index finger to be not less than 90 degrees, or the angle between the first knuckle and the second knuckle of the index finger to be greater than 150 degrees, thereby narrowing the prediction range of the neural network. The prediction range here is the prediction space of the neural network.
[0041] Generally speaking, neural networks will make inferences in the complete prediction space, including some impossible situations. In other words, neural networks need to use a large amount of data to learn situations that will not actually happen. Adding restrictions on angles in the loss function is equivalent to telling the neural network this part of knowledge, reducing the range of neural network predictions, thereby enhancing the computational efficiency of the neural network.
[0042] Step 102: Obtain the angle between the inertial sensor and the operating plane collected by the inertial sensor.
[0043] In the above step 102, the angle between the inertial sensor and the operating plane is obtained by processing the three-axis acceleration data and the three-axis angular velocity data collected by the inertial sensor using a Madgwick filter algorithm.
[0044] The specific process of obtaining the angle between the inertial sensor and the operating plane by processing the collected three-axis acceleration data and three-axis angular velocity data using the Madgwick filtering algorithm is a prior art and will not be described in detail here.
[0045] Step 104: regress and predict the angle between the inertial sensor and the operating plane through a neural network to obtain the angle between the second finger joint and the operating plane and the angle between the first finger joint and the operating plane.
[0046] The angle between the inertial sensor and the operating plane in the above step 104 is the angle between the third knuckle of the finger and the operating plane.
[0047] The specific process of regressing and predicting the angle between the inertial sensor and the operating plane through a neural network to obtain the angle between the second finger joint and the operating plane and the angle between the first finger joint and the operating plane is prior art and will not be repeated here.
[0048] Step 106: Determine the displacement of the fingertip according to the angle between the inertial sensor and the operating plane, the angle between the second finger joint and the operating plane, the angle between the first finger joint and the operating plane, and the white box model.
[0049] Specifically, step 106 may include step (1) to step (2):
[0050] Step (1): Obtain the preset length of the finger knuckles.
[0051] Step (2): Determine the displacement of the finger tip through a white box model according to the angle between the inertial sensor and the operating plane, the angle between the second finger joint and the operating plane, the angle between the first finger joint and the operating plane, and the length of the finger joint.
[0052] In the above step (1), the finger joint refers to the first joint, the second joint or the third joint. The length of each finger joint of a human finger can be a preset fixed value, or can be dynamically set according to a mapping ratio (cd ratio). The cd ratio is the ratio of the displacement information of the finger tip mapped to the cursor movement, and the cd ratio is a preset value; for example, the cd ratio is 2.
[0053] The pre-set lengths of the finger knuckles are pre-stored in the smart ring.
[0054] In the above step (2), the specific processing process of determining the displacement of the fingertip by the white box model is as follows: In order to facilitate the calculation of the displacement of the fingertip, a coordinate system is pre-set, and the movement of the fingertip can be seen as the movement of the fingertip in the coordinate system. The displacement of the fingertip is obtained by calculating the coordinates of the position of the fingertip at different times. A finger placement position is pre-set, and the projection direction of the fingertip and the root of the finger on the operation plane is used as the longitudinal axis of the coordinate system, and the perpendicular direction of the projection direction of the fingertip and the root of the finger on the operation plane is used as the horizontal axis of the coordinate system, thereby obtaining a rectangular coordinate system for calculating the displacement of the fingertip. In the obtained rectangular coordinate system for calculating the displacement of the fingertip, the calculation process of the coordinates of the position of the fingertip at the current moment is: when the white box model obtains the angle between each knuckle and the operation plane, according to the angle between each knuckle and the operation plane and the length of each knuckle, the distance between the fingertip and the root of the finger at the current moment is calculated in the projection direction of the fingertip and the root of the finger on the operation plane. According to the distance between the finger tip and the base of the finger at the current moment, and the positions of the finger tip and the base of the finger in the coordinate system, the coordinates of the position of the finger tip at the current moment are obtained. According to the coordinates of the position of the finger tip at different moments, the displacement information of the movement between the fingers in the coordinate system can be obtained, thereby determining the displacement of the finger tip.
[0055] The displacement of the fingertip and the cd ratio can be mapped to the movement of the cursor.
[0056] The coordinates of the fingertips at different moments include: the coordinates of the fingertips at the current moment and the coordinates of the fingertips at different moments before the current moment. The coordinates of the fingertips at multiple moments before the current moment can be obtained by the calculation process of the coordinates of the fingertips at the current moment by the white box model at this moment, and the coordinates of the fingertips at multiple moments before the current moment are pre-stored in the white box model.
[0057] Among them, the movement of the finger on the operation plane can be regarded as a combination of translation, rotation, and extension. Among them, extension refers to the adjustment of the angle between the finger knuckle and the operation plane. When the finger performs a combination of translation, rotation, and extension, the movement of the fingertip on the operation plane can be regarded as a combination of the horizontal movement and / or vertical movement of the fingertip on the operation plane. In other words, when the finger moves on the operation plane, the fingertip will produce a corresponding displacement in the coordinate system.
[0058] The angles between each finger joint and the operating plane refer to: the angle between the first finger joint and the operating plane, the angle between the second finger joint and the operating plane, and the angle between the third finger joint and the operating plane.
[0059] Among them, the specific process of calculating the distance between the fingertip and the finger base at the current moment in the projection direction of the fingertip and the finger base on the operating plane based on the angle between each finger joint and the operating plane and the length of each finger joint is existing technology and will not be repeated here.
[0060] Among them, the specific process of obtaining the displacement information of the fingertip in the coordinate system according to the distance between the fingertip and the finger base at different times and the position of the fingertip and the finger base in the coordinate system is existing technology and will not be repeated here.
[0061] Optionally, the smart ring further includes a microphone sensor; and the method for implementing a touchpad function using a smart ring further includes steps (A1) to (A2):
[0062] Step (A1): obtaining motion information of a microphone sensor during a window; wherein the motion information includes a peak value of audio data of the microphone sensor during a window in the time domain and a time point at which the peak value is located.
[0063] Step (A2): Determine whether the peak value exceeds the first preset value; when the height of the peak value exceeds the first preset value, infer that the moment of the peak value is the click of the finger at the preset finger position; when the height of the peak value does not exceed the first preset value, infer that the moment of the peak value is the click of the finger near the preset finger position.
[0064] In the above step (A1), finger movements may be recognized based on audio data acquired by the microphone sensor.
[0065] The window refers to: in finger motion recognition, the size of the recognition window is generally fixed, that is, a recognition time period is fixed. For example, the recognition window is fixed to 1 second, and then it is recognized whether a certain action is included in the window.
[0066] Peak refers to the peak value of microphone audio data in the time domain.
[0067] In an embodiment of the present invention, after calculating the peak value of the finger action, it is placed at about 70% of the window. For example, if the window size is 1 second, the peak value is placed at 0.7 seconds. When the peak value is at 70%, the window can just contain one click action, that is, the click action ends just 0.3 seconds after the peak value appears. This cutting will not lose the action data, ensuring the integrity, and at the same time reduce the delay after the action is completed, allowing the model to determine whether the action has occurred more quickly.
[0068] Optionally, the motion information further includes: peak flatness, wherein the peak flatness refers to the slope of the audio data adjacent to the peak in the time domain; the method for realizing the touchpad function by using the smart ring further includes step (B1):
[0069] Step (B1): determining whether the peak flatness exceeds a second preset value;
[0070] When the peak flatness exceeds the second preset value, it is inferred that the user slides after clicking;
[0071] When the peak flatness does not exceed the second preset value, it is inferred that the user lifts up after clicking.
[0072] Optionally, the motion information further includes: the number of peaks and the duration between adjacent peaks; the method for implementing the touchpad function using a smart ring further includes step (C1):
[0073] Step (C1): when the motion information includes only one peak, it is inferred that the user's operation is a single-finger click;
[0074] When the motion information includes two peaks and the duration between adjacent peaks does not exceed a third preset value, it is inferred that the user's operation is a double-finger click.
[0075] In summary, the embodiment of the present invention provides a method for realizing a touchpad function by using a smart ring, by establishing a white box model having a motion mapping relationship between the angle between a preset finger position and an operation plane and the displacement of a fingertip; obtaining the angle between the inertial sensor and the operation plane collected by the inertial sensor; regressing and predicting the angle between the inertial sensor and the operation plane through a neural network to obtain the angle between the second finger joint and the operation plane and the angle between the first finger joint and the operation plane; determining the displacement of the fingertip according to the angle between the inertial sensor and the operation plane, the angle between the second finger joint and the operation plane, the angle between the first finger joint and the operation plane and the white box model; compared with the method in the related art in which a pure black box algorithm applied in a related input device cannot accurately determine the correspondence between features and results, by establishing a white box model of a motion mapping relationship between motion information of a preset finger position and motion information of a fingertip, the correspondence between the angle between the inertial sensor and the operation plane and the displacement of the fingertip can be accurately obtained without using a pure black box algorithm, and the interpretability of the overall algorithm is enhanced by the white box model while the demand for data is smaller.
[0076] Example 2
[0077] The above describes in detail a method for realizing a touchpad function using a smart ring provided in Example 1 of the present invention. The method can also be realized by a corresponding device. The following describes in detail a device for realizing a touchpad function using a smart ring provided in Example 2 of the present invention.
[0078] Embodiment 2 provides a device for realizing a touchpad function using a smart ring, which is used to execute a method for realizing a touchpad function using a smart ring proposed in the above embodiment 1.
[0079] See also Figure 2 FIG. 1 is a schematic diagram showing a structure of a device for realizing a touch panel function by using a smart ring provided by an embodiment of the present invention. Figure 2 As shown, the device for realizing the touchpad function by using the smart ring includes:
[0080] The model building module 200 is used to build a white box model having a motion mapping relationship between the angle between the preset finger position and the operation plane and the displacement of the fingertip.
[0081] The data acquisition module 202 is used to obtain the angle between the inertial sensor and the operating plane collected by the inertial sensor; the angle between the inertial sensor and the operating plane is regressed and predicted by a neural network to obtain the angle between the second finger joint and the operating plane and the angle between the first finger joint and the operating plane.
[0082] The displacement determination module 204 is used to determine the displacement of the fingertip according to the angle between the inertial sensor and the operation plane, the angle between the second finger joint and the operation plane, the angle between the first finger joint and the operation plane, and the white box model.
[0083] Optionally, the model building module 200 includes a restriction relationship acquisition submodule 2001 , a model range reduction submodule 2002 and a model training submodule 2003 .
[0084] The restriction relationship acquisition submodule 2001 is used to obtain the movement restriction relationship between different finger joints obtained from multiple groups of touchpad interaction numbers; wherein each group of touchpad interaction data in the multiple groups of touchpad interaction numbers includes: the angle between the preset finger position and the operation plane, and the movement position and displacement of the fingertip obtained by the touchpad while the preset finger position moves.
[0085] The model range reduction submodule 2002 is used to add the motion restriction relationship as a constraint condition of the neural network to the neural network, thereby reducing the prediction range of the neural network; the angle between the preset finger position and the operation plane in each set of touchpad interaction data is regressed and predicted by the neural network to obtain the angle between the second knuckle of the finger and the operation plane and the angle between the first knuckle of the finger and the operation plane.
[0086] The model training submodule 2003 is used to adjust the parameters of the white box model according to the angle between the preset position of the finger and the operation plane, the angle between the second knuckle of the finger and the operation plane, the angle between the first knuckle of the finger and the operation plane, the movement position of the finger tip, and the displacement of the preset position of the finger and the displacement of the finger tip in each group of the touch pad interaction data, so as to obtain a white box model with a motion mapping relationship between the angle between the preset position of the finger and the operation plane and the displacement of the finger tip.
[0087] In summary, an embodiment of the present invention provides a device for realizing a touchpad function by using a smart ring, by establishing a white box model having a motion mapping relationship between the angle between a preset finger position and an operation plane and the displacement of a fingertip; obtaining the angle between the inertial sensor and the operation plane collected by the inertial sensor; regressing and predicting the angle between the inertial sensor and the operation plane through a neural network to obtain the angle between the second finger joint and the operation plane and the angle between the first finger joint and the operation plane; determining the displacement of the fingertip according to the angle between the inertial sensor and the operation plane, the angle between the second finger joint and the operation plane, the angle between the first finger joint and the operation plane and the white box model; compared with the method in which a pure black box algorithm used in a related input device in the related art cannot accurately determine the correspondence between features and results, by establishing a white box model of a motion mapping relationship between motion information of a preset finger position and motion information of a fingertip, the correspondence between the angle between the inertial sensor and the operation plane and the displacement of the fingertip can be accurately obtained without using a pure black box algorithm, and the interpretability of the overall algorithm is enhanced by the white box model while the demand for data is smaller.
[0088] Example 3
[0089] This embodiment proposes a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the method for implementing a touchpad function using a smart ring described in the above embodiment 1 are executed. For specific implementation, please refer to the method embodiment 1, which will not be described in detail here.
[0090] In addition, see Figure 3 The structural diagram of an electronic device shown in FIG. 1 is a schematic diagram of a structure of an electronic device. This embodiment further provides an electronic device, which includes a bus 51 , a processor 52 , a transceiver 53 , a bus interface 54 , a memory 55 and a user interface 56 . The electronic device includes a memory 55 .
[0091] In this embodiment, the electronic device further includes: one or more programs stored in the memory 55 and executable on the processor 52, configured so that the processor executes the one or more programs to perform steps 100 to 106.
[0092] The transceiver 53 is used to receive and send data under the control of the processor 52.
[0093] Among them, the bus architecture (represented by bus 51), bus 51 can include any number of interconnected buses and bridges, and bus 51 links various circuits including one or more processors represented by processor 52 and memory represented by memory 55. Bus 51 can also link various other circuits such as peripherals, voltage regulators, and power management circuits together, which are all well known in the art, so this embodiment will not be further described. Bus interface 54 provides an interface between bus 51 and transceiver 53. Transceiver 53 can be one element or multiple elements, such as multiple receivers and transmitters, providing a unit for communicating with various other devices on a transmission medium. For example: transceiver 53 receives external data from other devices. Transceiver 53 is used to send data processed by processor 52 to other devices. Depending on the nature of the computing system, a user interface 56, such as a keypad, display, speaker, microphone, joystick, can also be provided.
[0094] The processor 52 is responsible for managing the bus 51 and general processing, such as running the general operating system 551 as mentioned above. The memory 55 can be used to store data used by the processor 52 when performing operations.
[0095] Optionally, the processor 52 may be, but is not limited to: a central processing unit, a single-chip microcomputer, a microprocessor or a programmable logic device.
[0096] It can be understood that the memory 55 in the embodiment of the present application can be a volatile memory or a non-volatile memory, or can include both volatile and non-volatile memories. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory can be a random access memory (RAM), which is used as an external cache. By way of example but not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM) and direct RAM bus random access memory (DRRAM). The memory 55 of the system and method described in the present embodiment is intended to include, but is not limited to, these and any other suitable types of memory.
[0097] In some implementations, the memory 55 stores the following elements, executable modules or data structures, or a subset thereof, or an extended set thereof: an operating system 551 and application programs 552 .
[0098] Among them, the operating system 551 includes various system programs, such as a framework layer, a core library layer, a driver layer, etc., which are used to implement various basic services and process hardware-based tasks. The application 552 includes various application programs, such as a media player (Media Player), a browser (Browser), etc., which are used to implement various application services. The program for implementing the method of the embodiment of the present application can be included in the application 552.
[0099] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.
Claims
1. A method for realizing a touchpad function by using a smart ring, the smart ring comprising: Inertial sensor, the smart ring is worn at a preset position on the finger; characterized in that the method comprises: Establishing a white box model with a motion mapping relationship between the angle between the preset finger position and the operation plane and the displacement of the fingertip; Acquire the angle between the inertial sensor and the operation plane collected by the inertial sensor; Regression prediction is performed on the angle between the inertial sensor and the operating plane by using a neural network to obtain the angle between the second finger joint and the operating plane and the angle between the first finger joint and the operating plane; The displacement of the fingertip is determined according to the angle between the inertial sensor and the operation plane, the angle between the second finger joint and the operation plane, the angle between the first finger joint and the operation plane, and the white box model.
2. The method according to claim 1, characterized in that: The white box model having a motion mapping relationship between the angle between the preset finger position and the operation plane and the displacement of the fingertip is established, including: Obtaining the motion restriction relationship between different finger joints obtained from multiple sets of touchpad interaction data; wherein each set of touchpad interaction data in the multiple sets of touchpad interaction data includes: the angle between the preset finger position and the operation plane, and the motion position and displacement of the fingertip obtained by the touchpad while the preset finger position moves; The motion restriction relationship is added to the neural network as a constraint condition of the neural network to narrow the prediction range of the neural network; the angle between the preset finger position and the operation plane in each set of touch panel interaction data is regressed and predicted by the neural network to obtain the angle between the second knuckle of the finger and the operation plane and the angle between the first knuckle of the finger and the operation plane; The parameters of the white box model are adjusted according to the angle between the preset position of the finger and the operation plane, the angle between the second joint of the finger and the operation plane, the angle between the first joint of the finger and the operation plane, the movement position of the finger tip, and the displacement of the preset position of the finger and the displacement of the finger tip in each group of the touch pad interaction data to obtain a white box model with a motion mapping relationship between the angle between the preset position of the finger and the operation plane and the displacement of the finger tip.
3. The method according to claim 2, characterized in that Determining the displacement of the fingertip according to the angle between the inertial sensor and the operation plane, the angle between the second finger joint and the operation plane, the angle between the first finger joint and the operation plane, and the white box model includes: Get the preset length of the finger knuckles; The displacement of the finger tip is determined by the white box model according to the angle between the inertial sensor and the operating plane, the angle between the second finger joint and the operating plane, the angle between the first finger joint and the operating plane, and the length of the finger joint.
4. The method according to claim 1, characterized in that: The smart ring also includes a microphone sensor; The method further comprises: Acquire motion information of the microphone sensor during a window; wherein the motion information includes a peak value of the audio data of the microphone sensor during a window in the time domain and a time point at which the peak value is located; Determining whether the peak value exceeds a first preset value; When the height of the peak value exceeds the first preset value, it is inferred that the moment of the peak value is the click of the finger at the preset finger position; When the height of the peak value does not exceed the first preset value, it is inferred that the moment when the peak value is located is the click of the finger near the preset position of the finger.
5. The method according to claim 4, characterized in that The motion information further includes: peak flatness, wherein the peak flatness refers to the slope of the audio data adjacent to the peak in the time domain; The method further comprises: Determining whether the peak flatness exceeds a second preset value; When the peak flatness exceeds a second preset value, it is inferred that the user slides after clicking; When the peak flatness does not exceed the second preset value, it is inferred that the user lifts up after clicking.
6. The method according to claim 4, characterized in that The motion information also includes: the number of peaks and the duration between adjacent peaks; The method further comprises: When the motion information includes only one peak, it is inferred that the user's operation is a single-finger click; When the motion information includes two peaks and the duration between adjacent peaks does not exceed a third preset value, it is inferred that the user's operation is a double-finger click.
7. A device for realizing a touch panel function using a smart ring, characterized in that: include: A model building module, used to build a white box model having a motion mapping relationship between the angle between the preset finger position and the operation plane and the displacement of the fingertip; a data acquisition module, configured to acquire the angle between the inertial sensor and the operating plane collected by the inertial sensor; and to perform regression prediction on the angle between the inertial sensor and the operating plane through a neural network to obtain the angle between the second finger joint and the operating plane and the angle between the first finger joint and the operating plane; The displacement determination module is used to determine the displacement of the fingertip according to the angle between the inertial sensor and the operation plane, the angle between the second finger joint and the operation plane, the angle between the first finger joint and the operation plane, and the white box model.
8. The device according to claim 7, characterized in that The model building module comprises: A restriction relationship acquisition submodule, used to acquire the motion restriction relationship between different finger joints obtained from multiple sets of touchpad interaction data; wherein each set of touchpad interaction data in the multiple sets of touchpad interaction data includes: the angle between the preset finger position and the operation plane, and the motion position and displacement of the fingertip acquired by the touchpad while the preset finger position moves; a model range reduction submodule, used to add the motion restriction relationship as a constraint condition of the neural network to the neural network, thereby reducing the prediction range of the neural network; regressively predict the angle between the preset finger position and the operation plane in each set of touchpad interaction data through the neural network, and obtain the angle between the second knuckle of the finger and the operation plane and the angle between the first knuckle of the finger and the operation plane; The model training submodule is used to adjust the parameters of the white box model according to the angle between the preset position of the finger and the operation plane, the angle between the second knuckle of the finger and the operation plane, the angle between the first knuckle of the finger and the operation plane, the movement position of the finger tip, and the displacement of the preset position of the finger and the displacement of the finger tip in each group of the touch panel interaction data, so as to obtain a white box model with a motion mapping relationship between the angle between the preset position of the finger and the operation plane and the displacement of the finger tip.
9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are executed.
10. An electronic device, characterized in that: The electronic device includes a memory, a processor and one or more programs, wherein the one or more programs are stored in the memory and are configured so that the processor executes the steps of the method according to any one of claims 1 to 6.