Gesture and voice controlled interface device
By combining biopotential sensors and inertial measurement units with processors, the technology recognizes user gestures and voice inputs, solving the problem of integrating gesture and voice control in human-machine interfaces. This enables more natural and intuitive user interaction and enhances the feedback and control capabilities of electronic devices.
Patent Information
- Application Number
- CN202380032572.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2022-02-28
- Filing Date
- 2023-02-14
- Publication Date
- 2024-11-19
AI Technical Summary
In existing technologies, human-machine interfaces struggle to effectively combine gestures and voice control when interacting with electronic devices, resulting in insufficient naturalness and intuitiveness in feedback and input.
The system uses a bioelectric potential sensor and an inertial measurement unit (IMU) to measure the bioelectric potential signal and inertial properties of the user's limbs. The data is then analyzed by a processor to recognize the user's gestures and voice inputs and generate corresponding output signals. The system simulates the physical parameters of virtual objects or environments through modulation functions, providing visual, auditory, and tactile feedback.
It enables more natural and intuitive user interaction, enhances the feedback and control capabilities of electronic devices, and improves the user experience.
Smart Images

Figure CN118984985A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to an interface for an electronic device, and more particularly to a gesture and voice controlled interface device. Background Art
[0002] Natural human communication is usually a combination of speech and gestures. Typically, speech is used to convey most of the communication information, while gestures are used as an additional contextual aid. The words exchanged between the participants of the conversation may carry most of the context, but without gestures, the same words may have different meanings. During the conversation, the participants of the conversation provide feedback on the reception and understanding of the information conveyed. Feedback is also usually a combination of speech and gestures. The recipient of the communication may nod and say "yes" to indicate agreement, and may also shake his head or shake his hand to indicate disagreement. Since human communication is the result of the development of natural human communication over time, the aspects and complexities of natural human communication can be intuitively understood by both parties of the conversation. On the other hand, human-computer interfaces with electronic devices and computers are relatively new and have developed slowly.
[0003] Before the invention of smart devices (e.g., smartphones and tablets), human-computer interfaces with computers were limited to basic input and output devices. Input was received by a keyboard and mouse, with the user continuously pressing keys to communicate with the computer. Feedback was provided on output devices such as displays and speakers. For example, feedback provided to confirm (such as clicking a pointing device (such as a computer mouse click)) may include a "ding" sound and a change in the lighting intensity of an object on a display screen (e.g., dimming or brightening).
[0004] A smart device including a touch screen can provide an additional layer of communication with an electronic device. A user of a smart device including a touch screen can touch, press, slide, and perform various gestures on the touch screen. The smart device can provide visual, auditory, and tactile feedback to the user. The screen can display visual effects (e.g., changes in the shape and color of a virtual button), can emit sounds, and can generate vibrations to provide feedback to the user of the smart device.
[0005] Voice and sound can also be used to communicate with electronic devices. Smartphones and smart speakers (e.g., Amazon Echo) can have virtual assistant artificial intelligence (AI) technology (e.g., Amazon Alexa) that allows users to issue commands, and the virtual assistant AI can then interpret the command and provide feedback on the command. Users can typically ask questions or provide commands to the virtual assistant.
[0006] There are electronic devices that recognize gestures (such as hand movements). Gesture recognition can be performed by analyzing video images of the user. Gesture recognition can also be performed by analyzing data received from sensors positioned on the user's body (e.g., an accelerometer on a wearable device attached to the user's arm).
[0007] It may be desirable to provide devices, systems, and methods for combined gesture and voice control of electronic devices. Summary of the invention
[0008] Therefore, according to an embodiment of the present invention, a device for estimating the force applied to an object is provided. The device may include one or more biopotential sensors configured to be placed on a user's limb and measure one or more biopotential signals from one or more muscles or one or more nerves of the limb. The device may also include an inertial measurement unit (IMU) to measure one or more inertial properties of the limb. The device may also include a processor configured to obtain the measured one or more biopotential signals and one or more measured inertial properties, and estimate the force applied to the object by the user's limb based on a model constructed from samples of biopotential signals and inertial properties collected from multiple users applying known forces.
[0009] According to some embodiments of the invention, the one or more inertial properties are selected from the group of inertial characteristics consisting of: orientation, angular velocity, linear velocity and acceleration of the limb.
[0010] According to some embodiments of the invention the processor is configured to estimate a weight of the object.
[0011] According to some embodiments of the invention, the processor is configured to estimate the torque applied to the lever.
[0012] According to some embodiments of the invention, the processor is configured to estimate the force applied to successfully connect the connector.
[0013] According to some embodiments of the present invention, a gesture-controlled interface device is provided. The device may include: one or more gesture sensors for sensing a user's gesture; and a processor configured to obtain one or more sensed gestures from the one or more gesture sensors, analyze the one or more sensed gestures to identify input from the user, and generate an output signal corresponding to the input to the controlled device, the output signal being modulated according to one or more properties of the one or more sensed gestures by a modulation function.
[0014] According to some embodiments of the invention, the modulation function simulates a physical parameter associated with the virtual object or the virtual environment.
[0015] According to some embodiments of the invention, the physical parameter is selected from the group consisting of: weight, texture, friction and resistance.
[0016] According to some embodiments of the invention, the device may comprise a feedback device selected from the group consisting of a screen, a haptic actuator, a tactile actuator and a speaker.
[0017] According to some embodiments of the invention, the one or more properties are selected from the group consisting of position of the hand, pressure between fingers, and distance between fingers.
[0018] According to some embodiments of the present invention, a gesture-controlled interface device is provided. The device may include: one or more gesture sensors for sensing a user's gesture; and a processor configured to obtain one or more sensed gestures from the one or more gesture sensors, analyze the sensed gestures to identify input from the user and identify successful connection of a connector, and generate an output signal corresponding to the input to the controlled device.
[0019] According to some embodiments of the invention, the processor is further configured to analyze the sensed gesture to identify disconnection of the connector.
[0020] According to some embodiments of the invention, the one or more sensors include a sensor selected from the group of sensors biopotential sensors and an inertial measurement unit for determining hand orientation or angular velocity acceleration.
[0021] According to some embodiments of the invention, the processor is configured to analyse sound when analysing the sensed gesture to identify a successful connection.
[0022] According to some embodiments of the invention, the sound comprises a clicking sound or a friction sound.
[0023] According to some embodiments of the invention, the processor is further configured to verify that the one or more sensed gestures are performed by the specific person based on comparing the one or more sensed gestures to a gesture signature of the specific person.
[0024] According to some embodiments of the present invention, there is provided a gesture and voice controlled interface device, comprising: one or more gesture sensors for sensing gestures of a user; one or more audio sensors for sensing sounds emitted by the user; and a processor configured to obtain one or more sensed gestures from the one or more gesture sensors and one or more sensed sounds from the one or more audio sensors, analyze the sensed gestures and the sensed sounds to identify input from the user, and generate an output signal corresponding to the input to the controlled device.
[0025] According to some embodiments of the invention the processor is configured to recognise the input only if the one or more sensed gestures and the one or more sensed sounds are sensed within a predetermined time period.
[0026] According to some embodiments of the invention, the one or more sensed gestures are sensed before the one or more sensed sounds.
[0027] According to some embodiments of the invention, the one or more sensed gestures are sensed after the one or more sensed sounds.
[0028] According to some embodiments of the invention, the one or more sensed gestures are sensed simultaneously with the one or more sensed sounds.
[0029] According to some embodiments of the present invention, the processor is configured to identify an initiation prompt in the one or more sensed gestures or in the one or more sensed sounds, extract a first measured gesture value from the one or more sensed gestures, extract a second measured gesture value from the one or more sensed gestures, and generate an output signal based on the first measured gesture value and the second measured gesture value.
[0030] According to some embodiments of the invention, the initiation prompt is used to select a function of the controlled device, wherein the first measured gesture value is used to select an initial state of a parameter of the function, and wherein the second measured gesture value is used to select a final state of the parameter of the function.
[0031] According to some embodiments of the invention, the function is selected from a group of functions consisting of: volume of an audio device, temperature of a climate control device, size of a virtual object, displacement of a virtual object, and zooming in or out.
[0032] According to some embodiments of the invention, the processor is configured to identify an initiation prompt based on at least one prompt in a prompt group consisting of: a user's voice, a user's gesture, a value of force measured from the user, and a signal generated by the processor.
[0033] According to some embodiments of the invention the processor is configured to wait for a predetermined period of time after identifying the initiation prompt before extracting the first measurement parameter.
[0034] According to some embodiments of the invention the processor is configured to identify a termination cue in the one or more sensed gestures or in the one or more sensed sounds.
[0035] According to some embodiments of the invention the processor is configured to identify the termination prompt based on identifying an exit from a predetermined parameter range or identifying stability within a predetermined parameter range.
[0036] According to some embodiments of the invention the processor is configured to generate the output signal based on a mathematical relationship selected from the group of mathematical relationships consisting of: ratio, addition, subtraction, multiplication and division.
[0037] According to some embodiments of the present invention, the one or more gesture sensors for sensing a user's gestures are configured to sense one or more parameters in a parameter group consisting of: displacement of the user, displacement of the user's limbs, displacement of the user's fingers, facial expressions (facial gestures), facial grimaces (facial grimace), values of biopotentials from nerves or muscles of the user's limbs or fingers, and eye movements of the user.
[0038] According to some embodiments of the invention the processor is configured to calculate a similarity measure between a gesture of the one or more sensed gestures and a predefined gesture and to generate an output signal only if the similarity measure is above a predetermined threshold.
[0039] According to some embodiments of the present invention, the processor is configured to: identify a specific person based on comparing the one or more sensed sounds with the specific person's voice signature; verify that the one or more sensed gestures are performed by the specific person based on comparing the one or more sensed gestures with the specific person's gesture signature; and generate an output signal corresponding to the input to the controlled device only when it is verified that the one or more sensed gestures are performed by the specific person.
[0040] According to some embodiments of the invention the processor is configured to calculate a similarity measure between a gesture of the one or more sensed gestures and a predefined gesture and to generate an output signal only if the similarity measure is above a predetermined threshold.
[0041] According to some embodiments of the invention, the processor is further configured to present to the user via the output device a pattern to be imitated corresponding to the predefined gesture.
[0042] According to some embodiments of the invention, the processor is further configured to present the plurality of patterns to a user via the output device to select a pattern to be emulated.
[0043] According to some embodiments of the invention, the predefined gesture is a variation of a pattern to be imitated.
[0044] According to some embodiments of the invention, the predefined gesture is the inverse of the pattern to be imitated.
[0045] According to some embodiments of the invention the processor is configured to generate a pattern to be imitated.
[0046] According to some embodiments of the invention, the pattern to be imitated is selected from a database of predefined patterns.
[0047] According to some embodiments of the invention, the modality of the pattern to be imitated is selected from the group of modalities consisting of: sound, vision, tactile and touch.
[0048] According to some embodiments of the invention, the output device is selected from the group consisting of: a display screen, a headset, an earphone, a speaker, a tactile actuator, and a haptic actuator.
[0049] According to some embodiments of the invention, the one or more sensed gestures include sensing pressure applied between fingers or sensing pressure resulting from making a fist.
[0050] According to some embodiments of the invention, the one or more sensed gestures further comprise sensing movement of a forearm of the user.
[0051] According to some embodiments of the invention, the motion is selected from the group of motions consisting of: rolling, pitching and yaw.
[0052] According to some embodiments of the invention the processor is configured to determine whether the sensed pressure and the sensed movement are synchronized.
[0053] According to some embodiments of the present invention, the one or more gesture sensors include one or more sensors selected from the following sensor group: reflectometer sensors, biopotential sensors, electromyography (EMG) sensors, surface nerve conduction (SNC) sensors, electrooculography (EOG) sensors, pressure sensors, inertial measurement unit (IMU) sensors, optical sensors and imaging sensors.
[0054] According to some embodiments of the present invention, a gesture-controlled interface device is provided, comprising: one or more gesture sensors for sensing a user's gesture; and a processor configured to obtain one or more sensed gestures from the one or more gesture sensors, analyze the one or more sensed gestures to identify input from the user, and generate an output signal corresponding to the input to a controlled device, wherein the processor is configured to calculate a similarity measure between a gesture in the one or more sensed gestures and a predefined gesture, and generate an output signal only when the similarity measure is above a predetermined threshold.
[0055] According to some embodiments of the invention, the processor is further configured to present to the user via the output device a pattern to be imitated corresponding to the predefined gesture.
[0056] According to some embodiments of the invention, the processor is further configured to present the plurality of patterns to a user via the output device to select a pattern to be emulated.
[0057] According to some embodiments of the invention, the predefined gesture is a variation of a pattern to be imitated.
[0058] According to some embodiments of the invention, the predefined gesture is the inverse of the pattern to be imitated.
[0059] According to some embodiments of the invention the processor is configured to generate a pattern to be imitated.
[0060] According to some embodiments of the invention, the pattern to be imitated is selected from a database of predefined patterns.
[0061] According to some embodiments of the invention, the modality of the pattern to be imitated is selected from the group of modalities consisting of: sound, vision, tactile and touch.
[0062] According to some embodiments of the invention, the output device is selected from the group consisting of: a display screen, a headset, an earphone, a speaker, a tactile actuator, and a haptic actuator.
[0063] According to some embodiments of the invention, the one or more sensed gestures include sensing pressure applied between fingers or sensing pressure resulting from making a fist.
[0064] According to some embodiments of the invention, the one or more sensed gestures further comprise sensing movement of a forearm of the user.
[0065] According to some embodiments of the invention, the motion is selected from the group of motions consisting of: roll, pitch and yaw.
[0066] According to some embodiments of the invention the processor is configured to determine whether the sensed pressure and the sensed movement are synchronized.
[0067] According to some embodiments of the present invention, there is provided an interface device for gesture and voice control, comprising: one or more gesture sensors for sensing gestures of a user; and a processor configured to obtain one or more sensed gestures from the one or more gesture sensors, identify a specific person based on comparing the one or more sensed sounds with a voice signature of the specific person, verify that the one or more sensed gestures are performed by the specific person based on comparing the one or more sensed gestures with a gesture signature of the specific person, and generate an output signal corresponding to an input to a controlled device only when it is verified that the one or more sensed gestures are performed by the specific person.
[0068] According to some embodiments of the present invention, a method for gesture and voice control is provided, comprising: using one or more gesture sensors to sense gestures of a user; using one or more audio sensors to sense sounds made by the user; and using a processor to obtain one or more sensed gestures from the one or more gesture sensors and one or more sensed gestures and one or more sensed sounds from the one or more audio sensors, analyzing the one or more sensed gestures and the one or more sensed sounds to identify input from the user, and generating an output signal corresponding to the input to the controlled device.
[0069] According to some embodiments of the present invention, there is provided a device for estimating the weight of an object, the device comprising: one or more biopotential sensors configured to be placed on a user's limb and to measure one or more biopotential signals from one or more muscles and nerves of the limb; an inertial measurement unit (IMU) for measuring one or more inertial properties of the limb; and a processor for obtaining the measured one or more biopotential signals and one or more measured inertial properties, and estimating the weight of an object supported by the user's limb or the force exerted by the limb based on the one or more biopotential signals and the one or more inertial parameters measured relative to a known weight or a known force.
[0070] According to some embodiments of the invention, the one or more inertial properties are selected from the group of inertial characteristics consisting of: orientation, angular velocity, linear velocity and acceleration of the limb.
[0071] According to some embodiments of the invention the processor is further configured to identify from the additionally measured one or more biopotential signals and the further measured one or more measured inertial properties simulating a physical interaction with the object.
[0072] According to some embodiments of the invention the processor is configured to generate the feedback signal based on the additionally measured one or more biopotential signals and the additionally measured one or more measured inertial properties.
[0073] According to some embodiments of the present invention, a gesture-controlled interface device is provided, comprising: one or more gesture sensors for sensing a user's gesture; and a processor configured to obtain one or more sensed gestures from the one or more gesture sensors, analyze the one or more sensed gestures to identify input from the user, and generate an output signal corresponding to the input to a controlled device, wherein the processor is configured to identify an initiation prompt in the one or more sensed gestures, extract a first measured gesture value from the one or more sensed gestures, extract a second measured gesture value from the one or more sensed gestures, and generate an output signal based on the first measured gesture value and the second measured gesture value, wherein the initiation prompt is used to select a function of the controlled device, wherein the first measured gesture value is used to select an initial state of a parameter of the function, and wherein the second measured gesture value is used to select a final state of the parameter of the function.
[0074] According to some embodiments of the present invention, the function is selected from a function group consisting of: volume of an audio device, temperature of an environment control device, size of a virtual object, displacement of a virtual object, and zooming in or out.
[0075] According to some embodiments of the present invention, a gesture-controlled interface device is provided, comprising: one or more gesture sensors for sensing gestures of a user; and a processor configured to obtain one or more sensed gestures from the one or more gesture sensors, analyze the one or more sensed gestures to identify input from the user, and generate an output signal corresponding to the input to a controlled device, the output signal being modulated by a modulation function according to one or more properties of the one or more sensed gestures.
[0076] According to some embodiments of the invention, the modulation function simulates a physical parameter associated with the virtual object or the virtual environment.
[0077] According to some embodiments of the invention, the physical parameter is selected from the group consisting of: weight, texture, friction and resistance.
[0078] According to some embodiments of the invention, the processor is configured to modulate one or more properties of one or more sensed gestures using a modulation function, and to provide feedback to a user using a feedback device using the modulated one or more properties of the one or more sensed gestures.
[0079] According to some embodiments of the present invention, a method for estimating the weight of an object or the force exerted by the object is provided, the method comprising: using one or more biopotential sensors configured to be placed on a user's limb to measure one or more biopotential signals from one or more muscles and nerves of the user's limb; using a processor to obtain the measured one or more biopotential signals; and using the processor to estimate the weight of an object supported by the user's limb or the force exerted by the limb based on the one or more biopotential signals measured relative to a known weight or a known force.
[0080] According to some embodiments of the present invention, a method of gesture control is provided, comprising: using one or more gesture sensors to sense a user's gesture; and using a processor to obtain one or more sensed gestures from the one or more gesture sensors, analyzing the one or more sensed gestures to identify input from the user, generating an output signal corresponding to the input to a controlled device, identifying an initiation prompt in the one or more sensed gestures, extracting a first measured gesture value from the one or more sensed gestures, extracting a second measured gesture value from the one or more sensed gestures; and generating an output signal based on the first gesture measurement value and the second gesture measurement value.
[0081] According to some embodiments of the present invention, the method also includes using a processor to select a function of the controlled device based on the initiation prompt, using the measured first gesture value to select an initial state of a parameter of the function, and using the second measured gesture value to select a final state of the parameter of the function.
[0082] According to some embodiments of the present invention, the function is selected from a function group consisting of: volume of an audio device, temperature of an environment control device, size of a virtual object, displacement of a virtual object, and zooming in or out.
[0083] According to some embodiments of the present invention, the one or more gesture sensors include one or more sensors selected from the following sensor group: reflectometer sensors, biopotential sensors, electromyography (EMG) sensors, surface nerve conduction (SNC) sensors, electrooculography (EOG) sensors, pressure sensors, inertial measurement unit (IMU) sensors, optical sensors and imaging sensors.
[0084] According to some embodiments of the present invention, a method of gesture control is provided, comprising: using one or more gesture sensors to sense a user's gesture; and using a processor to obtain one or more sensed gestures from the one or more gesture sensors, analyzing the one or more sensed gestures to identify input from the user, and generating an output signal corresponding to the input to a controlled device, wherein the output signal is modulated by a modulation function.
[0085] According to some embodiments of the invention, the modulation function simulates a physical parameter associated with the virtual object or the virtual environment.
[0086] According to some embodiments of the present invention, a gesture-controlled interface device is provided, comprising: one or more gesture sensors for sensing a user's gesture; and a processor configured to obtain one or more sensed gestures from the one or more gesture sensors, and analyze the one or more sensed gestures to identify a specific person based on comparing the one or more sensed gestures with a gesture signature of the specific person, the one or more sensed gestures being performed by the specific person.
[0087] According to some embodiments of the invention, the plurality of sensed gestures are selected from the group of sensors biopotential sensors, SNC sensors, EMG sensors, EEG sensors, EOG sensors.
[0088] According to some embodiments of the invention, the processor is configured to map gesture samples of users into a metric space such that identical gestures are mapped close to each other into an area with a perimeter and gesture samples of different users are mapped at a distance from each other. BRIEF DESCRIPTION OF THE DRAWINGS
[0089] In order to better understand the present invention and realize its practical application, the following drawings are provided and referenced below. It should be noted that the drawings are given as examples only and do not limit the scope of the present invention in any way. The same parts are represented by the same reference numerals.
[0090] Figure 1A is a schematic diagram of a gesture and voice controlled interface device according to some embodiments of the present invention.
[0091] Figure 1B is a schematic diagram of a gesture and voice controlled interface device according to some embodiments of the present invention, wherein the interface device includes a gesture recognition unit and a voice recognition unit.
[0092] Figure 1C is an isometric view of a gesture and voice controlled interface device according to some embodiments of the present invention.
[0093] Figure 1D It is on the user's wrist Figure 1C Side view of a gesture and voice controlled interface device.
[0094] Figure 2 is a flow chart of a method for gesture and voice control interfacing with an electronic device according to some embodiments of the present invention.
[0095] Figure 3is a schematic diagram of a smartphone configured to function as an interface device for gesture and voice control of an electronic device according to some embodiments of the present invention.
[0096] Figure 4 is a schematic diagram of an interface device and a controlled device for gesture and voice control according to some embodiments of the present invention.
[0097] Figure 5 is a flow chart of a method for an interface device for gesture and voice control of an electronic device according to some embodiments of the present invention, the method comprising detecting a user performing an initiation prompt.
[0098] Figure 6 is a schematic diagram of a user adjusting the sound volume of a media screen using a combination of gestures and voice according to some embodiments of the present invention.
[0099] Figure 7 is a schematic diagram of a user adjusting the temperature of an air conditioner using a combination of gestures and voice according to some embodiments of the present invention.
[0100] Figure 8 is a flow chart of a method for evaluating quality criteria for gestures used to interface with an electronic device according to some embodiments of the present invention.
[0101] Fig. 9 is an illustration of a continuous gesture showing the horizontal and vertical dimensions of the gesture for consideration when measuring the gesture according to some embodiments of the present invention.
[0102] Fig. 10A is a schematic diagram of discrete gestures according to some embodiments of the present invention.
[0103] Fig. 10B is a graph showing the confidence level in recognizing a gesture versus time.
[0104] Fig.11 is a graph showing two interfaces and a comparison of the two interfaces according to some embodiments of the present invention, the graph illustrating the relationship between a difficulty index (ID) and a failure index (IF) for each of the two interfaces.
[0105] Fig.12 is a schematic diagram of a series of gestures performed by a user according to examples of gestures shown on a display according to some embodiments of the present invention.
[0106] Fig.13 is a flow chart of a method of detecting a continuous adjustment gesture using an interface device by measuring the force applied by a user according to some embodiments of the present invention.
[0107] Fig.14is a graph showing force values extracted from a sensed pinch-to-zoom (zoom-out) gesture of a user as a function of time, according to some embodiments of the present invention.
[0108] Fig.15 is a graph showing force values extracted from a sensed pinch-to-zoom (zoom-in) gesture of a user as a function of time, according to some embodiments of the present invention.
[0109] Fig.16 is a flow chart of a method for detecting a continuous adjustment gesture using an interface device according to some embodiments of the present invention.
[0110] Fig.17 is a schematic diagram of a user performing a gesture according to a pattern displayed on a display according to some embodiments of the present invention.
[0111] Fig.18A is an illustration of a user wearing an interface device pressing fingers together in response to instructions on a display of the interface device according to some embodiments of the present invention.
[0112] Fig.18B is an illustration of a user of a wearable interface device pressing fingers together in response to instructions displayed on augmented reality glasses according to some embodiments of the present invention.
[0113] Fig.18C is a schematic diagram of a user of a wearable interface device pressing fingers together in response to sound from an audio headset according to some embodiments of the present invention.
[0114] Fig.18D is a schematic diagram of a user wearing an interface device pressing fingers together in response to a vibration signal generated by the interface device according to some embodiments of the present invention.
[0115] Fig.19 is a graph showing a pattern according to some embodiments of the present invention and the value of the measured force applied by a user as the user presses fingers together in response to the pattern as a function of time.
[0116] Fig. 20 is a graph showing a pattern according to some embodiments of the present invention and the value of the measured force applied by a user when the user's fingers are pressed together opposite to the pattern as a function of time.
[0117] Fig.21is a graph according to some embodiments of the present invention, which shows a pattern, a first set of measured force values applied by a user pressing a middle finger and a thumb together in response to the pattern, and a second set of measured force values applied by the user pressing an index finger and a thumb together in response to the pattern.
[0118] Fig.22A is an illustration of a user's hand performing a scrolling gesture while pressing fingers together, and a graph showing hand scrolling as a function of time, according to some embodiments of the present invention.
[0119] Fig. 22B is an illustration of a user's hand moving the hand up and down while pressing the fingers together, and a graph showing hand pitch as a function of time, according to some embodiments of the present invention.
[0120] Fig. 22C is an illustration of a user's hand moving the hand left to right while pressing the fingers together, and a graph showing hand deflection as a function of time, according to some embodiments of the present invention.
[0121] Fig.23A is a graph illustrating force values for a predefined pattern associated with a predetermined gesture according to some embodiments of the present invention.
[0122] Fig. 23B is a graph showing force measurements obtained from consecutive gestures performed by a user imitating the predefined pattern of FIG. 23a.
[0123] Fig.23C is a graph showing the calculated similarity measure between the predefined pattern of FIG. 23a and the force measurements of FIG. 23b.
[0124] Fig.24 is a flow chart of a method for detecting rhythmic path mimicry using an interface device according to some embodiments of the present invention.
[0125] Fig.25 is a flow chart of a method for detecting rhythmic pathway mimicry according to some embodiments of the present invention.
[0126] Fig.26 is an illustration of a user's hand wearing an interface device controlling a cursor on a display according to some embodiments of the present invention.
[0127] Fig. 27 is a flow chart of a method of using an interface device to move a cursor, compensating for rotation of an inertial measurement unit (IMU), according to some embodiments of the present invention.
[0128] Fig.28is a graph illustrating horizontal movement of a cursor and horizontal movement of a user's hand controlling the cursor according to some embodiments of the present invention.
[0129] Fig.29 is a flow chart of a method for controlling a cursor using an interface device having an IMU and a biopotential sensor according to some embodiments of the present invention.
[0130] Fig. 30A is an illustration of a user's hand rotating a wrench, the hand being equipped with an interface device for estimating force, according to some embodiments of the present invention.
[0131] Fig. 30B FIG. 2 is a diagram showing a method for estimating force according to some embodiments of the present invention. Fig. 30A A graph of the measured values of hand force and rotation speed.
[0132] Fig.31A is a flow chart of a method for estimating force using an interface device having an IMU and a biopotential sensor according to some embodiments of the present invention.
[0133] Fig.31B is a flow chart of a method for calibrating an interface device to estimate force according to some embodiments of the present invention.
[0134] Fig.31C is a flow chart of a method of estimating force using an interface device according to some embodiments of the present invention.
[0135] Fig.32A is a schematic diagram of a user's hand lifting an object, the hand being worn with an interface device for estimating the weight of the object, according to some embodiments of the present invention.
[0136] Fig.32B is a diagram showing a method for estimating Fig.32A The weight of the object Fig.32A A graph of the hand force measurements.
[0137] Fig.32C is a diagram showing a method for estimating Fig.32A The weight of the object Fig.32A A graph of the measured values of hand speed.
[0138] Fig.33A is a flow chart of a method for estimating the weight of an object using an interface device having an IMU and a biopotential sensor according to some embodiments of the present invention.
[0139] Fig.33B is a flow chart of a method for calibrating an interface device to estimate the weight of an object according to some embodiments of the present invention.
[0140] Fig.33C is a flow chart of a method of estimating the weight of an object using an interface device according to some embodiments of the present invention.
[0141] Fig.34A is a flow chart of a method for training a metric-based meta-learner to uniquely identify a user based on gestures and sounds made by multiple users obtained using an interface device according to some embodiments of the present invention.
[0142] Fig.34B is a flow chart of a method for tuning a metric-based meta-learner to a particular user's personal gesture signature and voice signature according to some embodiments of the present invention.
[0143] Fig.34C is a flow chart of a method for authorizing user input based on personal gesture signatures and voice signatures using an interface device with a tuned metric-based meta-learner according to some embodiments of the present invention.
[0144] Fig.35A A user's wrist with a wearable interface device configured to detect pressure applied by a user wearing a heads-up display, thereby displaying a virtual hand squeezing a virtual ball, is described according to some embodiments of the present invention.
[0145] Fig.35B is a graph of a modulation function depicting the radius of the virtual ball as a function of the estimated pressure applied by the user.
[0146] Fig.36 A shows a user wearing a wrist interface device and headphones according to some embodiments of the present invention.
[0147] Fig.37A is a graph of a sinusoidal modulation function describing the virtual pattern as a function of position along the screen.
[0148] Fig.37B is a graph describing the possible movements of the hand along the x-axis as a function of time.
[0149] Fig.37C is a graph showing texture values as a function of time as a result of moving the hand along the x-axis.
[0150] Fig.37D is a graph of the texture values after a high pass filter has been applied to it. DETAILED DESCRIPTION
[0151] In the following detailed description, many specific details are set forth in order to provide a thorough understanding of the present invention. However, it will be understood by those skilled in the art that the present invention can be practiced without these specific details. In other instances, well-known methods, procedures, packages, modules, units and / or circuits are not described in detail to avoid obscuring the present invention.
[0152] Although embodiments of the present invention are not limited in this regard, discussions using terms such as "processing," "computing," "calculating," "determining," "establishing," "analyzing," "checking," and the like may refer to operations and / or processes of a computer, computing platform, computing system, or other electronic computing device that manipulate and / or convert data represented as physical (e.g., electronic) quantities within a computer's registers and / or memory into other data similarly represented as physical quantities within a computer's registers and / or memory or other information non-transitory storage medium (e.g., memory) that can store instructions to perform operations and / or processes. Although embodiments of the present invention are not limited in this regard, the term "plurality" as used herein may include, for example, "multiple" or "two or more." The term "plurality" may be used throughout the specification to describe two or more components, devices, elements, units, parameters, and the like. Unless explicitly stated, the method embodiments described herein are not limited to a particular order or sequence. In addition, some of the described method embodiments or elements thereof may occur or be executed simultaneously, at the same point in time, or in parallel. As used herein, the conjunction "or" should be construed as inclusive (any or all of the stated options) unless otherwise stated.
[0153] In some embodiments of the present invention, an interface device for gesture and voice control of an electronic device may be configured to sense a user's voice and gesture, recognize input from the sensed user's voice and gesture, and generate an output signal to the controlled electronic device. The user of the interface device can control the electronic device using a sound signal, a gesture, and / or a combination of sound and gesture. The use of voice can be performed simultaneously and / or separately from the execution of a gesture. For example, a user can issue a command such as "increase the volume" and then raise his hand to perform a gesture that increases the volume to a desired level. The interface device can capture the user's voice via an audio sensor (e.g., a microphone) and interpret the user's voice as a command or information for controlling an electronic device (e.g., a smart speaker) using a processor. The interface device can also recognize a user's gesture (e.g., a gesture that recognizes a raised hand) and converts the distance traveled by the detected hand into a range (e.g., level) of a parameter to be adjusted (e.g., increase the volume of the speaker).
[0154] In some embodiments of the invention, the interface device may include a repository of recognized gestures and voice commands associated with commands and functions. The interface device may be configured to recognize gestures and / or voice commands of a user and associate commands or functions from the repository with gestures and / or voice commands.
[0155] In some embodiments of the present invention, the interface device may sense a user's gesture. The gesture sensor of the interface device may sense a gesture performed by the user (such as a gesture in space). The user's gesture may refer to an action or movement performed by the user, such as pressure applied between the user's fingers (e.g., maintaining, increasing, or decreasing pressure), pressure applied between the user's fingers and the user's palm (or object), movement of the user's fingers (such as increasing or decreasing the distance between the user's fingers or finger parts), movement of the user or the user's part (such as movement or tilt of the user's head, movement or rotation of the user's hand or hand part, movement of the user's torso), movement of the user's eyes, static gestures of the user's hand (e.g., clenching a fist and / or raising one or more fingers upward), sliding motion of the user's fingers or hand, etc. A gesture in space may refer to a gesture performed by the user in space without actively touching an input device (such as a touch screen, a pointing device (e.g., a computer mouse) or a keyboard). For example, a gesture sensor may sense a gesture in space, such as increasing and decreasing the distance between the user's fingers to control the adjustment of a parameter (e.g., volume, temperature, etc.). Although a gesture performed on an input device (e.g., a touch screen) may have a clear gesture start (e.g., when the user touches the touch screen) and a clear gesture end (e.g., when the user moves the hand away from the touch screen), it may be difficult to identify a gesture in space (e.g., the start and end of a gesture in space). The interface device may continuously sense the user's gesture (the user moves continuously), and it may be difficult to clearly identify when a gesture starts, when a gesture ends, and whether the gesture is intentional or random.
[0156] It may be desirable to ensure that gestures and movements of a user that should not be interpreted as actuation gestures are not recognized as actuation gestures. To avoid interpreting random movements as commands, the interface device may be configured to perform a strict evaluation of the user's gestures and require the execution of very specific gestures to be recognized and interpreted as commands. The interface device may be configured to recognize the start, end, complete execution of a gesture, and determine that a particular gesture was successfully executed.
[0157] It may be desirable to define gestures that are unlikely to be random gestures (eg, everyday type random movements).The interface device may include a repository of predefined gestures that are unlikely to be performed randomly by the user.
[0158] In some embodiments of the present invention, the interface device may identify whether the user's gesture is a gesture for interfacing with an electronic device. The interface device may identify a command from the user by associating the user's gesture with a gesture in a set of predetermined gestures recognized by the interface device. For example, a user snapping his fingers once may be an arbitrary movement. A user pinching his fingers twice in a row (e.g., within a predetermined duration) may be a recognized gesture associated with a command (e.g., a command to turn on the lights in a certain room).
[0159] In some embodiments of the present invention, the interface device can provide feedback to the user. The interface device may include an audio output device (e.g., a speaker and / or an interface to a speaker), a visual output device (e.g., a screen, a light), and / or one or more tactile feedback actuators for providing feedback to the user. The interface device may provide feedback (e.g., auditory, visual, tactile) to the user to indicate that the input from the user is recognized. The interface device may provide feedback to the user to indicate that the sensed movement and / or voice from the user is not recognized as input. The interface device may provide instructive or suggestive feedback to the user for performing movements and sentences that are considered input by the interface device.
[0160] In some embodiments of the present invention, the interface device may include a sensor for detecting an audio signal generated by a user and / or a user's gesture. The interface device may include one or more audio sensors (e.g., microphones) for detecting a user's voice signal (e.g., sound, verbal message). The interface device may include a speech recognition unit linked to one or more audio sensors for detecting a user's voice command. The speech recognition unit may be implemented in hardware (e.g., a processor) and / or software. The speech recognition unit may receive the sensed voice signal from the audio sensor and analyze it to determine whether it is noise or a signal designated as a user's voice command. The speech recognition unit may be connected to a processor. The speech recognition unit may communicate with the processor for sending information related to the sensed voice data. The speech recognition device may communicate with the processor by sending a signal indicating that the obtained voice data is recognized as an expected voice command from the user.
[0161] In some embodiments of the present invention, the interface device may include one or more gesture sensors for sensing the user's gestures. For example, the interface device may include one or more cameras as sensors for tracking and detecting the user's gestures. The one or more cameras may be linked to a video analysis unit for analyzing the video obtained by the one or more cameras and for detecting the user's gestures in the obtained video. In some embodiments, the interface device may include one or more depth sensors for sensing the user's gestures.
[0162] In some embodiments of the invention, the interface device may include a wearable device worn by a user. The wearable device may include one or more sensors for detecting gestures.
[0163] In some embodiments of the present invention, the wearable device may include a flexible user interface. The flexible user interface may include a printed circuit board (PCB) having a plurality of elements configured to allow an interface between a user and a processor. The PCB of the flexible user interface may be integrated into an elastic band so that the flexible user interface may be deformed to obtain a cylindrical shape that can fit onto a user's limb (such as a flexible band for a watch).
[0164] In some embodiments of the invention, a PCB of a flexible user interface may include a plurality of biopotential sensors and an array of tactile feedback actuators, wherein the conductive strips have corresponding wiring for the biopotential sensors and / or for the tactile feedback actuators. Each biopotential sensor may include at least two electrodes configured to be in direct contact with the user's skin.
[0165] In some embodiments of the invention, the biopotential sensor may be a surface electromyography (sEMG) sensor, and the conductive strip may have several conductive layers for electrical power transmission and signal transmission. In some embodiments, other sensors (e.g., capacitive electromyography (cEMG) sensors) may be used instead of sEMG. It should be noted that an electromyography sensor is capable of detecting signals originating from muscle movement, wherein these signals may be able to be transmitted along a limb.
[0166] In some embodiments of the invention, the biopotential sensor may be a surface nerve conduction (SNC) sensor capable of detecting nerve signals from the wrist bones, wherein the signals are caused by the user's movement and / or intent to move (e.g., when the user intends to move, the biopotential sensor may sense signals of intent to move). For example, the SNC sensor may be configured to detect signals from three major nerves: the median nerve, the ulnar nerve, and the radial nerve, as performed in a standard medical diagnostic nerve conduction study. In some embodiments including SNC sensors, at least three SNC sensors may be required in order to accurately detect nerve activity from the three major nerves (e.g., one sensor for each major nerve). In some embodiments of the invention, one or more SNC sensors may be required to detect nerve activity.
[0167] In some embodiments of the present invention, the gesture sensor can be a pressure sensor, a photoplethysmogram (PPG) sensor, and an RF sensor. The pressure sensor can be used to detect tendon movement. For example, the interface device can recognize the user's gesture by analyzing tendon movement. A PPG sensor (such as a sensor based on light emitting diode technology) can be used to measure changes in wrist volume during wrist tendon movement.
[0168] In some embodiments of the invention, because each nerve produces a signal in a specific location (e.g., a sensor on the back of the arm may not detect a signal on the front of the arm), the biopotential sensors can be arranged in multiple pairs to detect different sources of electrical activity.
[0169] In some embodiments of the present invention, the sensor may have a differential configuration corresponding to the nature of the detected signal and capable of detecting an electro-action potential formed near these sensors by measuring the voltage difference between at least two corresponding electrodes of each sensor. Such electrodes are typically dry electrodes that can engage the user's skin without the need for additional substances (such as gel) to improve skin conductivity. When the biopotential sensor is attached to the user's body, the biopotential sensor can detect gestures and / or forces applied by the user due to the generated electro-action potential.
[0170] In some embodiments of the present invention, the PCB of the flexible user interface may include an array of tactile feedback actuators. The tactile feedback actuator array may be used as a user tactile feedback mechanism instead of screen-based visual feedback to create a closed-loop feedback. The closed-loop feedback can be learned by the user with minimal conscious effort and can provide an important layer as humans adapt to such interfaces. The interface device (and closed-loop feedback) can be used in games (such as video games and virtual reality simulators or metaverses) to control the game and / or simulation and give the user a sense of augmented or virtual reality.
[0171] In some embodiments of the invention, the PCB may include a wireless communication controller for providing wireless communication (e.g., using a Bluetooth transceiver) to nearby devices. For example, the wireless communication controller may be a Bluetooth Low Energy (BLE) controller that provides reduced power consumption for wireless communication.
[0172] In some embodiments of the present invention, the PCB may include a motion sensor. The motion sensor may be a micro-electromechanical system (MEMS) and may include an accelerometer (detecting acceleration), a gyroscope (detecting orientation), a magnetometer, or any other suitable sensor combination.
[0173] In some embodiments of the present invention, the gesture sensor of the wearable device may include an IMU for detecting movement, such as a gyroscope and an accelerometer. The gesture sensor of the wearable device may include electrodes for sensing electrical signals indicating movement by the user.
[0174] In some embodiments of the invention, the system may include a gesture recognition unit connected to one or more gesture sensors for recognizing gestures of a user. The gesture recognition unit (and any dedicated movement detection unit (such as a video analysis unit or a depth analysis unit)) may include dedicated hardware (e.g., a processor) and / or software components. The gesture recognition unit may receive sensed gesture data from the gesture sensor and detect whether the gesture data is noise or a recognized gesture corresponding to a specific command or function. The gesture recognition unit may communicate with the processor for sending information about the detected gesture.
[0175] In some embodiments of the present invention, the interface device may include a processor. The processor may be configured to obtain sensed data from one or more gesture sensors and to obtain sensed data from one or more audio sensors. The processor may be configured to recognize commands from sensed gestures and sensed audio signals. For example, the processor may receive a command about an inquiry from a user. For example, a user may ask about the time or the weather. In some embodiments of the present invention, a gesture and voice controlled interface device may have one or more processors (e.g., a dedicated processor configured to obtain sensed data from one or more gesture sensors and another dedicated processor configured to obtain sensed data from one or more audio sensors).
[0176] In some embodiments of the invention, the interface device can connect to the Internet and retrieve information requested by the user. For example, the processor can receive a command from the user to set a reminder or update the user's calendar. The interface device can access the user's information (e.g., access the contents of the user's smartphone), such as email, contact list, and calendar. The interface device can include virtual assistant AI technology.
[0177] In some embodiments of the present invention, the interface device can be connected to one or more controlled devices. For example, the interface device can be connected to electronic devices at the user's residence, such as air conditioners, lights, kitchen appliances, and devices supporting the Internet of Things (IoT). The interface device can be configured to convey and / or send output signals to one or more controlled devices. The interface device can control the controlled electronic device. For example, the controller can turn on and off the media screen, change channels, and change the volume. The interface device can communicate with environmental control devices such as air conditioners, smart fans, and smart heaters. The interface device can communicate with electronic devices supporting the Internet of Things such as smart speakers, smart kitchen appliances, smart home devices, etc. The interface device can receive feedback from the controlled electronic device, such as receiving confirmation from the electronic device when receiving the output of the interface device.
[0178] Figure 1A 1 is a schematic diagram of an interface device for gesture and voice control according to some embodiments of the present invention. The interface device 100 may include one or more gesture sensors 104 linked to a processor 108 and one or more audio sensors 114 linked to the processor 108. The user 101 may make a sound 112 and perform a gesture 102. The processor 108 may obtain sound signals and gesture signals from the gesture sensors and the sound sensors, analyze these signals to determine whether the user has generated these audio signals and gesture signals for the expected input (e.g., command, data), and if so, generate output signals to the controlled device 107. The processor 108 may be connected to a feedback device 110 (e.g., a display) for providing feedback to the user.
[0179] Figure 1B 1 is a schematic diagram of an interface device for gesture and voice control of an electronic device according to some embodiments of the present invention, the interface device having a separate gesture recognition unit 105 and a voice recognition unit 115. The interface device 100 may include an audio unit 113 and the gesture unit 103. The audio unit 113 may include one or more audio sensors 114 configured to sense the voice 112 of the user 101. The voice recognition unit 115 may be configured to analyze the audio picked up by the one or more audio sensors 114 to detect the user's voice. The audio unit 113 may be linked to the processor 108.
[0180] The interface device 100 may further include a gesture unit 103. The gesture unit 103 may be configured to sense and recognize a gesture 102 performed by the user 101. The gesture unit 103 may include one or more gesture sensors 104, such as a camera, an imaging device, a depth sensor, a depth imaging camera, a proximity sensor, etc. The gesture sensor 104 may include a force sensor 104a (e.g., a biopotential sensor) for sensing a force applied by a user (e.g., a force applied by a finger of a user). The gesture sensor 104 may be configured to sense a gesture of the user 101. A gesture recognition unit 105 may be connected to the gesture sensor 104. The gesture recognition unit 105 may obtain sensed data from the gesture sensor 104. The gesture recognition unit 105 may analyze the sensed data and recognize one or more gestures of the user. For example, the gesture recognition unit 105 may be configured to detect a gesture, compare the detected gesture with a set of predetermined gestures, such as stored in a repository, and if similarity is determined, recognize the sensed gesture as a gesture from the set of predetermined gestures. In another example, the gesture recognition unit 105 may analyze the sensed gesture and determine, based on the analysis, whether the sensed gesture is a recognized gesture from the set of predetermined gestures. The gesture unit 103 may be linked to the processor 108. The processor 108 may obtain an indication of the recognized gesture 106 from the gesture unit 103.
[0181] In some embodiments of the invention, the processor 108 may send an output 109 to the controlled device 107. The processor 108 may be connected to the controlled device 107 wirelessly (e.g., WIFI, Bluetooth) or via wiring, and send the desired input (e.g., data, command, or function) to the controlled device.
[0182] In some embodiments of the invention, the interface device 100 can be configured to sense speech and gestures from one or more users. The interface device 100 can be configured to identify different users and associate the sensed audio data and the sensed motion data with different users. For example, the interface device 100 can identify unique speech features (such as pitch and tone) of different users and associate the sensed audio with a specific user. The interface device 100 can be used to obtain input from different users.
[0183] In some embodiments of the invention, the interface device 100 may include a feedback device 110 connected to the processor 108 for providing feedback to the user. For example, the feedback device may include a display configured to provide visual feedback to the user.
[0184] Figure 1C is an isometric view of a gesture and voice controlled interface device according to some embodiments of the present invention. Figure 1D It is on the user's wrist Figure 1C 1. A side view of an interface device for gesture and voice control. The interface device 100 may include an elastic band 11 that may be wrapped around the user's wrist 5 by connecting the edge 19 of the interface device 100. The interface device 100 may sense the user's gesture using a biopotential sensor 12 having a biopotential electrode 16 and using a motion sensor 15 (e.g., an inertial measurement unit). The biopotential sensor may be connected to a reference driver 18 (e.g., to provide a reference voltage to the biopotential sensor 16) via a connecting strip 17. The microphone 114a may sense the sound emitted by the user. The processor 108 may obtain the sensed gesture and sound of the user. The interface device 100 may have a feedback device (such as a tactile actuator 14) for providing vibration as feedback to the user. In some embodiments of the present invention, the interface device 100 may completely or partially wrap around the user's wrist (e.g., the interface device 100 may not be completely wrapped around the user's wrist). In some embodiments of the present invention, the interface device may be configured to be worn by the user. The interface device may be configured to be tied to the user's hand. The interface device may be configured to be strapped to a user's wrist. The interface device may be configured as a watch strap.
[0185] In some embodiments of the present invention, the interface device 100 may include one or more gesture sensors (such as the biopotential sensor 12) for sensing the user's gestures, and the interface device 100 may include one or more audio sensors (such as the microphone 114a) for sensing the sounds (and voices) emitted by the user. The interface device 100 may include a processor 108, which is configured to obtain one or more sensed gestures from the one or more gesture sensors, and obtain one or more sensed sounds from the one or more audio sensors. The processor 108 may be configured to analyze the sensed gestures and sensed sounds to identify inputs from the user and generate output signals corresponding to the inputs to the controlled device.
[0186] In some embodiments of the present invention, the processor 108 may be configured to recognize the input only when the one or more sensed gestures and the one or more sensed sounds are sensed within a predetermined time period. The one or more sensed gestures may be sensed before the one or more sensed sounds. The one or more sensed gestures may be sensed after the one or more sensed sounds. The one or more sensed gestures may be sensed simultaneously with the one or more sensed sounds.
[0187] In some embodiments of the present invention, one or more gesture sensors (e.g., sensor 12) for sensing a user's gestures are configured to sense one or more parameters in a parameter group consisting of: displacement of the user, displacement of the user's limbs, displacement of the user's fingers, values of biopotentials from nerves or muscles of the user's limbs or fingers, and eye movements of the user.
[0188] In some embodiments of the present invention, the one or more gesture sensors may include one or more sensors selected from the group of sensors consisting of: a reflectometer sensor, a biopotential sensor, an electromyography (EMG) sensor, a surface nerve conduction (SNC) sensor, an electrooculography (EOG) sensor, a pressure sensor, an inertial measurement unit (IMU) sensor, an optical sensor, and an imaging sensor.
[0189] Figure 2 is a flow chart of a method for combined gesture and voice control of an electronic device according to some embodiments of the present invention. The method may include sensing a user's voice using one or more audio sensors (210).
[0190] In some embodiments of the present invention, a method for combined gesture and voice control of an electronic device may include sensing a user's gesture using one or more gesture sensors (220).
[0191] In some embodiments of the present invention, a method for combined gesture and voice control of an electronic device may include using a processor to recognize gestures and voice commands from a user by analyzing the sensed gestures and voices (230). In some embodiments of the present invention, the method may include sensing the user's voice simultaneously with or separately from sensing the user's gestures. The method may include associating the sensed voice and gestures with input (e.g., a query, command, function, or data) for input to the electronic device.
[0192] In some embodiments of the present invention, the method may further include generating an output signal to the controlled electronic device (240). The method may include controlling the controlled device according to input from the user determined from the analyzed user voice and gesture. The method may include associating the user's input with a query, command, function, data for the controlled device. For example, a user may use a gesture and voice controlled input device according to some embodiments of the present invention to query and obtain data about the temperature setting of the air conditioner from a controlled air conditioner. The user may also use the gesture and voice controlled input device to adjust the temperature setting of the air conditioner. The user may also use the gesture and voice controlled input device to set an environmental control plan that will be activated on the air conditioner later.
[0193] In some embodiments of the present invention, the processor can verify that the gesture or voice is performed by the specific person based on comparing one or more sensed gestures or sounds of the specific person with the signature of the specific person (e.g., gesture signature, voice signature, personal profile signature). The signature of a person (or user) can be derived by obtaining samples and data of the user (such as gestures, sounds and images of the user). The processor can generate a database related to the data obtained from the user, and calculate (e.g., using a neural network program, artificial intelligence and the like) the signature of the user. For example, the processor can be configured to map gesture samples from the user into a metric space so that the same gestures are mapped into the area close to each other, and gesture samples of different users are mapped at a distance from each other. The sample of a specific user performing a specific gesture can be mapped to a smaller area (perimeter) with a center and a radius inside the mapping area of the specific gesture, excluding all other users, which will be mapped outside the perimeter, so that the gesture can be attributed to the specific user by mapping the gesture sample of the specific user to the space and verifying whether the gesture falls within the perimeter of the user sample of the gesture. Similarly, the processor can be configured to map sound samples from users into the metric space such that identical sounds are mapped into regions close to each other and sound samples of different users are mapped at a distance from each other. Sound samples of a particular user performing a particular sound can be mapped to a smaller region (perimeter) having a center and a radius inside the mapped region of the particular sound, excluding all other users, who will be mapped outside the perimeter, such that the sound can be attributed to the particular user by mapping the sound sample of the particular user into the space and verifying whether the sound falls within the perimeter of the user's sample of the sound.
[0194] The processor may use the gestures and voice of the particular person to identify the particular person. The processor may allow or deny the user to interact with the digital device or the function of the digital device. For example, the interface device may be configured to allow a user (such as the owner of the vehicle) to interface with the vehicle and prevent other users from interfacing with the vehicle. The interface device may allow a group of users to interface with the digital device and prevent another group of users from using the digital device. For example, the interface device may identify that the user is from a group of allowed users (by comparing the user's gestures with the user's signature) and allow the user to operate the digital device (such as accessing specific content on the digital device or operating a specific function of the digital device). The interface device may allow different levels of access to different user groups.
[0195] In some embodiments of the present invention, the processor may be configured to identify a specific person based on comparing the one or more sensed sounds with a voice signature of the specific person. The processor may be configured to verify that the one or more sensed gestures are performed by the specific person based on comparing the one or more sensed gestures with a gesture signature of the specific person. The processor may be configured to generate an output signal corresponding to the input to the controlled device only when it is verified that the one or more sensed gestures are performed by the specific person.
[0196] Figure 3 304a. The interface device 300 may include a camera 304a. The camera 304a may be used to capture a video of the user. The captured video may be analyzed to detect and recognize the user's gestures. The interface device 300 may include multiple cameras (304a, 304b).
[0197] In some embodiments of the present invention, the interface device 300 may include a microphone 314 for sensing a user's voice signal. The interface device 300 may include a display 307. The interface device 300 may be configured to provide visual feedback to the user via the display 307. The interface device 300 may also have one or more speakers 308 for providing audio feedback to the user.
[0198] In some embodiments of the present invention, the interface device 300 may include hardware and / or software components for detecting gestures and voice of the user. The interface device 300 may include a processor for analyzing the sensed voice and gestures of the user.
[0199] In some embodiments of the invention, an interface device (such as Figures 1A to 1D The interface device 100 may be coupled to (e.g., Bluetooth, wireless connection, wired connection) an electronic device (such as a smart phone or smart watch) or combined with the electronic device. Some of the functions related to obtaining and recognizing gestures, obtaining and analyzing voices, and obtaining output signals of the controlled device may be performed by the electronic device.
[0200] Figure 4is an illustration of an interface device for distributed combined gesture and voice control of electronic devices and controlled devices according to some embodiments of the present invention. The interface device 100 may have a band 404, which includes electrodes and a sensor 403, which is used to sense the gestures of a user wearing the wearable device. The interface device 100 can detect the movement of the user's hands and fingers, and can detect the force with which the user presses the fingers together, which is generally referred to as one or more gestures of the user hereinafter. The interface device 100 can send the sensed data and / or gesture recognition data 406 to a cloud 408 (e.g., a wireless network). The cloud 408 may include one or more servers, remote computers, cloud services, and other remote computing systems. The cloud 408 may have computing capabilities for receiving input from a user, which is generated based on the sensed gestures of the user.
[0201] In some embodiments of the present invention, the interface device 100 can be linked to a smart speaker 413. The smart speaker 413 may include a microphone 414 for obtaining the sensed audio data of the user, and a speaker 415 for generating sound (e.g., feedback to the user). The smart speaker 413 may include a speech recognition unit 412. The speech recognition unit 412 may be a dedicated hardware component (e.g., an audio processor) and / or a software component. The speaker 413 may be configured to send the obtained sensed audio data 416 or user voice data to the cloud 408.
[0202] In some embodiments of the present invention, the cloud 408 can analyze (e.g., using one or more processors located on one or more remote servers) the user's gestures and voice. The cloud 408 can be configured to receive input from the user based on the sensed gestures and voice. The cloud 408 can be connected to a controlled device 407 (e.g., an air conditioner). The cloud 408 can send output 409 to the controlled device 407 based on the input determined from the user's gestures and voice. The cloud 408 can communicate with the controlled device 407. The cloud 408 can be connected to one or more controlled devices 407.
[0203] Figure 5Flowchart of a method for gesture and voice control of an electronic device according to some embodiments of the present invention, wherein a user performs an initiation prompt. Method 500 may include identifying an initiation prompt (501) by an interface device for gesture and voice control of an electronic device. An initiation prompt (such as a power-on gesture) may be designed to be a controlled electronic device function that a user wishes to interact with. According to some embodiments of the present invention, method 500 includes identifying an initiation prompt (501) performed by a user. The interface device may be configured to identify one or more gestures, voice commands, distances from a sensor, a user's position, a force applied by a user (e.g., on their fingers), and / or prompts generated by the interface device as initiation prompts. Initiation prompts may be different from other prompts and gestures. For example, an up and down movement of a user's hand may represent a prompt, and it may also be a random movement. Initiation prompts may be gestures that a user is unlikely to perform, such as a combination of unique gestures. For example, initiation prompts may include pressing an index finger against a thumb while moving the hand over. In another example, a user may make a pinching gesture with his hand and move his hand away to perform an initiation prompt.
[0204] In some embodiments of the present invention, the interface device may continuously monitor the user's movements in anticipation of an initiation prompt. There may be one or more initiation prompts associated with one or more functions. For example, a gesture of pressing the middle of the index finger with the thumb (e.g., mimicking a thumb press on a remote control) may indicate to the interface device that the user is making an initiation prompt with the intent to control an electronic device, such as a media screen. In another example, a gesture of closing and opening a fist twice may indicate to the interface device that the user is executing a different initiation prompt, intended to control the lights in the room. Once the interface device recognizes an initiation prompt, the interface device may anticipate a command from the user associated with the initiation prompt.
[0205] In some embodiments of the present invention, method 500 may include sensing a voice signal (502). The voice signal may be analyzed to identify a voice command from a user. The interface device may check whether the voice command is recognized (503). For example, the user may perform an initiation prompt and also issue a command to control the electronic device. The interface device may wait until the interface device senses a voice signal from the user.
[0206] In some embodiments of the present invention, the interface device may have a predetermined time period to wait until a voice command from the user is recognized. For example, the interface device may have a timer that counts the time (e.g., 10 seconds) elapsed from the moment the interface device recognizes the initiation prompt. If the voice command is not recognized (e.g., after a period of time has passed since the initiation prompt), the interface device is idle until a new initiation prompt is recognized (501). For example, a user may perform an initiation prompt but fail to provide a voice command. In another example, the interface device may mistakenly recognize the user's movement as an initiation prompt. When the interface device does not recognize a voice command after recognizing the initiation prompt, the interface device is idle until the user performs a new initiation prompt.
[0207] The interface device may be configured to pick up a voice command issued by the user (e.g., during and / or after executing the initiation prompt). In some embodiments of the invention, method 500 may include selecting a function (504) based on the recognized voice command. For example, the user may wish to select a function for the air conditioner, such as adjusting a temperature setting. The user may issue a voice command, such as "change temperature." The interface device may recognize the voice command and select a temperature setting for the air conditioner function.
[0208] In some embodiments of the present invention, the interface device may generate an output signal to a controlled electronic device (e.g., an air conditioner) for selecting a function (e.g., a temperature setting function). Prior to communicating with the controlled electronic device, the interface device may remain idle until the interface device obtains the parameters of the selected function from the user. For example, the interface device may wait for the user to communicate the temperature they want the air conditioner to have, and send an output to the air conditioner for adjusting the temperature according to the user's command. The interface device may select functions for one or more controlled devices. For example, the interface device may select (e.g., according to a command from a user) to turn on lights in multiple rooms. For another example, the interface device may choose to adjust the temperature settings of multiple air conditioners at the same time (e.g., according to the air conditioners selected by the user).
[0209] In some embodiments of the present invention, method 500 may include recognizing a user's gesture to fine-tune a selected function (505). After selecting a function (e.g., using a voice command to select a function), the user may perform a gesture to adjust the parameters of the function and fine-tune the selected function. The interface device may recognize the user's gesture and control the electronic device (e.g., operate the selected function of the electronic device) according to the user's gesture. For example, if the user selects a temperature setting function for an air conditioner, the user may raise or lower their hand to indicate that they wish to increase or decrease the temperature setting, respectively.
[0210] In some embodiments of the present invention, the interface device can recognize a user's gesture for adjusting a selected function and send an output signal to the controlled electronic device. Once the gesture ends, the interface device can send an output to the controlled electronic device (for example, the interface device can change the temperature of the air conditioner until the user stops moving his hand). In some embodiments, the interface device can continue to adjust the controlled function of the electronic device during the user's gesture until the gesture ends. For example, the interface device can lower the temperature setting while the user's hand moves.
[0211] In some embodiments of the invention, method 500 may include idling until a new initiation prompt is identified (eg, returning to step 501 ), such as after an output signal is generated for the controlled electronic device or once step 505 ends.
[0212] Figure 6 6 is a schematic diagram of a user using a combination of gestures and voice to adjust the sound volume of a media screen according to some embodiments of the present invention. The interface device 600 for gesture and voice control of an electronic device can sense gestures and voice commands from a user to control a media screen. A user 601 can perform an initiation prompt. A camera 604 can obtain a video clip of the user. The interface device 600 can identify an initiation prompt performed by the user. Once the interface device 600 identifies the initiation prompt, the interface device can wait for a period of time until the interface device 600 identifies a voice command from the user. The user can also use a voice command 612 to select a function of the media screen 600 (e.g., change channels, adjust volume, change brightness, etc.). The interface device 600 can use a microphone 614 to identify the user's voice command. The interface device 600 can recognize the user's voice command and control the media screen to select the volume adjustment function 607. The interface device can wait for a period of time until the interface device 600 identifies a gesture from the user to adjust the selected function (e.g., change the parameters of the selected function, such as reducing and / or increasing the volume of the media screen). The interface device 600 can generate an output signal to adjust the volume of the media screen. For example, the user may raise his hand to affect an increase in the volume level. The user may lower his hand to affect a decrease in the volume level. After ending control of the electronic device (e.g., a media screen), the interface device may remain idle until a new initiation prompt is identified.
[0213] Figure 7702a is a schematic diagram of a user using a combination of gestures and voice to adjust the temperature of an air conditioner according to some embodiments of the present invention. User 701 may wish to control air conditioner 707. According to some embodiments of the present invention, a gesture and voice controlled interface device may sense gestures and voice signals of user 701 for controlling air conditioner 707. User 701 may wear an interface device 100 including a microphone 114a and a band 404. Band 404 may include a gesture sensor 403. As in 702a, user 701 may perform an initiation prompt by pressing an index finger and keeping the index finger against the thumb to wake up the interface device. Interface device 100 may sense and detect an initiation prompt and wait for a user's voice command to follow.
[0214] In some embodiments of the present invention, if the user 701 releases the pressure of the finger (for example, if the user separates the index finger from the thumb, and / or if the user releases the pressure of the pressed finger), the interface device 100 can be configured to sense and recognize the voice and / or voice command of the user 701. The interface device 100 can also wait for a new initiation prompt from the user. The interface device 100 can be configured to sense and detect the user's voice command and / or gesture as long as the user maintains the initiation prompt, or to sense and detect the voice command and / or gesture within a predetermined time period thereafter. In some embodiments, there will be no time limit, and the voice command can be received and action can be taken at any time after the initiation prompt is executed.
[0215] When the user 701 presses the fingers together, the user may issue a voice signal (e.g., a verbal command, such as "select air conditioner" and "set room temperature") to select the air conditioner 707 and select the temperature function of the air conditioner 707 (respectively). The interface device 100 may recognize the user's voice command "set room temperature" and may select the air conditioner closest to the user (e.g., the air conditioner in the room where the user is standing). The interface device 100 may select the air conditioner 707 as a controlled device and select a function from at least one available controlled device function. For example, the interface device 100 may have one or more available controlled devices and functions, such as the air conditioner 707 (and the functions of the air conditioner 707) and the media screen 707A (and the corresponding functions of the media screen 707A). The interface device 100 may generate an output to the air conditioner 707 for controlling the air conditioner 707 and select the temperature setting function of the air conditioner 707.
[0216] After selecting the air conditioning temperature setting function, the user can bend the thumb once 702b to increase the temperature by one degree. The user can bend the index finger 702c to affect the temperature to be lowered by one degree. The user can perform one or more gestures to increase or decrease the temperature. For example, the interface device 100 can detect and identify a gesture for increasing or decreasing the temperature setting by one degree (e.g., by involving a gesture of moving the thumb 702b) and / or a gesture of five degrees (e.g., involving a gesture of moving the pointing finger 702c). Once the desired temperature is reached, the user can perform a completion gesture, such as by tapping the finger 702a, thereby indicating to the interface device 100 that the user has ended the control command they expect. The interface device 100 can also release control of the electronic device (e.g., release control of the air conditioner 707) and remain idle until a new initiation prompt is detected by sensing and detection.
[0217] Figure 8 is a flow chart of a method for measuring and modeling human gestures according to some embodiments of the present invention. Fitts's law can be used to model human-computer interaction. Fitts's law can be used to evaluate a pointing device in a setting where a user uses a pointing device such as a computer mouse to move a cursor along a distance on a computer screen to select a target. A target with a random width W can be drawn at a random distance D from the cursor on the screen. The target can have a difficulty index ID calculated according to the following formula:
[0218]
[0219] The time t can be measured as the time it takes for the user to move the cursor to the target and select the target. Multiple iterations of moving the cursor to the target can be performed to obtain multiple measurements of the time it takes the user to move the cursor to the selected target. Linear regression can be applied to the measurements to find the line parameters a and b that describe the relationship between the estimated time MT and the difficulty index ID:
[0220] MT=a+b×ID (2)
[0221] Figure 8A flow chart of a method 801 for evaluating quality criteria for gestures for interfacing with an electronic device according to some embodiments of the present invention is shown. The method 801 can be used to model multiple gestures. The modeled gestures can include static gestures (wherein a user can, for example, make a fist) and / or dynamic gestures (wherein a user can, for example, press fingers together while moving their hand along a line). The method can evaluate gesture interfaces for systems, devices, and methods, where there is a substantially linear relationship between a gesture failure index (IF) and a gesture difficulty index (ID). The relationship between the failure index and the difficulty index can be estimated using a linear regression applied to a measurement of a gesture failure index with a variable difficulty index.
[0222] In some embodiments of the present invention, method 801 may include setting the detection method to a random gesture difficulty index ID (802). The gesture difficulty index ID may be set to a random level by setting the gesture duration d and the required confidence c according to the following formula:
[0223] ID=d×c (3)
[0224] Equation (3) above may describe the relationship between the ID and parameters associated with the ID, such as duration d and required confidence level c. The ID may be the product of duration and required confidence level. As the duration of a gesture increases, it may be more difficult for the user to perform the gesture. For example, a gesture of raising the user's hand for a long time may be more difficult to perform than the same gesture performed in a shorter time. As the confidence required for a gesture increases, the gesture may become more difficult for the user. For example, if the confidence level required for a gesture is high, the user's margin of error may be smaller, and the interface device may fail to recognize the gesture due to a small difference between the required gesture and the movement actually performed by the user. The required confidence level c may be described by dividing the constant k by the sensitivity level s:
[0225]
[0226] In some embodiments of the present invention, the method may include measuring a gesture failure index IF (803) for the gesture performed. Repeated measurements of the failure index may be performed by repeatedly performing the gesture. For each gesture performed, false negatives (failure to detect the gesture) and true positives (correct detection of the gesture) are counted together. In addition, when other actions other than gestures are performed, the failure index IF may be calculated based on the counted false positives (misdetecting random movements as gestures) and true negatives. The more difficult the gesture is for the user, the more failed attempts are expected to be counted, the more false negative detections are, and the fewer successful attempts (true positive detections are). It can also be expected that the more difficult the gesture is, the fewer false positives are and the more true negatives are. The failure index may have the following relationship with false negative detections FN, false positive detections FP, true positive detections TP, and true negative detections TN:
[0227]
[0228] In some embodiments of the invention, method 801 may include adding a data point to a graph at (ID, IF) (804). When a data point at (ID, IF) is obtained, the data point may be added to a data structure for storage and / or display, such as a graph, an array, a table, and other data structures suitable for data storage and / or display. The data point or data points at (ID, IF) may describe a relationship between IF and ID. A graph including a data point at (ID, IF) may describe a relationship between IF and ID at coordinates x=ID, y=IF.
[0229] In some embodiments of the present invention, method 801 may include checking whether enough data points have been obtained (805). If the data points are insufficient (e.g., the relationship between IF and ID is unclear according to the metric), steps 802 to 804 may be repeated multiple times. When steps 802 to 804 are repeated, the data (and other data structures) represented in the graph may present a clearer relationship between ID and IF. For example, when more data points are obtained, the signal-to-noise ratio increases. In some embodiments of the present invention, Monte Carlo sampling may be applied to determine how many times steps 802 to 804 should be repeated to meet a threshold value of the quality of the relationship between ID and IF.
[0230] In some embodiments of the present invention, the method may further include performing linear regression to estimate the relationship between ID and IF (806). Linear regression may be applied to the measured values to find line parameters a and b describing the relationship between the failure index IF and the difficulty index ID:
[0231] IF=a×ID+b (6)
[0232] Fig. 9907 is a schematic diagram of a continuous gesture according to some embodiments of the present invention, which shows the horizontal and vertical dimensions of the gesture for consideration when measuring the gesture. A user may perform a continuous gesture. For example, in order to correctly recognize a gesture, a user may be required to press the index finger against the thumb 902 while moving the hand along the arrow 906 by a predetermined distance 903 without crossing the boundary lines 904 and 905. The boundary lines 904 and 905 may be separated by a distance 907. For example, if the user crosses line 905 or line 904, the gesture may not be registered as a recognized gesture. For another example, if the user does not complete sweeping the hand across the entire distance 903, the gesture may not be registered as a recognized gesture. Distance 907 may represent the sensitivity of the gesture detection method. The larger the distance 907, the easier it is for the user to register a gesture. In addition, the larger the distance 907, the easier it is to mistakenly register a random movement as a qualified gesture. The smaller the distance 907, the more concentrated the user may be required to successfully complete the task (e.g., the more difficult it may be for the user to register a gesture). Additionally, the smaller the distance 907 is, the more difficult it may be to falsely register a random movement as a qualified gesture.
[0233] In some embodiments of the present invention, distance 903 may represent the duration of a gesture. The longer distance 903 is, the longer the user may need to maintain concentration to avoid crossing line 904 or line 905. The longer distance 903 is, the more difficult it may be for a user to register a gesture. In addition, the longer distance 903 is, the more difficult it may be to mistakenly register random movements as gestures. The shorter distance 903 is, the easier it may be for a user to register a gesture. In addition, the shorter distance 903 is, the easier it may be to mistakenly register random movements as gestures.
[0234] To estimate the gesture ID in this example, the required confidence c can be described as a constant k divided by the distance 907 denoted as d1:
[0235]
[0236] The index of the difficulty ID may be expressed as the product between the duration associated with the distance 903, represented here as d2, and the confidence requirement c:
[0237] ID=d2×c (8)
[0238] Therefore, the difficulty index may be expressed as a function of the distance 903 denoted as d2 and the distance 907 denoted as d1 as follows:
[0239]
[0240] Fig. 10A is a schematic diagram of discrete gestures according to some embodiments of the present invention. Fig. 10B1001 is a graph showing the confidence level over time when recognizing a gesture. In some embodiments of the present invention, a user may make a discrete gesture 1001. For example, a user may raise a hand, extend the thumb and pinky and bend the other fingers. Graph 1002 may show an example of a confidence level (the confidence level of correctly recognizing a gesture, e.g., the confidence level may be a value calculated by an algorithm for detecting gestures) associated with a gesture (e.g., gesture 1001) over time. Once the confidence level is greater than a confidence threshold 1003 for a certain duration, the gesture may be registered (recognized). For a gesture to be registered, the duration required for the confidence level to remain above the confidence threshold 1003 may be longer than the shortest duration above the time threshold 1004.
[0241] In some embodiments of the present invention, the longer the time threshold 1004 is, the more difficult it may be for a user to register a gesture. In addition, the longer the time threshold 1004 is, the more difficult it may be to mistakenly register a random movement as a gesture. The shorter the time threshold 1004 is, the easier it may be for a user to register a gesture. In addition, the shorter the time threshold 1004 is, the easier it may be to mistakenly register a random movement as a gesture.
[0242] In some embodiments of the present invention, the higher the confidence threshold 1003, the more difficult it may be for a user to register a gesture. In addition, the higher the confidence threshold 1003, the more difficult it may be to mistakenly register a random movement as a gesture. The lower the confidence threshold 1003, the easier it may be for a user to register a gesture. In addition, the lower the confidence threshold 1003, the easier it may be to mistakenly register a random movement as a gesture.
[0243] To estimate the gesture difficulty index ID in this example, ID may be expressed as the product between the duration associated here with the time threshold 1004 denoted as d and the confidence requirement c associated here with the confidence threshold 1003:
[0244] ID=d×c (10)
[0245] Fig.11is a graph showing a comparison of two interfaces according to some embodiments of the present invention, which shows the relationship between the difficulty index (ID) and the failure index (IF) of each of the two interfaces. The (ID, IF) data points of interface A and interface B can be collected and presented in the graph. The relationship between the failure index and the difficulty index of each interface can be estimated using a linear regression applied to the (ID, IF) measurement. The properties of interface A and interface B can be derived from the graph. For example, it can be determined from the graph whether interface B is more user-friendly than interface A. The user-friendly nature of the interface can be determined based on the tendency of the (ID, IF) relationship of the interface. The tendency of the (ID, IF) relationship can be inferred by performing a linear regression on the (ID, IF) data points of the interface. The linear regression of the (ID, IF) data points can form a relationship between ID and IF:
[0246] IF=a×ID+b (11)
[0247] The inferred parameter a may represent a tendency of the relationship that may be associated with the user-friendliness of the interface. For example, if a is larger, the interface may be considered less user-friendly, and if a is smaller, the interface may be considered more user-friendly.
[0248] Fig.12 12 is a schematic diagram of a continuous gesture performed by a user according to an example of a gesture displayed on a display according to some embodiments of the present invention. The user can wear the interface device 100 on the wrist and perform a gesture 1202 according to the instructions displayed on the display 1201. The interface device can evaluate the difficulty index (ID) and the failure index (IF) of the gesture 1202. The interface device can provide visual feedback to the user to indicate whether the user is performing the gesture 1202 according to the instructions 1203. For example, if the user is performing the gesture 1202 within the allowed boundary limits 1204 and 1205, the interface device can provide feedback.
[0249] Fig.13 is a flow chart of a method 1300 of detecting a continuous adjustment gesture using an interface device by measuring the force applied by a user according to some embodiments of the present invention.
[0250] A continuous adjustment gesture (such as a pinch-to-zoom gesture) may refer to a continuous gesture performed by a user to adjust a parameter of an electronic device. For example, a user may raise or lower a hand, increase or decrease the distance between fingers, increase or decrease the pressure applied between fingers, and perform similar continuous gestures to adjust a parameter of an electronic device (e.g., increase or decrease the volume, increase or decrease the temperature, zoom in or out, etc.).
[0251] A pinch-to-zoom gesture may be a gesture in which a user pinches two fingers (e.g., a user performs the gesture to command an increase and / or decrease in the value of a function of an electronic device). In a pinch-to-zoom gesture, a user may move an index finger closer to and / or further away from a thumb in a pinching manner. For example, a user using a touch screen displaying an image may place a thumb and an index finger on an image displayed on the touch screen, and move the fingers apart or closer to increase or decrease the size of the image, respectively (zoom in or out). A user may place the thumb and index finger at a first distance (also referred to as a reference distance) on the image. The user may slide these fingers on the touch screen to increase the distance between the thumb and the index finger, thereby enlarging the image. The user may reduce the distance between the thumb and the index finger by sliding the thumb and the index finger closer to each other on the touch screen to zoom out. When the gesture is completed, the image may be made larger or smaller by calculating the ratio between the initial distance between the user's thumb and the index finger and the final distance between the user's thumb and the index finger.
[0252] It may be difficult to detect a continuous adjustment gesture in space (e.g., the user performs the continuous adjustment gesture in the air and is not touching the touch screen). For example, when performing a pinch-to-zoom gesture, a reference distance between the user's fingers and a final distance between the user's fingers may be easily obtained on the touch screen, but when performing the gesture in space, it may be necessary to identify an expected reference distance and an expected final distance between the user's fingers.
[0253] According to some embodiments of the present invention, the continuous adjustment gesture can be used for various commands and applications where a user may desire to increase or decrease a value. For example, a user can use the continuous adjustment gesture as a command to increase or decrease the volume level of a speaker (such as in a media screen or smart speaker). In another example, a user can use the continuous adjustment gesture as a command to increase or decrease the temperature setting of an air conditioner.
[0254] In some embodiments of the invention, a gesture and voice controlled interface device may be used to detect a continuous adjustment gesture performed by a user. The interface device may continuously monitor the user to detect the continuous adjustment gesture. The interface device may sense gestures (e.g., using a video feed obtained by a camera, using a biopotential sensor, etc.) and / or sensed voice signals of the user (e.g., using a microphone to sense audio).
[0255] In some embodiments of the present invention, method 1300 may include extracting values of measured parameters, such as the distance between fingers, the force applied between fingers, etc., from one or more sensed gestures performed by a user (1301). In some embodiments of the present invention, method 1300 may include identifying an initiation prompt, such as checking whether the value of the measured parameter is above a threshold (1302). The user may perform movements and gestures. The interface device may obtain the user's movement data. For example, the interface device may include a camera and / or a wearable device (e.g., a smart watch) including multiple biopotential sensors. The interface device may extract the value of the force or other measured parameter (e.g., the distance between fingers) as a function of time. The extracted value of the measured parameter may represent the force applied by the user between the index finger and the thumb. The extracted value of the measured parameter may be related to the distance between the index finger and the thumb of the user. For example, when the user moves the thumb and the index finger apart, the measured parameter may be the changed distance between the fingers recorded by the camera, the force applied between the fingers, the measured force applied by the user's muscles to move the fingers away from or toward each other, and the measured force is measured by the biopotential sensor.
[0256] In some embodiments of the invention, the interface device may continuously obtain the measured values of the measured parameter. For example, the interface device may check whether the user presses the thumb and index finger together hard enough (e.g., applies a force exceeding a predetermined threshold, thereby satisfying the first condition). The user may press the thumb and index finger together hard enough to indicate to the interface device that the user wishes to perform a continuous adjustment gesture.
[0257] If no initiation prompt is detected, for example, the extracted value of the measured parameter is not higher than the threshold, the interface device can return to sensing the gesture and extracting the value of the force (1301). If the initiation prompt is recognized, for example, the extracted value of the measured parameter is higher than the threshold, the interface device can wait for a predetermined time period (e.g., a first time period represented as dt (such as milliseconds)) (1303). For example, the first time period can span from a few milliseconds to a few seconds. The user can move the index finger and thumb during the first time period so that, for example, the fingertip of the index finger and the fingertip of the thumb are at an initial distance (e.g., a reference distance) from each other. The interface device can use the extracted value of the measured parameter indicating the reference distance between the user's index finger and thumb to later determine the parameters of the continuous adjustment gesture (e.g., the degree of enlargement or reduction). In some embodiments of the present invention, the interface device can extract a reference value "r" of the measured parameter (1304). After the first time period has passed, the interface device can extract a reference value of the force corresponding to the reference distance between the user's index finger and thumb or the pressure applied between the user's fingers or the distance measured by the camera.
[0258] In some embodiments of the present invention, the interface device may perform step 1305 by performing mathematical calculations based on the values of force and r. After the interface device performs step 1304 (e.g., extracting the reference value r), the interface device may continuously extract the value of the force. The interface device may continuously monitor the distance between the user's index finger and thumb or continuously extract the value of the force. The interface device may calculate the ratio between the extracted value of the force and r (e.g., for calculating the output signal). The ratio between the extracted value of the force and r may determine the output signal of the continuous adjustment gesture. For example, if the ratio is greater than 1, it may be associated with the user increasing the distance between the index finger and the thumb (e.g., for zooming in). If the ratio is less than 1, it may be associated with the user reducing the distance between the index finger and the thumb (e.g., for zooming out). The interface device may continuously control the electronic device during the continuous adjustment gesture. For example, while the user increases the distance between the thumb and the index finger, the interface device may continuously increase the size of the displayed image (e.g., until the pinch-to-zoom gesture ends).
[0259] In some embodiments of the invention, the interface device may extract a second extracted value of the force, e.g., for performing mathematical calculations to generate a final output signal. If the extracted value of the force is within a threshold for a period of time, the interface device may extract a second value (1306). If the extracted value of the force is between predetermined thresholds for a predetermined period of time, the interface device may end the interaction (1307) (and generate an output signal) or return to extracting the value of the force (1305). For example, the user may stabilize the finger for a predetermined period of time to indicate to the interface device that the user wishes to end the continuous adjustment gesture.
[0260] In some embodiments of the present invention, the device may provide feedback to the user to indicate the execution and / or completion of one or more actions of method 1300. When the extracted value of the force is higher than a threshold, the device may provide feedback to the user, for example, the device may provide audio and / or visual and / or tactile feedback. The feedback may help the user to correctly perform the desired gesture, for example, adjusting the distance between the thumb and the index finger to set a reference distance for extracting r. When the interface device extracts the value of r (e.g., step 1304), the interface device may provide feedback to the user. When the continuous adjustment gesture ends (e.g., step 1307), the interface device may provide feedback to the user.
[0261] Fig.14 1410 is a graph showing extracted values of force obtained from a sensed pinch-in gesture of a user over time according to some embodiments of the present invention. A user may perform a continuous adjustment gesture, such as a pinch-in gesture, to zoom in on image 1410.
[0262] In some embodiments of the present invention, the interface device can continuously sense the user's gestures. The device can extract a force value. The extracted force value can be related to the amount of force applied by the user between the index finger and the thumb. When the user presses the index finger against the thumb with force, the extracted force value can increase, and when the user reduces the pressure and / or increases the distance between the index finger and the thumb, the extracted force value can decrease. It will be appreciated by those skilled in the art that the user can use other finger pairs to perform a continuous adjustment gesture. For example, the user can use the thumb with the middle finger, or the index finger with the pinky finger.
[0263] In some embodiments of the invention, the user may press the thumb against the index finger 1430 to initiate a continuous adjustment gesture. At time t1, the extracted value of the force may be equal to or above the start threshold 1401. The device may wait for a first time period (e.g., a few milliseconds of dt) during which the user may adjust the distance (or the force applied) between the index finger and the thumb to set a reference distance. After the first time period has passed (e.g., at t2) (1402), the device may extract a reference value of the force r associated with the reference distance.
[0264] In some embodiments of the invention, the interface device may sense that the user has reduced the pressure and / or has increased the distance between the index finger and thumb 1440. As a result, the displayed portion of the image may increase (1420).
[0265] In some embodiments of the invention, the user may stabilize the index finger and thumb at time t3 1403 to indicate to the device that they wish to end the pinch-to-zoom gesture. The interface device may check whether the extracted value of force remains within the upper threshold 1409a and the lower threshold 1409b (defining a relative range in which the graph of force(t) can be considered flat) for a predetermined duration (between t3 1403 and t5 1405). The device may detect fluctuations in force(t) and automatically determine whether force(t) remains within limits 1409a and 1409b based on the detected fluctuations.
[0266] In some embodiments of the invention, the user may abandon the continuous adjustment gesture (1404) at time t4 to indicate to the device that the pinch-to-zoom gesture is complete. The interface device may determine whether the slope of the extracted value of force (dashed line 1406) is greater than a predetermined slope value, and / or whether the extracted value of force drops by more than a predetermined threshold between time t4 (boundary line 1404) and time t5 (boundary line 1405).
[0267] In some embodiments of the invention, a processor of an interface device (such as Figure 1CThe processor 108 of the control device may be configured to recognize an initiation prompt in the one or more sensed gestures, extract a first measured gesture value from the one or more sensed gestures, extract a second measured gesture value from the one or more sensed gestures, and generate an output signal based on the first measured gesture value and the second measured gesture value. The initiation prompt may be used to select a function of the controlled device, wherein the first measured gesture value is used to select an initial state of a parameter of the function, and wherein the second measured gesture value is used to select a final state of the parameter of the function. The function is selected from the group of functions consisting of: the volume of an audio device, the temperature of an environmental control device, the size of a virtual object, the displacement of a virtual object, and zooming in or out.
[0268] Fig.15 15 is a graph showing extracted values of force extracted from a sensed pinch-out gesture of a user as a function of time according to some embodiments of the present invention. A user may perform a continuous adjustment gesture to reduce the size of the displayed portion of image 1510 (eg, a pinch-out gesture).
[0269] In some embodiments of the invention, the user may press the thumb and index finger together to indicate to the device the initiation of the continuous adjustment gesture. At time t1, the extracted value of the force may be equal to or higher than the start threshold 1501. The device may wait for a first time period (e.g., dt milliseconds) during which the user may adjust the distance between the index finger and the thumb to set the reference distance. After the first time period has passed (e.g., at t2) (1502), the device may extract a reference force value r (related to the reference distance).
[0270] In some embodiments of the invention, the user may reduce the distance between the index finger and the thumb. The displayed portion of the image may be reduced (1520). The displayed portion of the image may be reduced by zooming out the image, so that the diameter d is reduced. The diameter d of the image may be reduced by multiplying the diameter by a ratio of the extracted force value to the reference force value r (e.g., for zooming out, the ratio may be less than 1).
[0271] In some embodiments of the invention, the user may stabilize the index finger and thumb at time t3 1503 to indicate to the device that they wish to end the continuous adjustment gesture. For the end of the continuous adjustment gesture, the interface device may determine whether the force extraction value remains flat for a predetermined duration (between t3 1503 and t4 1504 and within limits 1509a and 1509b). The interface device may alternatively or additionally check whether the force extraction value drops sharply, as shown by the dashed line 1506 between t3 1503 and t4 1504.
[0272] Fig.16is a flow chart of a method for detecting a continuous adjustment gesture using an interface device according to some embodiments of the present invention. Method 1600 may include identifying an initiation prompt (1601). The initiation prompt may be a combination of a voice command, a gesture, a position (e.g., using position tracking), a prompt generated by the interface device, and / or checking whether the extracted force value is above a threshold. Method 1600 may include idling for a predetermined period of time (1602).
[0273] In some embodiments of the invention, method 1600 may include obtaining a reference value, such as a first value of an extraction parameter of a user's gesture or a measurement of a gesture from a user (1603). The reference value may be an extraction value of the force at the end of a predetermined time period. The extraction parameter may be related to the amount of force applied by the user on the finger. The extraction parameter may be related to the distance between the index finger and thumb of the user. The extraction parameter may be related to the amount of force applied by the user when pressing the thumb and index finger together. The extraction parameter may be related to the distance of the user from the sensor and / or the position of the user.
[0274] In some embodiments of the invention, method 1600 may include obtaining a second value of an extracted parameter of a user's gesture or a second value of a measurement from a user's gesture (1604). The second value may be obtained after the apparatus obtains the reference value.
[0275] In some embodiments of the invention, method 1600 may include performing a mathematical calculation between the second value and the reference value (1605). The interface device may generate an output signal based on the mathematical calculation. The output signal may be sent to a controlled electronic device for controlling the controlled electronic device. The mathematical calculation may involve a ratio, addition, subtraction, multiplication, division, or any mathematical transformation.
[0276] In some embodiments of the present invention, the interface device may recognize a user's voice command for identifying an initiation prompt, thereby prompting the interface device to idle for a first period of time (e.g., 1303 of method 1300). For example, a user may say "pinch to zoom" to indicate to the device that they wish to initiate a pinch to zoom gesture.
[0277] In some embodiments of the invention, the interface device may be configured to receive a user's voice commands, sounds, keywords, and tones for identifying initiating prompts (e.g., the device may have a library or repository of predetermined voice commands and keywords associated with the first condition).
[0278] In some embodiments of the present invention, the interface device may recognize a user's gesture for identifying initiation of a prompt, thereby prompting the interface device to be idle for a first period of time. For example, the user may quickly (or twice) press the index finger and thumb to indicate to the device that he wishes to initiate a continuous adjustment gesture. The interface device may be configured to recognize a user's gesture and movement for identifying initiation of a prompt (e.g., the device may have a library or repository of predetermined gestures and movements associated with initiation of a prompt).
[0279] In some embodiments of the present invention, the interface device may sense the user's position tracking data to identify the initiation prompt. When the interface device detects whether the position that the user is pointing to (e.g., on the display) is within the area (e.g., using position tracking), the initiation prompt can be identified. For example, the user can point to an object on the monitor. The device can be configured to perform position tracking (e.g., by sensing the user's movements and gestures) and detect where the user is pointing. The first condition can be met when the device detects that the user is pointing to a position within the area (e.g., the user is pointing to an image or button on the display).
[0280] In some embodiments of the invention, the interface device may have a library or repository of predetermined areas and buttons associated with initiation prompts.For another example, the interface device may be configured to recognize an initiation prompt when a user points to an object (eg, an image) for a period of time.
[0281] In some embodiments of the invention, a processor of an interface device (such as Figure 1C The processor 108 of the embodiment may be configured to identify an initiation prompt in one or more sensed gestures or in one or more sensed sounds. The processor may be configured to extract a first measured gesture value from the one or more sensed gestures, extract a second measured gesture value from the one or more sensed gestures, and generate an output signal based on the first measured gesture value and the second measured gesture value.
[0282] In some embodiments of the present invention, the initiation prompt can be used to select a function of the controlled device, wherein the first measurement gesture value is used to select an initial state of a parameter of the function. The second measurement gesture value can be used to select a final state of the parameter of the function. The function can be selected from the group consisting of: the volume of an audio device, the temperature of an environmental control device, the size of a virtual object, the displacement of a virtual object, and zooming in or out.
[0283] In some embodiments of the invention, the processor may be configured to identify an initiation prompt based on at least one prompt in a prompt group consisting of: a user's voice, a user's gesture, a value of a force measured from a user, and a signal generated by the processor. The processor may be configured to wait for a predetermined period of time after identifying the initiation prompt before extracting the first measurement parameter.
[0284] In some embodiments of the present invention, the processor may be configured to identify a termination prompt in the one or more sensed gestures or in the one or more sensed sounds. The processor may be configured to identify a termination prompt based on identifying an exit from a predetermined parameter range or identifying stability within a predetermined parameter range.
[0285] In some embodiments of the invention, the processor may be configured to generate the output signal based on a mathematical relationship selected from the group of mathematical relationships consisting of ratio, addition, subtraction, multiplication and division.
[0286] Fig.17 is a diagram of a user making a gesture while imitating a pattern in one or more patterns displayed on a display according to some embodiments of the present invention. In some embodiments of the present invention, an interface device for gesture and / or voice control of an electronic device can detect rhythmic path mimicry. Rhythmic path mimicry detection can involve sensing one or more continuous gestures performed by a user (such as a combination of the user's movement and the applied force), calculating a similarity metric between one or more gestures and a predetermined gesture, and checking whether the similarity metric is above a threshold. If the similarity metric is above a threshold, the interface device can generate an output signal to the controlled electronic device.
[0287] In some embodiments of the present invention, the display 1701 may present one or more predefined gestures (such as gestures displayed as patterns) to the user so that the user can imitate and / or follow one or more of these patterns. For example, the display 1701 may present a first pattern 1703 and a second pattern 1704 to the user 1705. For example, the first pattern 1703 may indicate to the user 1705 an instruction to perform a counterclockwise movement of the hand. The second pattern 1704 may indicate to the user 1705 an instruction to perform a clockwise movement of the hand.
[0288] In some embodiments of the present invention, the patterns displayed on display 1701 can each be associated with a specific command (e.g., different commands (such as "turn on the device", "select a channel", "display menu", etc.) and / or in some examples contradictory commands (e.g., "turn up the volume" or "turn down the volume", "increase the brightness" or "decrease the brightness", etc.)) and can be selected from a database of predetermined gestures and / or pattern repositories.
[0289] According to some embodiments, the user may be invited to select a desired command by selecting a pattern to be imitated. The user may be required to imitate the pattern by following the pattern and synchronizing with the pattern. For example, when the user selects to imitate pattern 1704, the user may be required to move the hand along the pattern 1704 (while pressing the fingers) and synchronize with the movement of the ball 1704a (the ball 1704a may move along the displayed pattern 1704 at a predetermined movement rate).
[0290] In some embodiments of the present invention, camera 1702 may obtain a video clip of user 1705. The interface device may include a video analysis algorithm for identifying gestures and movements of the user. User 1705 may imitate second pattern 1704 to perform gesture 1706. The interface device may detect and recognize gesture 1706 using the video obtained from camera 1702, thereby performing video analysis on the obtained video and comparing the sensed gesture 1706 with gestures from a database of predetermined patterns and gesture repositories.
[0291] In some embodiments of the present invention, the interface device may calculate a similarity measure between the sensed gesture (e.g., movement and applied force) and the predetermined gesture (presented as a pattern) based on the synchronization between the sensed gesture and the predetermined gesture. The interface device may use mathematical operations (such as correlation, parameter comparison, polynomial extraction, moving average, and comparison of the sensed gesture with the frames of the predetermined gesture) to calculate the similarity measure. When the interface device determines that the similarity measure is above a predetermined threshold, imitation is detected, and the interface device may generate an output signal to control the electronic device. The interface device may provide feedback to the user indicating that the imitation is detected (e.g., via display 1701).
[0292] In some embodiments of the invention, a processor of an interface device (such as Figure 1C The processor 108 of the device may be configured to calculate a similarity measure between the gesture in the one or more sensed gestures and the predefined gesture, and to generate an output signal only when the similarity measure is above a predetermined threshold. The processor may be configured to present a pattern to be imitated corresponding to the predefined gesture to the user via an output device. The processor may be configured to present a plurality of patterns to the user via an output device to select from the pattern to be imitated. The predefined gesture may be a variation or the opposite form of the pattern to be imitated. The processor may be configured to generate a pattern, and / or may select a pattern from a database of predefined patterns. The modality of the pattern may be selected from a modality consisting of: sound, vision, touch, and tactile. The output device may be selected from an output device consisting of: a display screen, headphones, earphones, a speaker, a tactile actuator, and a tactile actuator.
[0293] In some embodiments of the invention, the one or more sensed gestures may include sensing pressure applied between fingers or sensing pressure formed by a clenched fist. The one or more sensed gestures may also include sensing movement of a forearm of the user, wherein the movement is selected from the group of movements consisting of: roll, pitch, and yaw. The processor may be configured to determine whether the sensed pressure and the sensed movement are synchronized.
[0294] Fig.18A 1 is a diagram of a user wearing an interface device according to some embodiments of the present invention pressing fingers together at a certain intensity in accordance with instructions on a display of the interface device. User 1811 can wear interface device 100. Interface device 100 may include display 1812b. The interface device may present one or more patterns (e.g., from a pattern database) on the display. One or more patterns may be visually presented to the user on the display of interface device 100. For example, display 1812b may visually present a pattern for the user to imitate. The display may display, for example, meter bar 1812a. Bar 1812a may indicate to the user how hard the user should press fingers 1813 together. A full bar may indicate high pressure between pressed fingers. An empty bar may indicate low pressure between pressed fingers 1813. User 1811 may imitate the pattern displayed by bar 1812a on the display, for example, pressing fingers together in sync with the state of bar 1812a. If the bar is full, the user can intuitively press harder with a finger, and if the bar is empty, the user can intuitively release the pressure between the fingers. Any intermediate state of the bar can be satisfied by the user intuitively applying intermediate pressure between the fingers. If the pattern changes rhythmically, the device can detect the rhythmic path mimicry, and the user can imitate the pattern accordingly (e.g., synchronize with the pattern).
[0295] Fig.18B It is an illustration of a user of a wearable interface device according to some embodiments of the present invention pressing fingers together at a certain strength as indicated in instructions on augmented reality glasses. The user may wear glasses 1821 (e.g., virtual reality glasses and / or augmented reality glasses). Glasses 1821 may display one or more patterns. For example, glasses 1821 may display one or more bars associated with one or more patterns (or predetermined gestures). The interface device may use a sensor interface device 100 (e.g., the interface device 100 may include a biopotential sensor, an IMU, and other gesture sensors) to sense the force applied by the user between the fingers. The user may imitate one of the one or more patterns displayed by one or more bars on the display of the wearable device. The device may detect similarity (and / or synchronization) between the force applied when the user presses the fingers together (1823) and one of the one or more patterns.
[0296] Fig.18C 18 is a schematic diagram of a user wearing an interface device according to some embodiments of the present invention pressing fingers together at a certain intensity in response to a sound signal presented to the user via an audio headset 1831. The interface device 100 may include a database of one or more audio signal patterns stored in a memory of the device. For example, the headset 1831 may play a sound signal 1832 to the user that rises and falls in volume and / or pitch, thereby indicating a rising and falling pattern. The user can respond to the played audio signal by pressing the fingers together (1833) and adjusting the pressure according to the changed sound pattern 1832, thereby mimicking the pattern.
[0297] Fig.18D 1 is a diagram of a user wearing an interface device according to some embodiments of the present invention pressing fingers together with alternating intensities in accordance with vibrations of alternating intensities generated by the interface device on the user's wrist. The interface device may have a database of one or more vibration signal patterns stored on a memory of the device. The interface device 100 may include a tactile actuator unit 1841 (e.g., a vibration motor) for generating vibration signals. The interface device 100 may generate one or more vibration signal patterns 1845 for the user to imitate. The interface device 100 may generate vibrations 1842 that rise and fall in intensity and / or pulse rate to indicate a pattern that rises and falls in intensity. The user may imitate the vibration signal by pressing the fingers together (1843) under alternating pressure that imitates the pattern of the generated vibration signal.
[0298] Fig.19 1904 is a graph showing patterns according to some embodiments of the present invention and values of sensed pressure applied by a user pressing fingers together according to a pattern from a pattern database as a function of time. The interface device can present a first pattern 1902 and a second pattern 1903 to the user. The user can imitate the first pattern 1902 to alternate the pressure applied between the fingers, as presented by force lines 1904. The interface device can compare the measured force signal 1904 with patterns 1902 and 1903 to determine whether any of these patterns is actually imitated. The interface device can calculate a similarity measure between the value of the obtained force 1904 and each of the patterns 1902 and 1903. The interface device can determine that the sensed force signal 1904 is indeed similar to any of these patterns (pattern 1903 in this example).
[0299] In some embodiments of the invention, the interface device may calculate a similarity measure between the sensed force signal 1904 and each of the patterns presented to the user by the interface device. The interface device may detect similarity between the sensed force signal 1904 and patterns found to be similar to the signal.
[0300] Fig. 20 2001 and 2003. Graphs showing patterns according to some embodiments of the present invention and sensed force signals obtained from a user pressing fingers together opposite to the pattern (in an offset phase) as a function of time. The interface device may present pattern 2001 to the user. The user may press fingers together (2003) with force 2002, which may be an opposite form or variation of pattern 2001 (e.g., the sensed gesture is a predefined gesture derived from a variation or opposite form of a pattern to be imitated (such as pattern 2001)). For example, the interface device may present one pattern at a time, and the user may choose to imitate the pattern or the opposite form of the pattern (e.g., in a phase substantially 180 degrees relative to the presented pattern). The interface device may sense force 2002 and detect that there is a similarity between the opposite form of pattern 2001 and force 2002. When the interface device detects a similarity between the inverse of pattern 2001 and force 2002, the interface device may generate a first output signal, and when the interface device detects a similarity between pattern 2001 (as presented) and force 2002, the interface device may generate a second output signal.
[0301] In other embodiments of the invention, the interface device may detect similarities between force 2002 and the inverse form of pattern 2001 and conclude that mimicry exists (the interface device may ignore phase shifts between force 2002 and pattern 2001).
[0302] In some embodiments of the invention, the opposite form or variation of pattern 2001 can refer to an offset relative to pattern 2001 (e.g., a 90 degree, 180 degree, 270 degree offset), a negative (e.g., a substantially 180 degree offset of the periodic pattern, or a negative, mirrored and / or offset pattern compared to pattern 2001).
[0303] Fig.21 2 is a graph showing a pattern 2101 according to some embodiments of the present invention, a first sensed force signal 2103 obtained from a user pressing a middle finger and a thumb together according to the pattern (2103A), and a second sensed force signal 2102 obtained from a user pressing an index finger and a thumb together according to the pattern (2102A). The device can obtain one or more sensed force signals. The device can obtain the first force signal 2102 and the second sensed force signal 2103.
[0304] In some embodiments of the present invention, the device can detect a rhythmic path mimicry between a first sensed force signal and a second sensed force signal. The device can detect whether there is a similarity or correlation between the two sensed force signals obtained (e.g., 2102 and 2103). If the similarity measure between the first set of force values 2102 and the second set of force values 2103 is above a threshold, the device can detect a rhythmic path mimicry.
[0305] In some embodiments of the present invention, the device may calculate a first similarity metric between the sensing force signal 2102 and the pattern 2101, and may also calculate a second similarity metric between the second sensing force signal 2103 and the pattern 2101. The device may generate an output signal to the controlled electronic device based on the first similarity metric, the second similarity metric, or both. For example, the device may generate an output signal when and only when both calculated similarity metrics are above a certain threshold.
[0306] In some embodiments of the invention, the device may calculate similarity measures between different gestures, such as hand movement, pressure applied on the user's fingers, user position, predetermined patterns from a pattern repository, and patterns generated by the interface device.
[0307] Fig.22A 22 is a diagram of a user's hand performing a scrolling gesture while pressing fingers together according to some embodiments of the present invention, and a graph 2212 showing the scrolling of the hand as a function of time. The interface device 100 can detect similarities between the force measured when the user pinches the fingers together and the scrolling of the scrolling gesture 2211.
[0308] Fig. 22B 22 is a schematic diagram of a user's hand moving the hand vertically (up and down) while pressing the fingers together, and a graph 2222 showing the hand pitch as a function of time according to some embodiments of the present invention. The interface device 100 can detect the similarity between the force measured when the user pinches the fingers together and the pitch of the user's hand performing the vertical gesture 2221.
[0309] Fig. 22C 2231 is a schematic diagram of a user's hand moving the hand horizontally (left and right) while pressing the fingers together, and a graph 2232 showing the hand deflection as a function of time according to some embodiments of the present invention. Interface device 100 can detect similarities between the force measured when the user pinches the fingers together and the deflection of the user's hand performing horizontal gesture 2231.
[0310] Figure 23a is a graph showing the force values of a predefined pattern associated with a predetermined gesture according to some embodiments of the present invention. Figure 23b is a graph showing the extracted values of the force obtained from a continuous gesture performed by a user imitating the predefined pattern of Figure 23a. Figure 23c is a graph showing the calculated similarity measure between the predefined pattern of Figure 23a and the measured values of the force of Figure 23b. The first graph 2301 shows the force values of the predefined pattern associated with the predetermined gesture. The second graph 2302 shows the extracted values of the force obtained from the continuous gesture performed by the user imitating the predefined pattern of the first graph 2301. The interface device can measure the similarity between the first group of frames 2305 of the first graph 2301 and the second group of frames 2306 of the second graph 2302. The first group of frames 2305 and the second group of frames 2306 can represent parts (e.g., time windows) of the first graph 2301 and the second graph 2302, respectively.
[0311] In some embodiments of the invention, a third graph 2307 may illustrate a similarity metric calculation between the predetermined pattern shown in the first graph 2301 and the extracted force values obtained from the user shown in the second graph 2302. The processor of the interface device may calculate a similarity metric value between the frame 2305 of the first graph 2301 and the frame 2306 of the second graph 2302. When the value of the similarity metric at frame 2308 is above a high first threshold value (denoted as Hi Th), the processor may start counting frames. The processor may count the number of consecutive frames for which the value of the similarity metric is above the high first threshold value. When the similarity metric is below a low second threshold value at point 2309, the processor may stop counting frames.
[0312] In some embodiments of the present invention, if the number of counted frames is higher than a predetermined number n, the processor may detect a rhythmic path mimicry between the predetermined pattern and the gesture performed by the user.
[0313] Fig.24 2401 is a flow chart of a method for detecting rhythmic path mimicry using an interface device according to some embodiments of the present invention. Method 2401 may include sensing a user's gesture to detect rhythmic path mimicry by calculating a similarity metric with a predefined gesture. Method 2401 may include calculating a similarity metric between two frames (2402), wherein one of the two frames may be a frame extracted from the sensed user's gesture and a second of the two frames may be a frame extracted from a predefined gesture (or pattern). Method 2401 may include checking, by the interface device, whether the similarity metric is above a first threshold (2403), and if so, the interface device may start counting frames (2404). Otherwise, the method may return to step 2402 until the similarity metric is above the threshold.
[0314] In some embodiments of the invention, method 2401 may include calculating a similarity metric between two frames (2405), and checking, by the interface device, whether the similarity metric is below a second threshold (2406). In order for the interface device to detect rhythmic path mimicry, the interface device may count a predetermined number of frames for which the similarity metric is not below the second threshold. If the similarity metric is below the second threshold (e.g., before the count of the predetermined number of frames is reached), the method may return to calculating the similarity metric between the two frames (2402) until the similarity metric is above the first threshold (2403).
[0315] If the similarity metric is not below the second threshold, the method 2401 may include checking, by the interface device, whether the number of frames is above a predetermined value (2407). If not, the method may return to calculating the similarity metric between the two frames (2405) until the number of frames exceeds the predetermined number. If the number of frames is above the predetermined value, the method 2401 may include detecting a rhythmic path mimicry (2408). When a rhythmic path mimicry is detected, the interface device may generate an output signal to the electronic device, provide feedback to the user to indicate that a rhythmic path mimicry is detected, and return to calculating the similarity metric between the two frames (2402).
[0316] Fig.25 2500 is a flow chart of a method for detecting rhythmic path mimicry using an interface device according to some embodiments of the present invention. Method 2500 may include providing a pattern to a user. Method 2500 may include sensing one or more gestures (2501). Method 2500 may include calculating a similarity metric (2502) based at least on the one or more gestures and a predefined gesture (e.g., provided to the user as a pattern). Method 2500 may include generating an output signal if the similarity metric is above a predefined threshold.
[0317] Fig.26 Schematic diagram of a user's hand controlling a cursor on a display according to a wearable interface device of some embodiments of the present invention. In some embodiments of the present invention, a gesture and / or voice controlled interface device can control a cursor on a display (e.g., an air mouse). The interface device can control an air mouse (e.g., move a cursor around and / or click) by identifying a user's gesture and / or force and / or movement (e.g., using an IMU, a biopotential sensor, and / or a camera). For example, the interface device can move a cursor based on the user's hand movement. The interface device can perform a click based on an identified user's gesture (e.g., pressing the index finger and thumb together) and / or a measured user's force (e.g., when the measured force is above a threshold).
[0318] In some embodiments of the present invention, the user can use an air mouse to move the cursor 2603. When the user moves the hand, the IMU of the interface device 100 can detect the movement of the hand. The device can move the cursor 2603 on the display 2602 according to the movement of the hand.
[0319] In some embodiments of the present invention, the user can select a function (e.g., a pen type or a brush) from the toolbar 2605. For example, the user can select a brush 2606 by moving the cursor 2603 to the brush 2606. The user can perform gestures for selecting different functions and brushes. For example, the user can select the brush 2606 by pressing the index finger and thumb together. The user can select the color of the brush 2606 by selecting a color using the cursor 2603. The user can use gestures to select the color of the brush 2606.
[0320] In some embodiments of the invention, the user may draw a shape by pressing the index finger and thumb together 2604. After the force measured from the user is above a threshold, the air mouse device may begin drawing a line. For example, after the user presses the index finger and thumb together and the measured force is above a threshold, the device may draw a line.
[0321] In some embodiments of the present invention, the user can change the width of the line according to the strength of the force he applies on his fingers. For example, when the user presses the index finger and thumb together with force, the line can become wider. When the user presses the index finger and thumb together lightly, the line can become narrower.
[0322] Fig. 27 1 is a flow chart of a method for using an interface device to move a cursor, compensating for rotation of an IMU, according to some embodiments of the present invention. In some embodiments of the present invention, the interface device can control the cursor based on movement measurements from the IMU. When the user moves the arm, the interface device can convert the movement of the user's hand into movement of the cursor. The interface device can move the cursor while ignoring the rotation of the IMU (e.g., when the IMU rotates based on the rotation of the user's hand).
[0323] In some embodiments of the invention, method 2701 may include defining p as a unit vector (e.g., a raw unit vector) on the rotation axis when the IMU is detected to have rotated by q (2702). The interface device may compensate for the rotation of the IMU around the rotation axis to prevent accidental movement of the cursor.
[0324] In some embodiments of the present invention, the interface device may perform a rotation transformation of a unit vector p multiplied by q according to the following formula:
[0325] p′=qpq -1 (10)
[0326] In the rotation transformation equation (10), p" can represent a new unit vector that has undergone the rotation transformation. The rotation of q around the rotation axis can be multiplied by the original unit vector and by q representing the reciprocal of q. -1 .
[0327] In some embodiments of the invention, method 2701 may include finding the difference between p and p' to determine the movement of the cursor:
[0328] dx=p′ x -p x (11)
[0329] dy=p′ y -p y (12)
[0330] dx can represent the difference along the x-axis of the cursor, p' x The x component of the p' unit vector can be represented by the rotation, and p x can represent the x-component of the original unit vector p. dy can represent the difference along the y-axis of the cursor, p' y can represent the y component of the rotated p' unit vector, and p y can represent the y component of the original unit vector p.
[0331] In some embodiments of the invention, method 2701 may include moving the cursor position by:
[0332] (dx·h,dy·v) (13)
[0333] h may represent a parameter of a movement factor of the cursor along the horizontal axis x of the cursor (eg, horizontal speed). v may represent a parameter of a movement factor of the cursor along the vertical axis y of the cursor (eg, vertical speed).
[0334] Fig.28 28 is a graph showing horizontal movement of a cursor and horizontal movement of a user's hand controlling the cursor according to some embodiments of the present invention. Graph 2801 shows movement of the cursor along the vertical axis of the screen over time. Line 2803 is movement of the cursor according to method 2701, wherein the interface device compensates for the rotation of the IMU. Line 2802 is movement of the cursor, wherein the interface device moves the cursor according to the rotation of the IMU.
[0335] Fig.29is a flow chart of a method for controlling a cursor using an interface device having an IMU and a biopotential sensor according to some embodiments of the present invention. In some embodiments of the present invention, method 2901 may include identifying a user's gesture (2902). Method 2901 may include using the IMU to sense movement of a user's hand. Method 2901 may include selecting a function based on the gesture (2904). Method 2901 may include moving a cursor based on the sensed movement of the user's hand.
[0336] Fig. 30A 3003 is a schematic diagram of a user's hand rotating a wrench according to some embodiments of the present invention, and the hand is worn with an interface device for estimating force. The user can hold the wrench 3002 with hand 3001 to tighten the nut 3003. The user can apply force in the clockwise direction. The user can wear the interface device 100. The interface device 100 may include one or more biopotential sensors and an IMU. The one or more biopotential sensors can measure the force applied by the user on the wrench 3002 to tighten the nut 3003. When the nut 3003 is tightened, the nut 3003 may not be rotated further clockwise, and the wrench 3002 may not be rotated further clockwise. When the wrench cannot be rotated further clockwise and the measured rotational movement (speed) is close to zero, the force can be estimated by measuring the force applied by the user.
[0337] In some embodiments of the invention, the interface device may be configured to estimate the weight of an object or the force applied. The interface device may include one or more biopotential sensors configured to be placed on a limb of the user and measure one or more biopotential signals from one or more muscles and nerves of the limb. The interface device may include a processor to obtain the measured one or more biopotential signals and estimate the weight of an object supported by a limb of the user or the weight of a force applied by the limb based on the one or more biopotential signals measured about a known weight or a known force.
[0338] In some embodiments of the present invention, the interface device may be configured to detect the successful connection or disconnection of a component of at least one type of connector to assess whether it is correctly connected or disconnected. This may be beneficial on a production line where the connector is forgotten to be assembled, or a poor assembly of the connector may cause product failure, safety issues, and may require great effort to detect and resolve the problem at the end of the assembly process. The interface device may include one or more biopotential sensors configured to be placed on the user's wrist and measure one or more biopotential signals from one or more muscles and nerves of the wrist. The interface device may be configured to estimate the force applied to tighten the connector. The interface device may include a microphone and may be configured to record sounds formed when tightening the connector, such as a click of the connection or a friction sound from the connector. The interface device may include an inertial measurement unit and may be configured to record accelerations formed when tightening the connector or the orientation of the hand holding the connector. The device may be configured to store samples of sound, biopotential or acceleration and the direction of the user's hand while correctly tightening at least one type of connector in a memory group of the correct example. The device may be configured to store samples of sound, biopotentials, or accelerations and the orientation of at least one hand of at least one user while correctly disconnecting at least one type of connector in the memory bank. The device may be configured to store samples of sound, biopotentials, or accelerations and the orientation of at least one hand of at least one user while making other unrelated movements and gestures to the memory bank. In some embodiments of the invention, a neural network configured as a 2-class classifier may be trained on the stored samples to detect the correct tightening of only one type of connector. In some embodiments of the invention, a neural network configured as a meta-learner may be trained to map the stored samples to a metric feature space such that samples of correctly tightening one type of connector will be mapped to coordinates that are close to each other and as far away as possible from coordinates that connect or disconnect other types of connectors and other unrelated movements and gestures. The meta-learner can be further trained to map the stored samples to the metric feature space so that samples obtained from a particular user will be mapped to coordinates that are closer to the same user and as far away from the coordinates of other users as possible, and so that samples of correctly tightening one type of connector by the user will be mapped closer to each other and away from samples of tightening other connectors within the area in the metric feature space to which the user samples are mapped, which is away from the area in the metric feature space to which other user samples are mapped.
[0339] In some embodiments of the present invention, the interface device can be configured to detect whether a new action performed by the user correctly tightens a type of connector by presenting a new sample from at least one of the following to a trained 2-class classifier: at least one biopotential sensor, a microphone, and an inertial measurement unit.
[0340] In some embodiments of the invention, the interface device may be configured to detect whether a new action performed by a user correctly tightens a connector by first using a trained meta-learner to map new samples from at least one of the following: at least one biopotential sensor, a microphone, an inertial measurement unit to coordinates on a metric feature space, and then checking whether it is mapped within a region of the metric space that correctly tightens a particular connector.
[0341] In some embodiments of the present invention, the detection accuracy of each user can be improved by using a trained meta-learner to map a set of exemplary samples of new users of each type of connector that are correctly fastened or disconnected from at least one of the following: at least one biopotential sensor, microphone, inertial measurement unit to coordinates on a metric feature space to obtain a set of coordinates for each action of the user, and training a classifier to classify the actions performed by the user based on their mapped coordinates.
[0342] Fig. 30B FIG. 2 is a diagram showing a method for estimating force according to some embodiments of the present invention. Fig. 30A Graph 3004 shows measurements of force and rotational velocity of a hand. Graph 3004 shows measurements 3005 from a biopotential sensor. Force f can be extracted from the measurements of the biopotential sensor. Rotational movement ω can be measured from the IMU. Graph 3004 shows that over time, force f increases, indicating that the user is applying more force to tighten the nut 3003, while rotational movement ω decreases, indicating that the wrench can not be turned further clockwise. When the value of the force data approaches a maximum value and the value of the movement data approaches a minimum value (e.g., close to zero), the force can be estimated by identifying the value of the force.
[0343] In some embodiments of the present invention, the interface device can estimate the force by comparing the value of the force extracted when the user turns the wrench 3002 (to tighten the nut 3003) with the value of the force extracted from the baseline reading (and calibration). When the user rotates a lever (such as a wrench 3002), the interface device 100 can obtain a baseline reading by extracting the value and measurement of the force from the IMU and combining the reading from the dynamometer (e.g., a dynamometer attached to the wrench). The interface device can obtain multiple baseline readings to have a sufficient sample database. For example, when the user turns different wrenches and nuts and / or is in different directions, the interface device can obtain a baseline reading. The processor of the interface device can be configured to apply a meta-learner regressor to all baseline readings of the user and obtain regression parameters specific to the user. Based on the user's regression parameters, when the user turns the wrench 3002, the processor can then use the extracted force value to estimate the force.
[0344] Fig.31A 3101 is a flow chart of a method for estimating force using an interface device having an IMU and a biopotential sensor according to some embodiments of the present invention. In some embodiments of the present invention, method 3101 may include obtaining force data from one or more biopotential sensors (3102). Method 3101 may include obtaining movement data from the IMU (3103). Method 3001 may include using the interface device to estimate force by identifying a value of force when a value of the force data is close to a maximum value and a value of the movement data (e.g., angular velocity or linear velocity) is close to a minimum value (3104).
[0345] Fig.31B is a flow chart of a method for calibrating an interface device to estimate force according to some embodiments of the present invention. Method 3111 may include a user holding a lever (such as a wrench with a dynamometer) so as to hold it at a known distance from a bolt (3112). It may be important for the user to hold the lever at the same distance from the rotating bolt so that the estimation and calibration of force are consistent. Method 3111 may include a user rotating the lever to close or open the bolt using only the user's wrist or another measured muscle (3113). It may be important for the user to use the same muscle group and the same movement when applying force so that the estimation and calibration of force are consistent.
[0346] In some embodiments of the invention, method 3111 may include obtaining data from biopotential sensor data and / or obtaining movement data from IMU and ergometer data (3114). The processor may obtain biopotential sensor data and / or movement data from IMU and ergometer data.
[0347] In some embodiments of the present invention, method 3111 may include checking, by a processor, whether sufficient force estimation samples in two directions (e.g., clockwise and counterclockwise rotation directions, or closing and opening directions) have been obtained from a sufficient number of different users (3115). "Sufficient samples" means that the model constructed based on the collected data allows sufficiently accurate force estimation (e.g., sufficiently small estimation errors). When the number of samples is insufficient, method 3111 may include returning to step 3112 to obtain additional samples. When the number of samples is sufficient, method 3111 may include adjusting the meta-learner regressor on all samples and deriving a feature space (e.g., a robust feature space) (3116).
[0348] In some embodiments of the invention, method 3111 may include adjusting the model for a new user via a meta-learner so that the regression parameters are specific to that user ( 3117 ).
[0349] Fig.31C is a flow chart of a method for estimating force using an interface device according to some embodiments of the present invention. Method 3121 may include holding a lever (such as a wrench with a dynamometer) by a user, wherein the user holds the lever at a known distance from a rotating bolt (3122). Method 3121 may include using the user's wrist or a specified muscle group to rotate the lever to close or open the bolt (3123). Method 3121 may include obtaining data from a biopotential sensor and / or obtaining movement data from an IMU (3124). Method 3121 may include running a conditioned meta-learner on the obtained data to infer the applied force and estimate the force (3125).
[0350] In some embodiments of the invention, based on the estimated force, the applied torque may be calculated and presented.
[0351] Fig.32A is a schematic diagram of a user's hand lifting an object, the hand being worn with an interface device for estimating the weight of the object, according to some embodiments of the present invention. Fig.32B is a diagram showing a method for estimating Fig.32A The weight of the object Fig.32A A graph of hand force measurements. Fig.32C is a diagram showing a method for estimating Fig.32A The weight of the object Fig.32AThe interface device 100 can estimate the weight of an object such as a cup 3201. The user can wear the interface device 100 (e.g., on their wrist or around a bicep), and the interface device 100 can measure the user's gestures and movements and extract the value of the force applied by the user (e.g., the force applied when holding and lifting the cup 3201). The interface device can extract the value of the force when the user holds the cup 3201 without lifting the cup (e.g., performing a baseline reading or calibration). The interface device can estimate the weight of the cup 3201 by extracting the value of the force when the user lifts the cup 3201 and comparing the extracted value of the force to the extracted value of the force obtained during the baseline reading.
[0352] The user can wear the interface device 100 around the biceps. The interface device can obtain a baseline reading of the force extraction value from the user's biceps muscle fibers. The interface device can obtain a baseline reading when the user holds but does not lift the cup 3201 while the user's forearm is substantially 90 degrees relative to the biceps. The interface device 100 can obtain multiple baseline readings when the user holds different cups with different weights. The processor of the interface device can be configured to apply a meta-learner regressor to all baseline readings of the user and obtain regression parameters specific to the user. Based on the user's regression parameters, the processor can then use the force extraction value to estimate the weight of the cup 3201 when the user lifts the cup 3201 (while flexing the biceps).
[0353] The user can wear the interface device 100 around the wrist. The interface device can obtain a baseline reading of the extracted value of the force associated with wrist abduction, wrist flexion, and / or finger clenching from the user's muscle fibers. The interface device can obtain a baseline reading when the user holds but does not lift the cup 3201. For example, when the user wraps his fingers around the cup 3201 without lifting the cup 3201, a baseline of wrist abduction and / or finger clenching can be obtained by the interface device. When the user places the cup 3201 on an open palm without lifting the cup 3201, a baseline reading of wrist flexion can be obtained by the interface device. The interface device 100 can obtain multiple baseline readings (e.g., using cups with different weights) and calculate regression parameters specific to the user. Based on the user's regression parameters, the processor of the interface device 100 can then use the extracted value of the force when the user lifts the cup 3201 to estimate the weight of the cup 3201.
[0354] In some embodiments of the present invention, the interface device can distinguish between the force extracted from the user's measurements when the user lifts the cup 3201 and when the user holds the cup 3201 without lifting the cup 3201. The interface device 100 can measure the speed of the user's arm movement, as shown in line 3203. When the user holds the cup 3201 without lifting the cup 3201, the speed is substantially zero, while the extracted value of the force (shown in line 3202) may change. At time 3204, when the user begins to lift the cup 3201, their arm moves. When the speed is not zero at time 3204, the interface device can determine that the cup 3201 is lifted.
[0355] Fig.33A 3301 is a flow chart of a method for estimating the weight of an object using an interface device having an IMU and a biopotential sensor according to some embodiments of the invention. In some embodiments of the invention, method 3301 may include obtaining force data from one or more biopotential sensors (3302). Method 3301 may include obtaining movement data from the IMU (3303). Method 3301 may include using the interface device to estimate weight by identifying a value of force when the value of the force data is close to a maximum value and before the value of the movement data is greater than zero (3304).
[0356] Fig.33B 3311 is a flow chart of a method for calibrating an interface device to estimate the weight of an object according to some embodiments of the present invention. Method 3311 may include holding an object having a known weight by a user (3312). It may be important for the user to hold the object in the same manner in each measurement so that the weight estimation and calibration are consistent. Method 3311 may include holding the user's hand steadily so that the measured biopotential signal can be obtained from the muscle or muscle group that holds the object against gravity (3313). It may be important for the user to use the same muscle group and the same movement when holding the object so that the weight estimation and calibration are consistent.
[0357] In some embodiments of the invention, method 3311 may include obtaining data from a biopotential sensor and / or obtaining movement data from an IMU and ergometer data (3314). The processor may obtain the biopotential sensor data and / or movement data from the IMU and ergometer data.
[0358] In some embodiments of the invention, method 3311 may include checking, by the processor, whether sufficient weight estimation samples from sufficient different users lifting sufficient different weights have been obtained from the relevant muscles (3315). Similarly, "sufficient samples" means that the model constructed based on the collected data allows for sufficiently accurate weight estimation (e.g., sufficiently small estimation errors). When the number of samples is insufficient, method 3311 may include returning to step 3312 to obtain additional samples. When the number of samples is sufficient, method 3311 may include building a meta-learner regressor model on all samples and deriving a feature space (e.g., a robust feature space) (3316).
[0359] In some embodiments of the invention, method 3311 may include conditioning the new user via the meta-learner so that the regression parameters are specific to that user (3317).
[0360] Fig.33C is a flow chart of a method for estimating the weight of an object using an interface device according to some embodiments of the present invention. Method 3321 may include obtaining an object for estimating the weight of an object by a user (3322). Method 3321 may include holding the user's hand stably so that the muscle or muscle group holding the object against gravity is stable when obtaining data (3323). Method 3321 may include recording 3324 biopotentials formed by the contraction of the muscle against gravity and / or movement data from the IMU. Method 3321 may include running a conditioned meta-learner on the recorded data to estimate the weight of the object (3325).
[0361] In some embodiments of the present invention, a gesture and voice controlled interface device may be configured to recognize gestures performed by a user for interacting with a virtual object, and provide feedback to the user for enhancing the user's virtual experience via sensory substitution. Sensory substitution may refer to the user's perception of a virtual object. When the user interacts with a virtual object, the interface device may provide feedback to the user to provide a perception of the quality of the virtual object (such as the weight, texture, elasticity, and mobility of the virtual object). Sensory substitution may enhance the user's virtual experience because the user may perceive (e.g., feel) the quality of the virtual object (mimicking interaction with a physical object). The sensory substitution provided by the interface device may be used for simulations (e.g., guides for interacting with physical objects such as operating machines, flight and driving simulations, simulations of performing surgery on patients), virtual and augmented reality, and games (e.g., video games on computing devices or gaming platforms such as the Metaverse, where the user's experience is enhanced by sensory substations and similar functions of the interface device).
[0362] In some embodiments of the present invention, the processor may be configured to obtain one or more sensed gestures from the one or more gesture sensors, analyze the one or more sensed gestures to identify input from the user, and generate an output signal corresponding to the input to the controlled device, wherein the output signal is modulated by a modulation function. The modulation function simulates a physical parameter associated with a virtual object or a virtual environment.
[0363] A modulation function is the way in which a parameter of a virtual object (such as weight, texture, elasticity, etc.) changes the virtual object's response to a user's interaction with the object or input (e.g., pressing, lifting, touching the object) via feedback such as visual display, sound, haptics, etc. For example, a virtual ball may react to a user squeezing the ball by changing its visual size based on the pressure applied by the user, and the ball's physical parameters may modulate the ball's response. For example, a sponge ball would have a pressure modulation function expressed as 1-e -压力 - Its visual size will decrease rapidly at first, but any increase in pressure will produce a smaller change in its size, whereas the size of a hard ball will not react to pressure. For example, the texture can be simulated by changing the voltage applied to a speaker to form a sound or the voltage applied to a tactile actuator through a high-pass filter. The texture can also be simulated by the size of pixels that a visual cursor moves across a screen. The modulation function of the texture is a two-dimensional function or matrix, such as a sine, triangle, or spike sequence, describing the height of the surface at each coordinate on the surface. When the position of the user's hand changes, the modulation function provides the height of the surface at that position as an output function. This function is sent to a feedback device to create the illusion of a real texture, and a high-pass filter can be applied to the output function to prevent any feedback from occurring when the hand position is static.
[0364] Fig.35A A user's wrist is depicted with a wearable interface device 3501 configured to detect pressure applied by a user wearing a heads-up display 3502, thereby displaying a virtual hand 3503 squeezing a virtual ball 3504, according to some embodiments of the present invention. Fig.35B is a graph 3505 of a modulation function depicting the radius of the virtual ball as a function of the estimated pressure applied by the user.
[0365] Fig.36 36 shows a user wearing a wrist interface device and headphones according to some embodiments of the present invention. The user moves his hand in front of a screen 3602 showing a virtual hand 3604, which moves according to the detected movements of the user's hand 3601, while touching a virtual surface 3603 with a sinusoidal texture. The user gets tactile feedback 3605 on his wrist and hears the sound of the hand being touched by the user. Fig.37D The sound feedback 3606 is determined by the virtual pattern shown in . Fig.37Ais a graph of a sinusoidal modulation function depicting a virtual pattern as a function of position along the width of the screen 3602. Fig.37B is a graph describing possible movements of the hand along the x-axis as a function of time. The hand starts out motionless, then quickly moves to the center of the screen and waits, and then continues to move to the end of the screen and stops. Fig.37C It shows that when Fig.37A The modulation function is applied to the Fig.37B A graph of texture values as a function of time for moving hand positions. In this graph, the feedback strength can remain high when the hand is still in the center of the screen. Fig.37D Feedback after a high pass filter is shown. Here, when the hand is not moving, the feedback value is zero. Thus, according to some embodiments of the present invention, a processor of an input device may modulate one or more properties of one or more sensed gestures using a modulation function, and use the modulated one or more properties of one or more sensed gestures to provide feedback to a user using a feedback device.
[0366] In some embodiments of the present invention, the interface device may be linked to a screen. The screen may display a virtual object for interaction with the user. The user may perform a gesture to attempt to lift the virtual object. The interface device may provide feedback to the user to provide a sense of the weight of the virtual object. The interface device may display that the virtual object begins to rise when the measured force applied by the user (e.g., the user presses the fingers together) is above a threshold. For example, the heavier the virtual object to be sensed, the higher the threshold (e.g., the user is required to apply more force to lift the virtual object). The user may perform a gesture for squeezing the virtual object (e.g., clenching their hands as if grasping a physical object). The interface device may display deformations in the virtual object based on the gesture. For example, when the user performs a squeezing gesture to provide a sense of a mailable object, the interface device may display deformations in the virtual object. To provide a sense of a rigid object, the interface device may display slight or no deformations in the virtual object. To provide a sense of an elastic object, the interface device may display deformations in the virtual object during the gesture and return the virtual object to its original shape after the gesture ends (e.g., for a sense of a plastic object, the deformations still exist after the gesture ends).
[0367] The user may perform a gesture for feeling the texture of a virtual object (e.g., moving an open hand in space as if touching or brushing the virtual object). The interface device may display a cursor (e.g., shaped like a hand) brushing over the virtual object based on the gesture. For example, to provide a sense of a textured or rough object, the interface device may display a cursor that moves slowly over the surface of the virtual object (e.g., compared to a less rough object). To provide a sense of a smooth or slippery object, the interface device may display a cursor that moves quickly over the surface of the virtual object (e.g., compared to a less smooth virtual object).
[0368] The interface device may provide secondary feedback to the user to enhance the user's virtual experience via sensory substitution. The interface device may have a tactile actuator that provides feedback to the user through vibration. For example, when the user interacts with a heavier or rougher virtual object, the tactile actuator may vibrate violently (compared to a lighter or smoother virtual object). The interface device may display visual effects on the screen to enhance the user's virtual experience via sensory substitution. For example, the interface device may display a meter (such as a bar) to indicate the magnitude of the force required for the user to interact with the virtual object. The interface device may display visual effects juxtaposed with the virtual object (e.g., surrounding the virtual object), wherein the virtual effect is modified based on the user's interaction with the virtual object. The interface device may display visual effects, such as a dotted line around the virtual object, which may rotate, change color, and / or expand circumferentially to indicate the quality of the virtual object (and / or the quality of the interaction with the virtual object). For example, the dotted line may rotate at a predetermined rate to indicate that the user is performing the virtual task correctly. The dotted line may expand circumferentially to indicate that the virtual object is lifted. Users can intuitively perceive the quality of virtual objects (and the quality of interaction with virtual objects) through the display of visual effects.
[0369] Fig.34A is a flow chart of a method 3401 for training a metric-based meta-learner to uniquely identify a user based on gestures and sounds made by multiple users according to some embodiments of the present invention.
[0370] The method 3401 may include recording 3402 a biopotential from one / multiple biopotential sensors on a hand of a user performing a predefined gesture. The method 3401 may also include recording 3403 a voice of the user saying a predefined word.
[0371] The method 3401 may also include determining 3404 whether enough samples are obtained from a sufficient number of users. If there is not a sufficient number of samples or a sufficient number of users, there may be, for example, a predetermined sufficient number of required samples and a predetermined number of sufficient users or a decision to learn, then further recording of biopotentials 3402 and further recording of speech proceeds with the selection of a user 3403 from more users. If there is a sufficient number of samples or sampled users, the method 3401 also includes establishing 3405 a gesture metric space for the configured gestures, such that samples from different users and different gestures will be far away from each other, while collecting the same gestures in clusters and collecting gestures of the same user in internal clusters, and establishing 3406 a voice metric space for the configured voices, such that samples from different users and different words will be far away from each other, while collecting the same words in clusters and collecting words of the same user in internal clusters.
[0372] Fig.34B is a flow chart of method 3411 for tuning a metric-based meta-learner to a particular user's personal gesture signature and voice signature according to some embodiments of the present invention.
[0373] The method 3411 may include recording 3412 biopotentials from one / multiple biopotential sensors on a hand of a user performing a predefined gesture and mapping them to a gesture metric space.
[0374] The method 3411 may also include determining 3413 whether sufficient samples were obtained. If there are not sufficient samples, for example, there may be a predetermined sufficient number of required samples or a learned decision, and further recording and mapping of biopotentials is performed. If there are sufficient samples, the method 3411 also includes finding 3414 the center c of the cluster of the mapped new gesture on the gesture metric space. g and extensions g Method 3411 also includes recording 3415 the speech of the user speaking the predefined words and mapping it to the speech metric space. Method 3411 may also include determining 3416 whether sufficient speech samples are obtained. If not, further recording 3415 the speech of the user speaking the predefined words and mapping it to the speech metric space. If a sufficient number of speech samples are obtained, method 3411 also includes finding 3417 the center c of the cluster of the mapped new words on the speech metric space. v and extensions v .
[0375] Fig.34Cis a flow chart of a method 3421 for authorizing user input based on a personal gesture signature and voice signature using an interface device according to some embodiments of the present invention. The method 3421 may include recording 3422 biopotentials from one / multiple biopotential sensors on the hand of a user performing a predefined gesture and mapping them to a gesture metric space. The method 3421 may also include calculating 3423 coordinates of the recorded gesture on the gesture metric space and c g The distance between g The method 3421 also includes recording 3424 the speech of the user speaking the predefined words and mapping it to the speech metric space. The method 3421 may also include calculating 3425 the coordinates of the recorded speech on the speech metric space and c v The distance between v The method 3421 may also include determining a distance d of the recorded gesture (from the center) g Is it smaller than gesture s? g If not, it is determined that the user is unauthorized. If yes, the distance d of the recorded speech (from the center) is determined v Is it smaller than the extension s of the voice? v If not, the user is determined to be unauthorized. g< s g and d v< s v True, the user is authorized to provide input.
[0376] In some embodiments of the present invention, the interface device can be used to detect: gestures made by moving the user's fingers, gestures made by moving the user's limbs, facial gestures made by moving the user's eyes, between two fingers of the user's hand, pressure applied by pressing the fingers of the user's hand against each other, force applied by the user's muscles, user actions (such as connecting a connector using one or more of the following sensors: at least one bioelectric sensor, IMU, PPG, microphone), where the sensor data passes through a pipeline including sensor data preparation, data upper representation, convolutional neural network (CNN) layer and output layer.
[0377] Sensor data preparation refers to the resampling, alignment, and normalization of sensor data from different sensors to fit into the same time frame and value range.
[0378] Upper representation refers to transforming sensor time series data into multiple time series of different frequency bands (MFCC) using methods such as continuous wavelet transform (CWT), discrete wavelet transform (DWT), short-time Fourier transform (STFT), Mel-frequency cepstral coefficients, etc. The choice of upper representation method depends on different sensor types and their uses. For example, it is well known that CWT best represents biopotential signals, while MFCC is often used for audio keyword detection. Upper representation creates a 2-dimensional heat map to graphically represent the events recorded by the sensor, for example, gesture events can be seen as dense spots of high values in the mid-frequency band, while noise can be seen as extended low values surrounding the heat map.
[0379] In some embodiments of the invention, a convolutional neural network is trained as a meta-learner that maps over-represented heatmaps to output vectors or coordinates on a metric feature space. During the training process, the center of a cluster ck is calculated for signals of a class ck having sk examples of that class by averaging the network output values for each sensor, converting the distance between the example center and its one-hot encoded label into a probability using, for example, SoftMax, and calculating a loss function (such as the cross entropy over the calculated probabilities and the one-hot encoded labels).
[0380] In some embodiments of the present invention, the interface device is configured to detect gestures, finger gestures, and other user actions by training a meta-learner to map signals from at least one biopotential sensor and an inertial measurement unit acquired from many users, so that examples of gestures by the same user will be mapped close to a shared cluster center and away from cluster centers of examples of other gestures and actions made by other users.
[0381] In some embodiments of the present invention, the above neural network can be trained as a regressor for estimating values, such as the pressure applied between fingers, the distance between fingers, the force applied by muscles. Auxiliary sensors (such as pressure sensors, distance sensors or force sensors) and signals from sensors (such as biopotential sensors, PPG, IMU) are used to simultaneously acquire ground truth signals. The network is trained to minimize a regression loss function, such as L = log(cosh(f(x i )-y i )), where y i is based on auxiliary ground truth, and f(x i ) is a single-valued network output. In some embodiments, multiple regression outputs may be trained to estimate the derivative of the auxiliary signal.
[0382] In some embodiments of the invention, the above pipeline can be configured to detect keywords recorded by a user using a microphone. The MFCC method can be used to represent the speech signal as a heat map. A neural network can be trained as a classifier where the output vector size is the number of keywords, such that p k represents the probability that the keyword belongs to category k. The network is trained using the classification loss of the difference between the network output vector and the actual class label.
[0383] In some embodiments of the present invention, the interface device can be used to use sensors (such as cameras, depth cameras, lidar, sonar, radar, etc.) to detect gestures made by moving the user's fingers, gestures made by moving the user's limbs, facial gestures made by moving the user's eyes, and the distance between two fingers of the user's hand through sensor data including a pipeline of dense segmentation, skeletonization and classification.
[0384] Dense segmentation is a pixel-by-pixel fine segmentation that produces a semantic segmentation that is carefully trained using a UV coordinate estimation method. Skeletonization is a keypoint detection network that produces a set of keypoint coordinates, such as finger base coordinates, fingertip coordinates, eye pupil coordinates, etc.
[0385] In some embodiments of the invention, a classifier is trained to recognize gestures from the positions of the skeletonized output.
[0386] In some embodiments of the invention, a dense segmentation and skeletonization network is trained to segment and detect the coordinates of the fingertips of the index finger and thumb of the user's hand, and a regressor is trained to estimate the distance between the fingertips of the index finger and thumb of the user's hand.
[0387] Different embodiments are disclosed herein. Features of certain embodiments may be combined with features of other embodiments. Therefore, certain embodiments may be a combination of features of multiple embodiments. The foregoing description of the embodiments of the present invention is presented for the purpose of illustration and description. It is not intended to be exhaustive or to limit the present invention to the precise form disclosed. It will be appreciated by those skilled in the art that, in accordance with the above teachings, many modifications, variations, substitutions, changes and equivalents are possible. Therefore, it will be appreciated that the appended claims are intended to cover all such modifications and changes that fall within the true spirit of the present invention.
[0388] Although certain features of the present invention have been illustrated and described herein, those skilled in the art will now recognize many modifications, substitutions, changes and equivalents. Therefore, it should be understood that the appended claims are intended to cover all such modifications and changes that fall within the true spirit of the invention.
Claims
1. A device for estimating a force applied to an object, the device comprising: one or more biopotential sensors configured to be placed on a limb of a user and to measure one or more biopotential signals from one or more muscles or one or more nerves of the limb; an inertial measurement unit (IMU) for measuring one or more inertial properties of the limb, and The processor is configured to: obtaining one or more measured biopotential signals and one or more measured inertial properties, and The force exerted by the limb of the user on the object is estimated based on a model constructed from samples of biopotential signals and inertial properties collected from multiple users exerting known forces.
2. The device according to claim 1, wherein: The one or more inertial properties are selected from the group of inertial characteristics consisting of: orientation, angular velocity, linear velocity and acceleration of the limb.
3. The device according to claim 1 or claim 2, wherein: The processor is configured to estimate a weight of the object.
4. The device according to any one of claims 1 to 3, wherein: The processor is configured to estimate a torque applied to the lever.
5. The device according to any one of claims 1 to 4, wherein: The processor is configured to estimate a force applied to successfully connect the connector.
6. A gesture-controlled interface device, comprising: One or more gesture sensors for sensing a user's gestures; as well as The processor is configured to: obtaining one or more sensed gestures from the one or more gesture sensors, analyzing the one or more sensed gestures to identify input from the user, and An output signal is generated corresponding to the input to the controlled device, the output signal being modulated by a modulation function according to one or more properties of the one or more sensed gestures.
7. The device according to claim 6, wherein: The modulation function simulates physical parameters related to the virtual object or the virtual environment.
8. The device according to claim 7, wherein: The physical parameter is selected from the group consisting of: weight, texture, friction and resistance.
9. The device according to any one of claims 6 to 8, wherein: The device includes a feedback device selected from the group consisting of a screen, a haptic actuator, a tactile actuator, and a speaker.
10. The device according to any one of claims 6 to 9, wherein: The one or more properties are selected from the group consisting of: position of the hand, pressure between fingers, and distance between fingers.
11. A gesture and voice controlled interface device, comprising: One or more gesture sensors for sensing a user's gestures; One or more audio sensors for sensing sounds emitted by the user; as well as A processor is configured to obtain one or more sensed gestures from the one or more gesture sensors and one or more sensed sounds from the one or more audio sensors, analyze the sensed gestures and the sensed sounds to identify input from the user, and generate an output signal corresponding to the input to the controlled device.
12. The device according to claim 11, wherein The processor is configured to recognize the input only when the one or more sensed gestures and the one or more sensed sounds are sensed within a predetermined time period.
13. The device according to claim 11 or claim 12, wherein: The processor is configured to identify an initiation prompt in the one or more sensed gestures or in the one or more sensed sounds, extract a first measured gesture value from the one or more sensed gestures, extract a second measured gesture value from the one or more sensed gestures, and generate the output signal based on the first measured gesture value and the second measured gesture value.
14. The device according to claim 13, wherein: The initiation prompt is used to select a function of the controlled device, wherein the first measurement gesture value is used to select an initial state of a parameter of the function, and wherein the second measurement gesture value is used to select a final state of the parameter of the function.
15. The device according to claim 14, wherein: The function is selected from a function group consisting of: volume of an audio device, temperature of an environment control device, size of a virtual object, displacement of a virtual object, and zooming in or out.
16. The device according to any one of claims 13 to 15, wherein: The processor is configured to identify the initiation prompt based on at least one prompt from a group of prompts consisting of: a voice of the user, a gesture of the user, a value of force measured from the user, and a signal generated by the processor.
17. The device according to any one of claims 11 to 16, wherein: The one or more gesture sensors for sensing a user's gestures are configured to sense one or more parameters from a parameter group consisting of: displacement of the user, displacement of the user's limbs, displacement of the user's fingers, facial expressions, facial grimaces, values of biopotentials from nerves or muscles of the user's limbs or fingers, and eye movements of the user.
18. The device according to any one of claims 11 to 17, wherein: The processor is configured to calculate a similarity measure between a gesture of the one or more sensed gestures and a predefined gesture and to generate the output signal only if the similarity measure is above a predetermined threshold.
19. The device according to any one of claims 11 to 18, wherein: The processor is configured to: identifying the particular person based on comparing the one or more sensed sounds to a voice signature of the particular person; verifying that the one or more sensed gestures were performed by the particular person based on comparing the one or more sensed gestures to a gesture signature of the particular person; and The output signal corresponding to the input to the controlled device is generated only when it is verified that the one or more sensed gestures are performed by the specific person.
