Method and system for identifying manipulation intent of user when operating using touch surface controller

Abstract

The human interface device includes: a touch receiver having a plurality of touch surfaces located in distinct planes; an elastic strain sensing element having a base and a plurality of legs extending from the base and contacting an associated touch surface; a first linear strain gauge mounted on each leg for detecting strain in a first direction and a second strain gauge mounted on each leg for detecting strain in a second direction different from the first direction; and a processor configured to receive the signal from the strain gauge and determine a command operation expected by a user of the human interface device.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application is a continuation-in-part of U.S. patent application No. 17 / 993,424 filed on November 23, 2022, the contents of which are incorporated herein by reference in their entirety. Technical Field

[0003] The present disclosure relates to methods and systems for using and filtering multiple strain sensor signals from a touch surface controller to identify the user's intent in operating the controller. Background Art

[0004] U.S. patent application No. 17 / 993,424, filed on November 23, 2022, describes a human-machine interface touch surface controller comprising: a contoured user interface having opposing front and rear touch surfaces, opposing left and right side surfaces, and a top surface; a strain plate having strain surfaces, each strain surface corresponding to one of the front touch surface, the rear touch surface, the left touch surface, or the right touch surface, such that pressure applied to the touch surface causes strain on the corresponding strain surface; a strain gauge associated with each strain surface; and an electrical connection between each strain gauge and a microcontroller. The application suggests that various user operations (manipulations) on the touch surface controller, such as applying torque about mutually perpendicular axes or applying pressure on opposite side touch surfaces, will generate a set of strain gauge signals that can be easily associated with specified vehicle operations. While this is generally the case, there is still a need to accurately distinguish between intentional user operations / manipulations on the user interface and unintentional or incoherent operations / manipulations of the interface. Summary of the invention

[0005] A human-machine interface device is described that is capable of accurately determining a user's intended command based on the user's operation (manipulation) of the device. The device includes: a touch receiver having a plurality of touch surfaces to which a user can apply a force or torque representing an intended user command; and an elastic strain sensing element having a base and a plurality of legs extending from the base, wherein each leg is in contact with or attached to an associated overlying touch surface, such that pressure applied to the touch surface causes strain in the associated leg. A first linear strain gauge is mounted on each leg to detect strain in a first direction, and a second linear strain gauge is mounted on each leg to detect strain in a second direction. A microprocessor is electrically connected to each of the strain gauges to receive an associated signal. The signal is processed to accurately associate the user operation (manipulation) with the intended command. The human-machine interface device provides a variety of user interaction functions for vehicle control applications in various situations. Therefore, it is named a universal interactive sensing device and abbreviated as a UIS device in this application.

[0006] Also disclosed is a processor that runs an algorithm that utilizes multiple sensor signal processing and machine learning techniques to identify a user's intended actions (manipulations) on a UIS device by inferring sensor signal patterns using an optimized trained neural network. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] Figure 1A It is a block diagram showing how user operations (manipulations) on a human interface device (UIS device) are transmitted and processed to determine an expected user command associated with the user's operations (manipulations) on the device.

[0008] Figure 1B is a block diagram showing additional details of the microprocessor.

[0009] Figure 2 is attached to Figure 3A and Figure 3B An enlarged plan view of a strain sensor plate on a leg of an elastic strain sensing element is shown.

[0010] Figure 3A It is a three-dimensional assembly diagram of a touch receiver and a strain sensing element having four legs.

[0011] Figure 3B is a perspective view of a UIS device having a touch receiver positioned over and in contact with a strain sensing element.

[0012] Figure 4 Is associated with a specific user manipulation of a UIS device Figure 3A and Figure 3B Vector representation of the signal pattern of the strain sensing element.

[0013] Figure 5 and Figure 6 Signal patterns associated with a clockwise yaw maneuver and a counter-clockwise yaw maneuver on a UIS device are shown, respectively.

[0014] Figure 7 and Figure 8 Signal patterns associated with a pitch-up maneuver and a pitch-down maneuver on a UIS device are shown, respectively.

[0015] Fig. 9 and Fig.10 Signal patterns associated with left and right scrolling manipulations on a UIS device are shown, respectively.

[0016] Fig.11 is a block diagram that conceptualizes the process of identifying user manipulation intentions.

[0017] Fig.12is a block diagram illustrating a system for recognizing a user's manipulation intention.

[0018] Fig.13 is the original signal pattern including expected user input, unexpected user input and other noise.

[0019] Fig.14 Is from Fig.13 A processed or mixed signal pattern of an original signal pattern from which unintended user input and other noise have been filtered out.

[0020] Fig.15 is a schematic diagram of a neural network applied to mixed signal pattern input to accurately identify the user's intent.

[0021] Fig.16 is a block diagram illustrating the process of neural network training and deployment to create machine learning-based software to accurately determine user intent. DETAILED DESCRIPTION

[0022] Figure 3A (partially disassembled) and Figure 3B A human-machine interface (UIS) device 10 is shown. Device 10 includes a touch receiver 12 having multiple touch surfaces, including a front touch surface 14, a rear touch surface 16, a left touch surface 18, a right touch surface 20, and a bottom touch surface 22. Device 10 also includes an elastic strain sensing element 24 having a base 26 and a plurality of legs 28, 29, 30, and 31 (a first leg, a second leg, a third leg, and a fourth leg, respectively). Each leg is in contact with or attached to an associated overlying touch surface of touch receiver 12, such that pressure and / or torque applied to the touch surface induces strain in the associated leg.

[0023] like Figure 2 As best shown, each leg (e.g., leg 28) has a first linear strain gauge 32 mounted thereon for detecting strain in a first direction (e.g., vertical direction) and a second strain gauge 34 mounted on the leg for detecting strain in a second direction (e.g., horizontal direction). When a user operates (manipulates) the device 10, the strain gauges 32, 34 transmit signals indicative of the vertical and horizontal strains applied to each leg. The processor 40 receives the strain signals from the strain gauges via the flexible circuit 42. The processor 40 is configured to process the strain signals to determine an intended user command associated with the user operation (manipulation) of the UIS device 10.

[0024] UIS device 10 may be in the form of a variety of shapes that look like a plate, a mouse, a joystick, a joystick, a handle, and / or other shapes that may allow a user to manipulate (steer) the device by hand.

[0025] User operations (manipulations) are listed, such as pushing, pulling, turning left, turning right, light pinching, strong pinching, etc., but are not limited to the user manipulation examples in Table 1. The operation (manipulation) of the UIS device 10 can generate forces and torques, which are applied to the UIS touch receiver 12 and transmitted to the strain sensing element 24. The strain sensing element 24 generates strain changes, which are simultaneously detected by the strain sensors 32 and 34 on the legs 28, 29, 30 and 31, respectively. The microprocessor 40 obtains the output of the strain sensor signals, processes these signals through the UIS strain signal processing software, and generates a strain signal pattern associated with the user operation (manipulation). The strain signal pattern is represented by a data set for each operation (manipulation) and is collected by the microprocessor 40 to establish a data source for the strain signal pattern of various user operations (manipulations) shown in Table 1.

[0026] According to the signal pattern data source and the annotated user manipulation, a neural network (UIS neural network) is trained. The trained and optimized UIS neural network is deployed in a microprocessor as a software component to input strain signal pattern data and output the identified user operation (manipulation) command intention through reasoning calculation of the UIS neural network.

[0027] Figure 2 The strain sensor board shown is a two-dimensional strain sensing component that outputs strain measurements of two vertical strains through strain sensors mounted vertically on a sensor circuit board. The strain sensor 32 is mounted to detect strain in the vertical direction. The strain sensor 34 is mounted to detect strain in the horizontal direction. The sensor board on which the two sensors are mounted is a PCB or flexible circuit board or other type of circuit base that is connected to or contacts the strain sensing element 24. The strain sensor board outputs signals Sv(t) and Sh(t) from the strain sensors 32 and 34, where t represents time, Sv(t) is the vertical strain, and Sh(t) is the horizontal strain.

[0028] The strain sensing element 24 may be a Figure 3A and Figure 3B ) will produce strain changes on the surface of its legs when the user manipulates the UIS device with force and torque applied to the UIS device, the strain sensing element 24 will detect the user manipulation input force and torque through the strain changes on the surface of its legs. The number of legs of the strain sensing element 24 can be one, two, three, four or more, depending on the configuration of the UIS device. Each leg is connected to a sensor plate segment ( Figure 2) connection, where x is the number of legs (1, 2, 3, 4 or more)

[0029] The leg signal vector is created based on the dual outputs Sv(t) and Sh(t) of the corresponding sensor board. For a given leg x:

[0030] Vector magnitude VM(x,t) = SQRT(Sv(x,t)^2+Sh(x,t)^2)

[0031] Vector direction reference relative to coordinate VD-angle (x, t) = ATAN (Sv (x, t) / Sh (x, t)) where SQRT is the square root function and ATAN is the inverse tangent function.

[0032] Figure 4 Four leg signal vectors thrown toward the target at a specific time are shown.

[0033] Leg 1VM=7,VD-angle=22.5°

[0034] Leg 2VM = 4, VD-angle = 315°

[0035] Leg 3VM = 9, VD-angle = 190°

[0036] Leg 4VM = 5, VD-angle = 247.5°

[0037] The signal pattern of the strain sensing element is a map where all leg vectors are calculated and expressed in magnitude and direction. Figure 4 In the map, all four leg vectors are represented by their magnitudes and direction angles shown above. The signal pattern is related to the user manipulation of the UIS device through the information transmission from the forces and torques generated by the manipulation to the strain changes of the strain sensing element and further to the measured vector map from the sensor board. The measured vector map is the real-time signal pattern. It is a response to the user manipulation on the UIS device.

[0038] Figure 5 and Figure 6 Simulated examples of signal patterns when a user rotates the UIS device in a clockwise yaw maneuver and a counterclockwise yaw maneuver are shown. Figure 5 and Figure 6 The north leg, west leg, south leg and east leg shown in Figure 4 Equivalent alternative names for legs 1, 2, 3, and 4 as defined in .

[0039] Figure 7 and Figure 8 Shown are simulated examples of signal patterns when a user pitches the UIS device up and down.

[0040] Fig. 9 and Fig.10 A simulated example of a signal pattern when a user rolls the UIS device left or right is shown. In the real world, when a user intends to manipulate with a specific action, such as tilting up on the UIS device, the tilt-up signal pattern will appear along with other noise vectors. This is because when a user manipulates with tilt-up, the user typically presses on the UIS device while applying some other forces and torques to the UIS device, causing a push-down and / or other type of signal pattern to appear along with the intended tilt-up manipulation. The real-world signal pattern will deviate from Figures 5 to 10 Theoretical pure signal pattern simulated in . Deviations are caused by unexpected forces and torques when the user manipulates for a specific intended operation.

[0041] Table 1 User manipulation intention - example

[0042]

[0043]

[0044] The user manipulation intention recognition system is mainly a UIS neural network reasoning system with a variable signal conditioning function and an intensity filtering function block, which creates a mixed signal pattern for UIS neural network reasoning. Fig.12 The system is demonstrated.

[0045] In the real world, when a user intends to manipulate with a specific action, such as tilting up on a UIS device, the tilt-up signal pattern will appear, but other noise vectors will also appear together. This is because when the user manipulates the tilt-up, he usually presses on the UIS device and / or applies some other forces and torques to the device, causing downward pressing and other types of signal patterns to appear along with the intended tilt-up action. Fig.13 Manipulated signal patterns containing expected inputs, unexpected inputs, and other noise are shown. Fig.14 The signal pattern of manipulation is shown to be intensity filtered by a cutoff signal with an amplitude less than a configurable threshold. The filtered signal pattern provides a clean mixed signal pattern input to the UIS neural network to identify user manipulation intentions.

[0046] Fig.15A UIS neural network for processing mixed signal pattern input to output the identified user manipulation intention is shown. The UIS neural network can be implemented as multiple layers, each layer having multiple nodes. Recurrent neural networks (RNNs) and other types of neural network techniques can be applied to effectively process signal patterns so as to output user manipulation intentions with an optimized minimum number of network nodes. The UIS neural network is trained based on a data source from user manipulation signal patterns. For each manipulation shown in Table 1 but not limited thereto, the relevant signal patterns are collected, recorded and marked. The data source is generated by different types of users, so the trained neural network is robust in identifying user manipulation intentions.

[0047] Fig.16 The process of UIS neural network training and deployment is shown. The data source block is the process of designing and organizing the data generation of user manipulations. The data recording block is the process of collecting data using tools and methods. The data labeling block is the process of automatically and manually labeling the recorded data to the associated manipulations based on tools and methods. The training block trains the UIS neural network using training tools based on the UIS signal pattern data source. The evaluation and optimization block is to test and verify the UIS neural network, as well as adjust to reduce the number of nodes of the neural network to achieve the best results. The deployment block deploys the UIS neural network model for Fig.12 The reasoning of the user manipulation intention recognition system shown.

[0048] The method and system for identifying user manipulation intention is based on a UIS neural network. The data source for training the network is generated based on the signal pattern of the user manipulation. The signal pattern is obtained by the UIS signal processing software, which filters multiple strain sensor signals and generates a recognizable user manipulation signal pattern. Once the UIS neural network is trained and optimized, it is deployed as a software component in the system's microprocessor and identifies the user manipulation intention by reasoning about the UIS signal pattern generated in real time by the strain signal.

[0049] The above description is intended to be illustrative, not limiting. The scope of the present invention should be determined by reference to the appended claims and the full scope of equivalents. It is expected and anticipated that future developments will occur in the art, and the disclosed devices, kits and methods will be incorporated into such future embodiments. Therefore, the present invention is capable of modification and variation and is limited only by the appended claims.

Claims

1. A human-machine interface device, comprising: a touch receiver having a plurality of touch surfaces, each touch surface being arranged on a plane distinct from the other touch surfaces; a resilient strain sensing element having a base and a plurality of legs extending from the base, each leg in contact with or attached to an associated overlying touch surface such that pressure applied to the touch surface induces strain in the associated leg; a first linear strain gauge mounted on each leg for detecting strain in a first direction, and a second linear strain gauge mounted on each leg for detecting strain in a second direction different from the first direction; as well as A processor is configured to receive signals from each of the strain gauges and determine a command intended by a user operation (manipulation) of the human-machine interface device.

2. The device according to claim 1, wherein: The first direction is substantially perpendicular to the second direction.

3. The device according to claim 1, wherein: Each of the touch surfaces is arranged in a vertical direction.

4. The device according to claim 1, wherein: The touch surfaces are arranged in substantially parallel pairs.

5. The device according to claim 1, wherein: The touch surface includes two pairs of substantially parallel touch surfaces.

6. The device according to claim 1, wherein: The first linear strain gauge of each leg is mounted to detect strain in a generally vertical direction, and the second linear strain gauge of each leg is mounted to detect strain in a generally horizontal direction.

7. The device according to claim 6, wherein: The microprocessor is configured to calculate a vector magnitude of the strain on each leg, the vector magnitude being equal to the square root of the sum of the square of the vertical strain signal strength and the square of the horizontal strain signal strength.

8. The device according to claim 6, wherein: The microprocessor is configured to calculate a vector direction of the strain on each leg, the vector direction being equal to the arc tangent of the quotient of the vertical strain signal strength divided by the horizontal strain signal strength.

9. The device according to claim 7, wherein: The calculated vector magnitudes and vector directions on each leg are grouped into strain signal patterns represented by the data set.

10. The device according to claim 9, wherein: The data set is collected by the aforementioned microprocessor to create a data source of strain signal patterns of various user operations (manipulations) of the human-machine interface device.

11. The device according to claim 10, wherein: The data source is used to train a neural network.

12. The device according to claim 11, wherein The trained neural network is used to identify the expected user commands for the operation (manipulation) of the human-machine interface device, and the identified user command intention is the inference output of the trained neural network to the input of the strain signal pattern generated by the user operation (manipulation).

13. The device according to claim 12, wherein: User operations (manipulations) are listed in Table 1 but are not limited to Table 1.

14. The apparatus according to claim 8, wherein: The calculated vector magnitudes and vector directions on each leg are grouped into strain signal patterns represented by data sets.

Images