A dual-modal human-computer interaction method, system, device, and medium based on touch micro-vibration modulation

Abstract

This invention discloses a dual-modal human-computer interaction method, system, device, and medium based on touch micro-vibration modulation, belonging to the field of human-computer interaction technology. The method includes: acquiring the user's baseline micro-vibration signal; extracting the user's baseline features; collecting measured touch signals during normal user interaction; identifying the user's touch commands; and extracting measured micro-vibration features, including measured basic parameters and micro-vibration patterns; identifying several candidate micro-vibration modulation commands based on the mapping relationship between the micro-vibration patterns and candidate micro-vibration modulation commands, as well as the mapping relationship between the measured basic parameters, the deviation of the user's baseline features, and the candidate micro-vibration modulation commands, and performing multiple screenings to obtain the optimal micro-vibration modulation command; and fusing the user's touch commands and the optimal micro-vibration modulation command to obtain the final interaction command. This invention can introduce the conscious micro-vibration features of touch into human-computer interaction, effectively improving the efficiency of human-computer interaction, and is suitable for ordinary users and users with hand movement disorders.

Description

Technical Field

[0001] This invention belongs to the field of human-computer interaction technology, specifically relating to a dual-modal human-computer interaction method, system, device, and medium based on touch micro-vibration modulation. Background Technology

[0002] Touchscreens have become the primary input method for modern smart terminals, widely used in smartphones, tablets, smartwatches, industrial control panels, and other devices. During touch operation, users' fingers produce micro-tremors, medically termed physiological tremors (hereinafter referred to as micro-tremors). These tremors typically have a frequency range of 4-12Hz and an amplitude generally between 0.1-0.5mm. Physiological tremors are an inherent characteristic of the human neuromuscular system, generated by the rhythmic activity of the central nervous system and peripheral reflex mechanisms. Studies have shown that even at rest, the fingers of healthy adults exhibit weak rhythmic vibrations. During fine touch operations, these micro-tremors are superimposed on the intended movement, forming a high-frequency oscillating component in the touch trajectory.

[0003] Existing technologies share a highly consistent approach to handling touch tremors: treating tremors as noise or interference that needs to be eliminated. Specific technical approaches include: (1) Filtering elimination technology: filtering out high-frequency tremor components and retaining low-frequency intentional motion trajectories through low-pass filtering, adaptive filtering, and other methods. This method is simple and direct, but it will lose some effective information. (2) Machine learning denoising technology: using deep learning and other methods to learn tremor features and achieve intelligent denoising. This method requires a large amount of training data and the model's generalization ability is limited. (3) Predictive compensation technology: estimating and compensating for touch deviations caused by tremors based on a predictive model of user intention motion. This method requires the establishment of an accurate motion model and has high computational complexity.

[0004] However, existing technologies have the following shortcomings:

[0005] Current touch interaction systems essentially eliminate microtremors, ignoring their inherent application value. Most users (including those with pathological tremors) have some control over their microtremor characteristics and can consciously adjust their frequency, amplitude, and pattern. Eliminating microtremors wastes valuable information. Furthermore, relying solely on finger movement for human-computer interaction results in relatively low efficiency, especially for users with hand movement disorders. Summary of the Invention

[0006] This invention addresses the shortcomings of existing technologies by providing a dual-modal human-computer interaction method, system, device, and medium based on touch micro-vibration modulation. This method can introduce the conscious micro-vibration characteristics of touch into human-computer interaction, effectively improving the efficiency of human-computer interaction. It is suitable for both ordinary users and users with hand movement disorders.

[0007] This invention provides the following technical solution:

[0008] Firstly, a dual-modal human-computer interaction method based on touch micro-vibration modulation is provided, including:

[0009] Acquire the reference micro-vibration signal when the user performs a free swiping operation on the touch interface, and extract the user reference features of the reference micro-vibration signal;

[0010] Collect measured touch signals when the user performs normal interactive operations on the touch interface, and identify the user's touch commands;

[0011] After separating the micro-vibration modulation signal from the measured touch signal, the measured micro-vibration characteristics of the micro-vibration modulation signal are extracted, including the measured basic parameters and micro-vibration mode;

[0012] The mapping relationship between micro-vibration mode and candidate micro-vibration modulation command is constructed, as well as the mapping relationship between measured basic parameters and user reference feature deviation and candidate micro-vibration modulation command. The candidate micro-vibration modulation command corresponding to the current micro-vibration mode and the candidate micro-vibration modulation command corresponding to the current measured basic parameters and user reference feature deviation are obtained. The optimal micro-vibration modulation command is obtained by multiple screening of all candidate micro-vibration modulation commands.

[0013] Based on preset fusion rules, the user's touch commands and the optimal micro-vibration modulation commands are fused to obtain the final interaction commands.

[0014] Optionally, the user reference features include: reference micro-flicker frequency and reference micro-flicker amplitude; the measured basic parameters include: measured micro-flicker dominant frequency and measured micro-flicker amplitude.

[0015] Optionally, the extraction of measured micro-vibration features from the micro-vibration modulation signal specifically includes:

[0016] Using a sliding window, the FFT method is employed to perform time-frequency analysis on the micro-flicker modulated signal, yielding the measured micro-flicker dominant frequency. The specific formula is as follows:

[0017] ;

[0018] in, For a moment The measured dominant frequency of the micro-vibration modulation signal. For frequency variables, To perform the Fast Fourier Transform operation, This indicates taking the square modulo 1. For a moment The amplitude of the micro-vibration modulated signal, For integration time variable, The length of the sliding window;

[0019] Calculate the measured micro-flicker amplitude of the micro-flicker modulated signal: ;in, For a moment Measured amplitude of the micro-flicker modulated signal;

[0020] Based on user baseline characteristics, the microtremor activation threshold is constructed using the following formula. ;

[0021] ; ;

[0022] in, For the user's baseline micro-vibration amplitude, The index for the stability of microfrequencies. The standard deviation of the reference micro-vibration signal frequency sequence, The reference micro-vibration frequency for the user;

[0023] Based on microtremor activation threshold The micro-vibration state of the micro-vibration modulation signal is defined according to the following formula;

[0024] ;

[0025] in, For a moment The micro-vibration state of the micro-vibration modulation signal is 0 or 1, where 1 represents active and 0 represents silent.

[0026] The micro-vibration modulation signal within the sliding window is divided into Each micro-vibration segment has the same micro-vibration state to construct the micro-vibration pattern of the micro-vibration modulation signal. : ;in, For the first The micro-vibration state of a micro-vibration segment For the first The duration of each micro-vibration segment.

[0027] Optionally, obtaining the candidate micro-vibration modulation command corresponding to the current micro-vibration mode and the candidate micro-vibration modulation command corresponding to the deviation between the current measured basic parameters and the user reference feature specifically involves:

[0028] Calculate the frequency offset of the measured micro-vibration dominant frequency relative to the reference micro-vibration frequency. Based on frequency offset The mapping relationship with candidate micro-flicker modulation commands is used to obtain the frequency offset. The corresponding first candidate micro-vibration modulation command;

[0029] Calculate the rate of change of the measured microfibrillation amplitude relative to the baseline microfibrillation amplitude. Based on the rate of change of amplitude The mapping relationship with candidate microfibrillation modulation commands is used to obtain the amplitude change rate. The corresponding second candidate micro-vibration modulation command;

[0030] Based on the micro-vibration mode of the micro-vibration modulation signal and the mapping relationship between the micro-vibration mode and the candidate micro-vibration modulation command, the third candidate micro-vibration modulation command corresponding to the micro-vibration mode is obtained.

[0031] Optionally, the step of obtaining the optimal micro-vibration modulation command by performing multiple screenings on all candidate micro-vibration modulation commands specifically involves:

[0032] All candidate microfiber modulation commands are subjected to four validity verifications, including: feature duration verification, feature change amplitude verification, confidence threshold verification, and user intent verification.

[0033] When performing confidence threshold verification, the overall confidence level of each candidate micro-vibration modulation command is calculated. : ;in, The confidence level weighting coefficient is... The feature matching degree of candidate microscream modulation commands. To ensure context consistency, the relevant information is obtained based on the correlation between the current application scenario and the current candidate micro-vibration modulation command.

[0034] The feature matching degree of the first candidate micro-vibration modulation command is calculated according to the following formula. ; ;in, The threshold value for the set frequency offset;

[0035] The feature matching degree of the second candidate micro-vibration modulation command is calculated according to the following formula. : ;in, The threshold value for the set rate of change of amplitude;

[0036] Feature matching degree of the third candidate micro-vibration modulation command Obtained through assignment;

[0037] The number of candidate micro-flicker modulation commands that pass the four-fold validity verification is counted: if there are 0, no command is output as the best micro-flicker modulation command; if there is only 1, the candidate micro-flicker modulation command is selected as the best micro-flicker modulation command; if there are multiple, three rounds of priority screening are performed. The first round of priority screening is based on the highest feature matching degree of the candidate micro-flicker modulation command; the second round of priority screening is based on the highest context consistency of the candidate micro-flicker modulation command; and the third round of priority screening is based on the highest historical operation correlation of the candidate micro-flicker modulation command.

[0038] The three-round priority screening is as follows: screening is performed round by round according to the preset priority. After each round, if there are only 1 or 0 candidate micro-vibration modulation instructions left, the corresponding best micro-vibration modulation instruction is output; otherwise, the next round of screening is entered. After the three rounds of screening are completed, if there are only 1 candidate micro-vibration modulation instruction left, the candidate micro-vibration modulation instruction is output; otherwise, no instruction is output as the best micro-vibration modulation instruction.

[0039] Optionally, it also includes: separating a reference micro-vibration signal from the reference touch signal when the user performs a free swiping operation on the touch interface, and performing the following processing on the reference touch signal or the measured touch signal when separating the reference micro-vibration signal or the micro-vibration modulation signal:

[0040] The displacement sequence of the reference touch signal or the measured touch signal is calculated according to the following formula, and the reference micro-vibration signal or micro-vibration modulation signal is obtained after high-pass filtering of the displacement sequence:

[0041] ; ; ;

[0042] in, , These are the reference touch signal or the measured touch signal, respectively. Each sampling point is direction and Displacement components in the direction; For the first Displacement amplitude at each sampling point.

[0043] Optionally, it also includes: determining whether the time elapsed since the acquisition of the reference micro-vibration signal exceeds a set time period; if it does, automatically updating the reference micro-vibration signal and performing a weighted summation of the user reference features corresponding to the updated reference micro-vibration signal and the user reference features before the update to obtain the updated user reference features; otherwise, maintaining the current user reference features.

[0044] Secondly, a dual-modal human-computer interaction system based on touch micro-vibration modulation is provided, comprising:

[0045] The baseline feature acquisition module is used to acquire the baseline micro-vibration signal when the user performs a free swiping operation on the touch interface, and extract the user baseline features of the baseline micro-vibration signal.

[0046] The touch command recognition module is used to collect measured touch signals when the user performs normal interactive operations on the touch interface and to recognize the user's touch commands;

[0047] The measured feature acquisition module is used to extract the measured micro-vibration features of the micro-vibration modulation signal after separating it from the measured touch signal, including the measured basic parameters and micro-vibration mode.

[0048] The micro-vibration modulation command acquisition module is used to construct the mapping relationship between micro-vibration mode and candidate micro-vibration modulation command, as well as the mapping relationship between measured basic parameters and user reference feature deviation and candidate micro-vibration modulation command, to obtain the candidate micro-vibration modulation command corresponding to the current micro-vibration mode and the candidate micro-vibration modulation command corresponding to the current measured basic parameters and user reference feature deviation. By performing multiple screenings on all candidate micro-vibration modulation commands, the optimal micro-vibration modulation command is obtained.

[0049] The dual-instruction fusion module is used to fuse the user's touch commands and the optimal micro-vibration modulation commands based on preset fusion rules to obtain the final interaction commands.

[0050] Thirdly, a computer device is provided, including a processor and a memory; wherein, when the processor executes a computer program stored in the memory, it implements the steps of the bimodal human-computer interaction method based on touch micro-vibration modulation as described in any one of the first aspects.

[0051] Fourthly, a computer-readable storage medium is provided for storing a computer program; when the computer program is executed by a processor, it implements the steps of the bimodal human-computer interaction method based on touch micro-vibration modulation as described in any one of the first aspects.

[0052] Compared with the prior art, the beneficial effects of the present invention are:

[0053] This invention breaks through the traditional paradigm of eliminating micro-vibrations in touch technology by fundamentally shifting the mindset. It innovatively proposes a new approach to utilizing micro-vibrations, transforming physiological micro-vibration signals, previously considered interference noise, into valuable input resources. Furthermore, users can utilize micro-vibration features without relying on any additional hardware; they can perform shortcut operations on existing touchscreens simply by consciously adjusting their own micro-vibration characteristics. The learning curve is low, as users can naturally invoke their own micro-vibration control capabilities. Operational efficiency is high, requiring no complex gestures for triggering; it has a wide range of applications, seamlessly adapting to various touch devices; and it offers good privacy, as the micro-vibration modulation process is invisible and silent, making it particularly suitable for public places. Finally, this invention fuses touch commands and micro-vibration modulation commands in a dual-modal manner, expanding touch interaction from a single dimension to a two-dimensional space, significantly improving operational efficiency. It provides ordinary users with a more efficient and convenient shortcut operation method and offers a new alternative input solution for users with hand movement disorders, possessing broad application prospects and significant social value. Attached Figure Description

[0054] Figure 1 This is a flowchart of the dual-modal human-computer interaction method based on touch micro-vibration modulation of the present invention;

[0055] Figure 2 This is a process diagram of extracting measured micro-tremor features in Embodiment 3 of the present invention;

[0056] Figure 3 This is a performance comparison chart of the method of the present invention and the prior art when performing human-computer interaction in Embodiment 3 of the present invention. Detailed Implementation

[0057] The present invention will be further described below with reference to the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solutions of the present invention and should not be used to limit the scope of protection of the present invention. It should be noted that the term "comprising" and any variations thereof in the specification, claims and the above-mentioned drawings of the present invention are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to these processes, methods, products or devices.

[0058] Example 1:

[0059] like Figure 1 As shown, a dual-modal human-computer interaction method based on touch micro-vibration modulation includes:

[0060] Step S1: Obtain the reference micro-vibration signal when the user performs a free swiping operation on the touch interface, and extract the user reference features of the reference micro-vibration signal;

[0061] Step S2: Collect measured touch signals when the user performs normal interactive operations on the touch interface, and identify the user's touch commands;

[0062] Step S3: After separating the micro-vibration modulation signal from the measured touch signal, extract the measured micro-vibration features of the micro-vibration modulation signal, including the measured basic parameters and micro-vibration mode;

[0063] Step S4: Construct the mapping relationship between micro-vibration mode and candidate micro-vibration modulation command, as well as the mapping relationship between measured basic parameters and user reference feature deviation and candidate micro-vibration modulation command, to obtain the candidate micro-vibration modulation command corresponding to the current micro-vibration mode and the candidate micro-vibration modulation command corresponding to the current measured basic parameters and user reference feature deviation. By performing multiple screenings on all candidate micro-vibration modulation commands, the optimal micro-vibration modulation command is obtained.

[0064] Step S5: Based on the preset fusion rules, the user's touch commands and the optimal micro-vibration modulation commands are fused to obtain the final interaction commands.

[0065] In this embodiment, the reference micro-vibration signal in step S1 is obtained by separating it from the reference touch signal when the user performs a free swiping operation on the touch interface. The reference touch signal is collected by the sensor of the existing touch interface. The touch interface usually includes a touch screen, touchpad or other touch virtual interface, etc.

[0066] In this embodiment, the specific steps for separating the reference micro-vibration signal from the reference touch signal, or separating the micro-vibration modulation signal from the measured touch signal, are as follows: Calculate the displacement sequence of the reference touch signal or the measured touch signal according to the following formula, and obtain the reference micro-vibration signal or the micro-vibration modulation signal after high-pass filtering the displacement sequence:

[0067] ; ; ;

[0068] in, , These are the reference touch signal or the measured touch signal, respectively. Each sampling point is direction and Displacement components in the direction; For the first The displacement amplitude at each sampling point. Typically, a first-order Butterworth filter is used to perform a 3Hz high-pass filter on the displacement sequence to extract the reference micro-vibration signal or the measured touch signal.

[0069] User reference features include: reference micro-flicker frequency and reference micro-flicker amplitude. Step S1 extracts the user reference features of the reference micro-flicker signal, specifically including: averaging the frequency sequence and amplitude sequence of the reference micro-flicker signal based on the number of sampling points to obtain the reference micro-flicker frequency and reference micro-flicker amplitude. Specifically, the frequency sequence of the reference micro-flicker signal is obtained. and amplitude sequence And calculate the reference micro-flicker frequency and reference micro-flicker amplitude according to the following formula;

[0070] ; ;

[0071] in, The reference micro-vibration frequency is expressed in Hz. The reference micro-flicker amplitude is expressed in mm. The number of sampling points for the reference micro-vibration signal is typically 7200 (corresponding to 30 seconds, with a sampling frequency of 240Hz). The reference micro-vibration signal is the first The frequency of micro-flicker at each sampling moment, in Hz; The reference micro-vibration signal is the first The amplitude of the micro-tremor at each sampling time, in mm.

[0072] In this embodiment, step S2 is based on the measured touch signal. To recognize user touch commands, existing technologies can be referenced, such as a combination of rule matching and geometric constraints, machine learning algorithms (e.g., random forests, hidden Markov models, etc.), or deep learning networks (long short-term memory networks, gated recurrent units, etc.).

[0073] In this embodiment, the measured micro-vibration features in step S3 include measured basic parameters and micro-vibration modes. The measured basic parameters include the measured micro-vibration dominant frequency and the measured micro-vibration amplitude. Step S3 extracts the measured micro-vibration features of the micro-vibration modulated signal, specifically including:

[0074] S3.1: Using a sliding window, the FFT method is employed to perform time-frequency analysis on the micro-vibration modulated signal, obtaining the measured micro-vibration dominant frequency of the micro-vibration modulated signal. The specific formula is as follows:

[0075] ;

[0076] in, For a moment The measured dominant frequency of the micro-vibration modulation signal. For frequency variables, To perform the Fast Fourier Transform (FFT) operation, the FFT method references existing techniques. This indicates taking the square modulo 1. For a moment The amplitude of the micro-vibration modulated signal, For integration time variable, This is the length of the sliding window, typically 500ms.

[0077] S3.2: Calculate the measured micro-flicker amplitude of the micro-flicker modulated signal according to the following formula:

[0078] ;

[0079] in, For a moment The measured amplitude of the micro-vibration modulation signal;

[0080] S3.3: Based on user baseline characteristics, construct the microtremor activation threshold using the following formula.

[0081] ; ;

[0082] in, The micro-flicker activation threshold is used to determine the micro-flicker state of the micro-flicker modulated signal, and its unit is mm. For the user's baseline micro-vibration amplitude, The microfibrillation frequency stability index, ranging from , The standard deviation of the reference micro-vibration signal frequency sequence, For the user's reference micro-vibration frequency, To maximize the function, ensure that the micro-vibration frequency stability exponent is non-negative.

[0083] S3.4: Based on microterror activation threshold The micro-vibration state of the micro-vibration modulation signal is defined according to the following formula.

[0084] ;

[0085] in, For a moment The micro-vibration state of the micro-vibration modulation signal is 0 or 1, where 1 represents active and 0 represents silent.

[0086] S3.5: Divide the micro-vibration modulation signal within the sliding window into... Each micro-vibration segment has the same micro-vibration state to construct the micro-vibration pattern of the micro-vibration modulation signal. :

[0087] :

[0088] in, For the first The micro-vibration state of a micro-vibration segment For the first The duration of each micro-vibration segment.

[0089] Step S4 specifically includes:

[0090] S4.1: Construct the mapping relationship between micro-vibration modes and candidate micro-vibration modulation commands, as well as the mapping relationship between measured basic parameters and user baseline characteristic deviations and candidate micro-vibration modulation commands.

[0091] Mapping relationship between measured basic parameters and user reference characteristic deviations and candidate micro-vibration modulation commands: including frequency offset Mapping relationship with candidate microfibrillation modulation commands, amplitude change rate Mapping relationship with candidate microscream modulation commands.

[0092] S4.2: Obtain the frequency offset The corresponding first candidate micro-vibration modulation command.

[0093] Calculate the frequency offset of the measured micro-vibration dominant frequency relative to the reference micro-vibration frequency. , Based on frequency offset The mapping relationship with candidate micro-flicker modulation commands is used to obtain the frequency offset. The corresponding first candidate micro-vibration modulation command.

[0094] As shown in Table 1, the frequency offset is given. An example of the mapping relationship with candidate microscream modulation instructions.

[0095] Table 1: Frequency Offset Mapping table with candidate microfibrillation modulation commands

[0096]

[0097] As shown in Table 1, when the frequency offset exceeds the set threshold... At that time, the first candidate micro-vibration modulation command is: fast operation; when the frequency offset is less than the set threshold. At that time, the first candidate micro-screwing modulation command is: slow operation; the frequency offset is at... If the value is within the specified range, it is determined that there is no clear operational intention, and the first candidate micro-vibration modulation instruction is: maintain the current operation.

[0098] S4.3: Obtain the rate of change of amplitude The corresponding second candidate micro-vibration modulation command.

[0099] Calculate the rate of change of the measured microfibrillation amplitude relative to the baseline microfibrillation amplitude. : Based on the rate of change of amplitude The mapping relationship with candidate microfibrillation modulation commands is used to obtain the amplitude change rate. The corresponding second candidate micro-vibration modulation command

[0100] Table 2 shows the rate of change of amplitude. An example of the mapping relationship with candidate microscream modulation instructions.

[0101] Table 2: Rate of Change in Amplitude Mapping table with candidate microfibrillation modulation commands

[0102]

[0103] As shown in Table 2, when the user amplitude change rate Exceeding the set threshold When the second candidate micro-fission modulation command is confirmed, the amplitude change rate is... Less than the set threshold At that time, the second candidate micro-flicker modulation command is a cancellation command; amplitude change rate exist If the value is within the specified range, it is determined that there is no clear operational intent, and the second candidate micro-vibration modulation command is a hold command.

[0104] S4.4: Obtain the third candidate microvibration modulation instruction corresponding to the microvibration mode.

[0105] Based on the preset mode judgment conditions in the mapping table between micro-vibration modes and candidate micro-vibration modulation commands, determine the third candidate micro-vibration modulation command corresponding to the current micro-vibration mode.

[0106] Table 3 provides an example of the mapping relationship between micro-vibration modes and candidate micro-vibration modulation commands.

[0107] Table 3: Mapping Relationship between Micro-vibration Modes and Candidate Micro-vibration Modulation Commands

[0108]

[0109] Table 3 For a moment micro-vibration mode The number of micro-vibration fragments, , and for The duration of the first, second, and third microfibrillation segments, For a preset short time threshold, such as 200ms, The preset time threshold, for example, 500ms. The preset continuous activation threshold, for example, 1000ms. The preset pause threshold is, for example, 800ms. As shown in Table 3, the "short-long-short" mode indicates that the user's finger produces a tension-relaxation-tension rhythm, corresponding to the undo command; the "continuous activation" mode indicates that the micro-vibration is active for more than 1 second, corresponding to the right-click menu command; the "silent pause" mode indicates that the micro-vibration signal disappears for more than 800ms, corresponding to the long press command.

[0110] S4.5: Obtain the optimal micro-vibration modulation command.

[0111] Step S4.5 specifically includes the following steps:

[0112] S4.5.1: Perform four-fold validity verification on all candidate micro-vibration modulation commands, including: feature duration verification, feature change amplitude verification, confidence threshold verification, and user intent verification.

[0113] The pass condition for feature duration verification is: the duration of the measured micro-vibration feature corresponding to the candidate micro-vibration modulation command exceeds the corresponding threshold, for example, more than 200ms.

[0114] The pass condition for feature change amplitude verification is: the change amplitude of the measured micro-vibration feature corresponding to the candidate micro-vibration modulation command exceeds the corresponding threshold, such as frequency greater than ±2Hz and amplitude ±30%.

[0115] The pass condition for confidence threshold verification is that the overall confidence of the candidate micro-vibration modulation command is greater than the set threshold, for example, greater than 0.7.

[0116] The conditions for passing user intent verification are: the user's touch command is consistent with the current candidate micro-vibration modulation command. For example, if the user's touch command is to view an image and the current candidate micro-vibration modulation command is one of the first candidate micro-vibration modulation commands, then the user intent verification passes; otherwise, the user intent verification fails.

[0117] When performing confidence threshold verification, the overall confidence level of each candidate micro-vibration modulation command is calculated. :

[0118] ;

[0119] in, This is the confidence level weighting coefficient, typically 0.7. The feature matching degree of the candidate micro-vibration modulation command is specifically set to the feature matching degree of the first candidate micro-vibration modulation command. Feature matching degree of the second candidate micro-vibration modulation command Feature matching degree with the third candidate micro-vibration modulation command , For context consistency, the relevant information is obtained based on the correlation between the current application scenario and the current candidate micro-vibration modulation command. For example, in an image viewing scenario, the relevant information is obtained for CMD_ZOOM_IN (fast operation). In text input scenarios, CMD_ZOOM_IN (fast operation) .

[0120] Specifically, the feature matching degree of the first candidate micro-vibration modulation command is calculated according to the following formula. ;

[0121] ;

[0122] in, The threshold for the set frequency offset, usually 2Hz.

[0123] The feature matching degree of the second candidate micro-vibration modulation command is calculated according to the following formula. :

[0124] ;

[0125] in, The threshold for the rate of change of amplitude is set, specifically 30%.

[0126] Feature matching degree of the third candidate micro-vibration modulation command The feature matching degree of the third candidate micro-vibration modulation instruction is obtained by assignment, and a specific example is given in Table 3. When the third candidate micro-vibration modulation instruction is CMD_UNDO (cancel instruction), the feature matching degree of the third candidate micro-vibration modulation instruction is obtained. The feature matching degree of the third candidate micro-vibration modulation command is 0.9 when the third candidate micro-vibration modulation command is CMD_RIGHT_CLICK (right-click menu command). The feature matching degree of the third candidate micro-vibration modulation command is 0.85 when the third candidate micro-vibration modulation command is CMD_LONG_PRESS (long press command). It is 0.8.

[0127] S4.5.2: Count the number of candidate micro-flicker modulation commands that pass the four-fold validity verification:

[0128] If there are 0, then no instruction is output as the optimal micro-scream modulation instruction.

[0129] If there is only one, then the candidate microscream modulation command is taken as the best microscream modulation command.

[0130] If there are multiple candidates, a three-round priority selection process is conducted. The first round of priority selection is based on the highest feature matching degree of the candidate micro-vibration modulation instruction. The second round of priority selection is based on the highest context consistency of the candidate micro-vibration modulation instruction. The third round of priority selection is based on the highest historical operation correlation of the candidate micro-vibration modulation instruction.

[0131] The highest correlation between candidate micro-vibration modulation commands and historical operations is found in the user's last interaction command within the last 5 seconds and the best micro-vibration modulation command (the value ranges from 0 to 1, and the higher the logical fit, the higher the score).

[0132] Three rounds of priority screening are performed, specifically: Screening is conducted round by round according to priority. After each round, if only 1 or 0 candidate micro-trigger modulation commands remain, the corresponding optimal micro-trigger modulation command is output (if only 0 candidate micro-trigger modulation commands remain, output "No command"; if only 1 candidate micro-trigger modulation command remains, output the remaining candidate micro-trigger modulation command). Otherwise, the next round of screening begins. After three rounds of screening, if only 1 candidate micro-trigger modulation command remains, output that candidate micro-trigger modulation command; otherwise, output "No command" as the optimal micro-trigger modulation command. In this embodiment, a 300ms time window voting mechanism is typically used for dejittering, requiring the command to appear at least twice within the window; otherwise, it is considered a false trigger, and "No command" is output as the optimal micro-trigger modulation command. The dejittering method is based on existing technologies.

[0133] In this embodiment, an example of a fusion rule for the user's touch command and the optimal micro-vibration modulation command mentioned in step S5 is given, as shown in Table 4.

[0134] Table 4: Dual Instruction Fusion Rule Table

[0135]

[0136] Example 2:

[0137] The difference between Example 2 and Example 1 is that the user reference features are obtained in different ways. Specifically, it is determined whether the time elapsed since the acquisition of the reference micro-vibration signal exceeds a set time period. If it does, the reference micro-vibration signal is automatically updated, and the user reference features corresponding to the updated reference micro-vibration signal and the user reference features before the update are weighted and summed to obtain the updated user reference features. Otherwise, the current user reference features are maintained.

[0138] The method for obtaining the user reference features corresponding to the updated reference micro-vibration signal is as follows: (Based on the number of sampling points of the updated reference micro-vibration signal, the frequency sequence and amplitude sequence of the updated reference micro-vibration signal are averaged to obtain the reference micro-vibration frequency and reference micro-vibration amplitude corresponding to the updated reference micro-vibration signal).

[0139] Example 3:

[0140] A specific example of applying the bimodal human-computer interaction method of the present invention to perform human-computer interaction is given.

[0141] I. Hardware environment.

[0142] Device: Smartphone with capacitive touchscreen.

[0143] Touch sampling frequency: 240Hz (increased sampling rate to capture micro-vibration details).

[0144] Coordinate accuracy: 0.01mm.

[0145] Processor: Octa-core CPU, 2.5GHz.

[0146] Memory: 6GB RAM.

[0147] Haptic feedback: linear vibration motor.

[0148] II. Software Environment.

[0149] Operating system: Android 13.

[0150] Programming language: Kotlin.

[0151] Signal preprocessing library: custom implementation (FFT, filtering).

[0152] III. Specific operating steps.

[0153] Step 1: Extract user baseline features.

[0154] The user performs a 30-second free swipe operation on the phone screen at a frequency of 240Hz (corresponding to the number of sampling points of the reference micro-vibration signal). The system acquires a reference touch signal (7200), separates the reference micro-vibration signal, and extracts the user's reference micro-vibration frequency. and baseline micro-vibration amplitude .

[0155] ; ; .

[0156] Step 2: Extract the measured micro-tremor features.

[0157] FFT analysis was performed using a 500ms sliding window to extract the measured micro-flicker dominant frequency within the frequency range of 4-12Hz and calculate the measured micro-flicker amplitude. The micro-flicker activation threshold was calculated based on user baseline characteristics. : Based on the micro-vibration activation threshold, a micro-vibration state sequence of the micro-vibration modulated signal is defined, and the micro-vibration mode of the micro-vibration modulated signal is constructed. For example... Figure 2 As shown, Figure 2 (a) in the figure represents the measured touch signal acquired in real time. Figure 2 (b) in the figure represents the micro-vibration modulation signal separated from the measured touch signal. Figure 2 (c) in the figure is the time-frequency spectrum of the micro-vibration modulated signal for time-frequency analysis. Figure 2 (d) in the diagram shows the process of extracting the measured micro-vibration dominant frequency. Figure 2 (e) in the figure is a tracking graph of the measured micro-flicker dominant frequency and the measured micro-flicker amplitude. Through... Figure 2 It can capture the dynamic changes of the measured micro-vibration main frequency and the measured micro-vibration amplitude.

[0158] Step 3: Obtain the user's touch commands.

[0159] To illustrate that different touch commands correspond to different final human-computer interaction commands, this embodiment provides three different user touch commands: image viewing, swiping browsing, and text input.

[0160] Step 4: Obtain the optimal micro-vibration modulation command and fuse the two commands.

[0161] The measured micro-vibration main frequency, measured micro-vibration amplitude, and micro-vibration mode of the micro-vibration modulated signal are calculated, and the optimal micro-vibration modulation command is obtained through four-fold verification and three-priority screening.

[0162] For example, when a user's touch command is to view an image, the user will consciously and slightly tighten their finger muscles to increase the amplitude of micro-vibration (similar to slightly pressing the screen without actually pressing it). The measured amplitude of this micro-vibration... Calculate the rate of change of amplitude. According to Table 2, we know that we will get CMD_CONFIRM (confirmation command). After verification and screening, we find that this command is the best micro-vibration modulation command at the current moment. According to Table 4, we combine the best micro-vibration modulation command (confirmation command) and the touch command (image viewing). The final output interaction command is: zoom in to 150%.

[0163] When the user's touch command is to swipe and browse, the user will consciously and slightly increase the rhythm of finger shaking to increase the micro-tremor frequency (similar to slightly trembling the fingers). At this time, the measured main frequency of micro-tremor is... Calculate the frequency offset According to Table 1, we will obtain CMD_ZOOM_IN (quick operation command). After verification and filtering, this command is determined to be the optimal micro-vibration modulation command for the current moment. According to Table 4, the optimal micro-vibration modulation command (quick operation command) and the touch command (swipe browsing) are combined, and the final output interaction command is: 3x fast scrolling.

[0164] When a user's touch command is text input, the user consciously controls the tension-relaxation-tension rhythm of their fingers to produce a "short-long-short" micro-vibration pattern. Combined with this micro-vibration pattern... According to Table 3, we know that CMD_UNDO (undo instruction) will be obtained. After verification and filtering, this instruction is the best micro-vibration modulation instruction at the current moment. According to Table 4, the best micro-vibration modulation instruction (undo instruction) and the touch instruction (text input) are combined, and the final output interactive instruction is: Undo the previous input.

[0165] Figure 3 (a) in the figure is a comparison chart of the accuracy of instruction recognition when using the method of the present invention and the prior art for human-computer interaction; Figure 3 (b) in the figure is a comparison chart of the false trigger rate when using the method of the present invention and the prior art for human-computer interaction; Figure 3 (c) in the figure is a comparison chart of response delay when using the method of the present invention and the prior art for human-computer interaction; Figure 3 (d) in the figure is a radar comparison chart of the comprehensive performance when using the method of the present invention and the prior art for human-computer interaction; according to Figure 3As can be seen, the instruction recognition accuracy of the method of the present invention is 92.3%, an improvement of 13.8 percentage points; the false trigger rate is 3.1%, a decrease of 9.2 percentage points; and the response latency is 45ms, a decrease of 75ms. This shows that the present invention performs well in all evaluation dimensions and verifies the superiority of the present invention in utilizing the micro-vibration concept compared with the existing technology.

[0166] Example 4:

[0167] A dual-modal human-computer interaction system based on touch micro-vibration modulation includes:

[0168] The baseline feature acquisition module is used to acquire the baseline micro-vibration signal when the user performs a free swiping operation on the touch interface, and extract the user baseline features of the baseline micro-vibration signal.

[0169] The touch command recognition module is used to collect measured touch signals when the user performs normal interactive operations on the touch interface and to recognize the user's touch commands;

[0170] The measured feature acquisition module is used to extract the measured micro-vibration features of the micro-vibration modulation signal after separating it from the measured touch signal, including the measured basic parameters and micro-vibration mode.

[0171] The micro-vibration modulation command acquisition module is used to construct the mapping relationship between micro-vibration mode and candidate micro-vibration modulation command, as well as the mapping relationship between measured basic parameters and user reference feature deviation and candidate micro-vibration modulation command, to obtain the candidate micro-vibration modulation command corresponding to the current micro-vibration mode and the candidate micro-vibration modulation command corresponding to the current measured basic parameters and user reference feature deviation. By performing multiple screenings on all candidate micro-vibration modulation commands, the optimal micro-vibration modulation command is obtained.

[0172] The dual-instruction fusion module is used to fuse the user's touch commands and the optimal micro-vibration modulation commands based on preset fusion rules to obtain the final interaction commands.

[0173] For more specific details about the above system, please refer to the relevant content disclosed in the foregoing embodiments, which will not be repeated here.

[0174] Example 5:

[0175] The present invention provides a computer device, including a processor and a memory; wherein, when the processor executes a computer program stored in the memory, it implements the steps of the above-described dual-modal human-computer interaction method based on touch micro-vibration modulation.

[0176] For more detailed information on the above methods, please refer to the relevant content disclosed in the foregoing embodiments, which will not be repeated here.

[0177] Example 6:

[0178] The present invention provides a computer-readable storage medium for storing a computer program; when the computer program is executed by a processor, it implements the steps of the above-described dual-modal human-computer interaction method based on touch micro-vibration modulation.

[0179] For more detailed information on the above methods, please refer to the relevant content disclosed in the foregoing embodiments, which will not be repeated here.

[0180] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. The systems, devices, and storage media disclosed in the embodiments are described simply because they correspond to the methods disclosed in the embodiments; relevant details can be found in the method section.

[0181] Those skilled in the art will clearly understand that the techniques in the embodiments of the present invention can be implemented using software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solutions in the embodiments of the present invention, or the parts that contribute to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in various embodiments or certain parts of the embodiments of the present invention.

[0182] The above are merely preferred embodiments of the present invention. The scope of protection of the present invention is not limited to the above embodiments. All technical solutions falling within the scope of the present invention's concept are within the scope of protection of the present invention. It should be noted that for those skilled in the art, any improvements and modifications made without departing from the principles of the present invention should be considered within the scope of protection of the present invention.

Claims

1. A dual-modal human-computer interaction method based on touch micro-vibration modulation, characterized in that, include: Acquire the reference micro-vibration signal when the user performs a free swiping operation on the touch interface, and extract the user reference features of the reference micro-vibration signal; Collect measured touch signals when the user performs normal interactive operations on the touch interface, and identify the user's touch commands; After separating the micro-vibration modulation signal from the measured touch signal, the measured micro-vibration characteristics of the micro-vibration modulation signal are extracted, including the measured basic parameters and micro-vibration mode; The mapping relationship between micro-vibration mode and candidate micro-vibration modulation command is constructed, as well as the mapping relationship between measured basic parameters and user reference feature deviation and candidate micro-vibration modulation command. The candidate micro-vibration modulation command corresponding to the current micro-vibration mode and the candidate micro-vibration modulation command corresponding to the current measured basic parameters and user reference feature deviation are obtained. The optimal micro-vibration modulation command is obtained by multiple screening of all candidate micro-vibration modulation commands. Based on preset fusion rules, the user's touch commands and the optimal micro-vibration modulation commands are fused to obtain the final interaction commands; The user baseline characteristics include: baseline micro-flicker frequency and baseline micro-flicker amplitude; the measured basic parameters include: measured micro-flicker dominant frequency and measured micro-flicker amplitude. The extracted measured micro-vibration features of the micro-vibration modulation signal specifically include: Using a sliding window, the FFT method is employed to perform time-frequency analysis on the micro-flicker modulated signal, yielding the measured micro-flicker dominant frequency. The specific formula is as follows: ； in, For a moment The measured dominant frequency of the micro-vibration modulation signal. For frequency variables, To perform the Fast Fourier Transform operation, This indicates taking the square modulo 1. For a moment The amplitude of the micro-vibration modulated signal, For integration time variable, The length of the sliding window; Calculate the measured micro-flicker amplitude of the micro-flicker modulated signal: ;in, For a moment Measured amplitude of the micro-flicker modulated signal; Based on user baseline characteristics, the microtremor activation threshold is constructed using the following formula. ; ； ； in, For the user's baseline micro-vibration amplitude, The index for the stability of microfrequencies. The standard deviation of the reference micro-vibration signal frequency sequence, The reference micro-vibration frequency for the user; Based on microtremor activation threshold The micro-vibration state of the micro-vibration modulation signal is defined according to the following formula; ； in, For a moment The micro-vibration state of the micro-vibration modulation signal is 0 or 1, where 1 represents active and 0 represents silent. The micro-vibration modulation signal within the sliding window is divided into Each micro-vibration segment has the same micro-vibration state to construct the micro-vibration pattern of the micro-vibration modulation signal. : ;in, For the first The micro-vibration state of a micro-vibration segment For the first The duration of a micro-vibration segment; The process of obtaining candidate micro-fiber modulation commands corresponding to the current micro-fiber mode and candidate micro-fiber modulation commands corresponding to the deviations between the current measured basic parameters and user reference characteristics specifically involves: Calculate the frequency offset of the measured micro-vibration dominant frequency relative to the reference micro-vibration frequency. Based on frequency offset The mapping relationship with candidate micro-flicker modulation commands is used to obtain the frequency offset. The corresponding first candidate micro-vibration modulation command; Calculate the rate of change of the measured microfibrillation amplitude relative to the baseline microfibrillation amplitude. Based on the rate of change of amplitude The mapping relationship with candidate microfibrillation modulation commands is used to obtain the amplitude change rate. The corresponding second candidate micro-vibration modulation command; Based on the micro-vibration mode of the micro-vibration modulation signal and the mapping relationship between the micro-vibration mode and the candidate micro-vibration modulation command, the third candidate micro-vibration modulation command corresponding to the micro-vibration mode is obtained.

2. The dual-modal human-computer interaction method based on touch micro-vibration modulation according to claim 1, characterized in that, The process of obtaining the optimal micro-vibration modulation command by performing multiple screenings on all candidate micro-vibration modulation commands specifically involves: All candidate microfiber modulation commands are subjected to four validity verifications, including: feature duration verification, feature change amplitude verification, confidence threshold verification, and user intent verification. When performing confidence threshold verification, the overall confidence level of each candidate micro-vibration modulation command is calculated. : ;in, The confidence weighting coefficient is... The feature matching degree of the candidate microscream modulation command. To ensure context consistency, the relevant information is obtained based on the correlation between the current application scenario and the current candidate micro-vibration modulation command. The feature matching degree of the first candidate micro-vibration modulation command is calculated according to the following formula. ; ;in, The threshold value for the set frequency offset; The feature matching degree of the second candidate micro-vibration modulation command is calculated according to the following formula. : ;in, The threshold value for the set rate of change of amplitude; Feature matching degree of the third candidate micro-vibration modulation command Obtained through assignment; The number of candidate micro-flicker modulation commands that pass the four-fold validity verification is counted: if there are 0, no command is output as the best micro-flicker modulation command; if there is only 1, the candidate micro-flicker modulation command is selected as the best micro-flicker modulation command; if there are multiple, three rounds of priority screening are performed. The first round of priority screening is based on the candidate micro-flicker modulation command with the highest feature matching degree; the second round of priority screening is based on the candidate micro-flicker modulation command with the highest context consistency; and the third round of priority screening is based on the candidate micro-flicker modulation command with the highest historical operation correlation. The three-round priority screening is as follows: screening is performed round by round according to the preset priority. After each round, if there are only 1 or 0 candidate micro-vibration modulation instructions left, the corresponding best micro-vibration modulation instruction is output; otherwise, the next round of screening is entered. After the three rounds of screening are completed, if there are only 1 candidate micro-vibration modulation instruction left, the candidate micro-vibration modulation instruction is output; otherwise, no instruction is output as the best micro-vibration modulation instruction.

3. The dual-modal human-computer interaction method based on touch micro-vibration modulation according to claim 1, characterized in that, Also includes: The reference micro-vibration signal is separated from the reference touch signal when the user performs a free swiping operation on the touch interface. When separating the reference micro-vibration signal or the micro-vibration modulation signal, the reference touch signal or the measured touch signal is processed as follows: The displacement sequence of the reference touch signal or the measured touch signal is calculated according to the following formula, and the reference micro-vibration signal or micro-vibration modulation signal is obtained after high-pass filtering of the displacement sequence: ； ； ； in, , These are the reference touch signal or the measured touch signal, respectively. Each sampling point is at direction and Displacement components in the direction; For the first Displacement amplitude at each sampling point.

4. The dual-modal human-computer interaction method based on touch micro-vibration modulation according to claim 1, characterized in that, Also includes: If the time elapsed since the acquisition of the reference micro-vibration signal exceeds a set time period, the reference micro-vibration signal is automatically updated. The user reference features corresponding to the updated reference micro-vibration signal and the user reference features before the update are weighted and summed to obtain the updated user reference features. Otherwise, the current user reference features are maintained.

5. A dual-modal human-computer interaction system based on touch micro-vibration modulation, comprising the steps of the dual-modal human-computer interaction method based on touch micro-vibration modulation as described in any one of claims 1-4, characterized in that, include: The baseline feature acquisition module is used to acquire the baseline micro-vibration signal when the user performs a free swiping operation on the touch interface, and extract the user baseline features of the baseline micro-vibration signal. The touch command recognition module is used to collect measured touch signals when the user performs normal interactive operations on the touch interface and to recognize the user's touch commands; The measured feature acquisition module is used to extract the measured micro-vibration features of the micro-vibration modulation signal after separating it from the measured touch signal, including the measured basic parameters and micro-vibration mode. The micro-vibration modulation command acquisition module is used to construct the mapping relationship between micro-vibration mode and candidate micro-vibration modulation command, as well as the mapping relationship between measured basic parameters and user reference feature deviation and candidate micro-vibration modulation command, to obtain the candidate micro-vibration modulation command corresponding to the current micro-vibration mode and the candidate micro-vibration modulation command corresponding to the current measured basic parameters and user reference feature deviation. By performing multiple screenings on all candidate micro-vibration modulation commands, the optimal micro-vibration modulation command is obtained. The dual-instruction fusion module is used to fuse the user's touch commands and the optimal micro-vibration modulation commands based on preset fusion rules to obtain the final interaction commands.

6. A computer device, characterized in that, It includes a processor and a memory; wherein, when the processor executes the computer program stored in the memory, it implements the steps of the dual-modal human-computer interaction method based on touch micro-vibration modulation as described in any one of claims 1-4.

7. A computer-readable storage medium, characterized in that, Used to store computer programs; when the computer programs are executed by the processor, they implement the steps of the dual-modal human-computer interaction method based on touch micro-vibration modulation as described in any one of claims 1-4.

Images