Adaptive human-machine interaction curved controller and system based on multi-modal perception

Abstract

The application relates to the technical field of human-computer interaction, in particular to a self-adaptive human-computer interaction curved surface controller based on multi-modal perception and a system thereof, which first acquires electromyographic signal data and pressure characteristic data of a UAV operator; secondly, a state correction coefficient is generated according to the electromyographic signal, a signal threshold triggered by action is dynamically corrected, a first gating judgment is carried out, and a physiological intention score is calculated; subsequently, an expected pressure intensity value that should be generated under the intention is predicted based on the electromyographic signal, a second gating judgment is carried out by comparing the deviation of a real-time pressure value and an expected value, and a physical action score is calculated; finally, a total evaluation value is calculated by combining the physiological intention and the physical action score, when a preset condition is met, the curved surface controller is driven to adaptively adjust its morphological parameters; through double verification, the risk of misoperation is reduced, and the accuracy and safety of human-computer interaction in a complex environment are improved.

Description

Technical Field

[0001] This invention relates to the field of human-computer interaction technology, specifically to an adaptive human-computer interaction surface controller and system based on multimodal perception. Background Technology

[0002] With the widespread application of drones in industrial inspection, emergency rescue, and operations in complex environments, human-machine interface control devices have gradually evolved from traditional buttons and joysticks to touch-sensitive curved surfaces. These curved surface controllers sense the operator's touch, pressing, or sliding actions and map them into equipment control commands, offering advantages such as intuitive operation and high integration.

[0003] However, most existing surface controllers rely solely on a single physical contact signal to determine whether to trigger an operation command. When operators are under stress, fatigue, or in complex environmental interference conditions, they are prone to misoperation due to accidental touches, sliding deviations, or unintentional pressing, which can lead to the accidental triggering of high-risk commands. Summary of the Invention

[0004] To address the problems in related technologies, this invention provides an adaptive human-computer interaction surface controller and system based on multimodal perception, thereby overcoming the aforementioned technical problems existing in the prior art.

[0005] To solve the above-mentioned technical problems, the present invention provides the following technical solution: an adaptive human-computer interaction surface control system based on multimodal perception, comprising:

[0006] The data acquisition module is used to acquire electromyographic signal data and stress characteristic data of the drone operator;

[0007] The real-time motion determination module is used to determine the real-time motion type corresponding to the current operation based on the electromyographic signal data and pressure feature data combined with a preset standard motion feature database, and to obtain the signal threshold and deviation threshold corresponding to the real-time motion type.

[0008] The electromyography (EMG) signal analysis module is used to determine a state correction coefficient representing the operator's fatigue level based on the EMG signal data, correct the signal threshold based on the state correction coefficient, and determine whether the amplitude of the EMG signal data is greater than the corrected signal threshold. If so, the first gating state is output as passed, and the physiological intention score is calculated.

[0009] The pressure feature analysis module is used to predict the expected pressure intensity value, which characterizes the physical pressure response generated by the physiological intention under the real-time action type, based on the electromyographic signal data and the physiological intention score when the first gate state is passed, and to calculate the real-time pressure intensity value based on the pressure feature data. It then determines whether the deviation between the real-time pressure intensity value and the expected pressure intensity value is less than the deviation threshold. If so, it outputs that the second gate state is passed and calculates the physical action score.

[0010] The curved surface control module is used to calculate the total evaluation value based on the physiological intention score and physical action score combined with the real-time action type when the second gate state is passed. Only when the total evaluation value meets the preset conditions, the curved surface controller adjusts the current form according to the real-time action type.

[0011] Preferably, determining the real-time action type corresponding to the current operation and obtaining the signal threshold and deviation threshold corresponding to the real-time action type includes the following steps:

[0012] The electromyography signal data and pressure feature data are cascaded to generate the real-time operation fingerprint of the current drone operator.

[0013] The similarity between each standard action feature data in the preset standard action feature database and the real-time operation fingerprint is calculated sequentially to obtain a similarity matrix;

[0014] Determine whether the maximum value of each similarity in the similarity matrix is ​​greater than a preset similarity threshold;

[0015] If so, it means that the current drone operator's operation has successfully matched the operation type corresponding to the standard action feature data with the similarity, and the real-time action type is obtained;

[0016] The minimum electromyographic amplitude and pressure deviation threshold corresponding to the standard action feature data corresponding to the similarity are labeled to obtain the signal threshold and deviation threshold corresponding to the real-time action type.

[0017] If not, calculate the topological completeness between the pressure distribution feature data in the pressure feature data and the standard pressure distribution feature in the standard action feature data corresponding to the highest similarity in the similarity matrix;

[0018] If the topological integrity is greater than the preset integrity threshold, then confidence compensation is performed on the electromyographic signal data based on the pressure feature data. A similarity matrix is ​​regenerated based on the compensated electromyographic signal data and the pressure feature data. If the maximum value of each similarity in the regenerated similarity matrix is ​​still less than or equal to the preset similarity threshold, it indicates that the action performed by the current UAV does not match the action type corresponding to each standard action feature data in the standard action feature database. An attitude lock signal is output to the surface controller to control the UAV to perform automatic hovering and wait for confirmation from the UAV operator.

[0019] If the topology completeness is less than or equal to the preset completeness threshold, it means that the action performed by the current UAV does not match the action type corresponding to each standard action feature data in the standard action feature database. The attitude lock signal is output to the surface controller to control the UAV to perform automatic hovering and wait for confirmation from the UAV operator.

[0020] The confidence compensation of electromyographic signal data based on pressure characteristic data includes the following steps:

[0021] The signal gain coefficient is calculated based on the topology integrity; the calculation formula is as follows:

[0022] ,

[0023] in, Represents the signal gain coefficient. Represents the sensitivity coefficient. Indicates topological completeness. This indicates the preset integrity threshold;

[0024] The electromyographic signal data is subjected to confidence compensation based on the signal gain coefficient to obtain compensated electromyographic signal data; the compensation formula is as follows:

[0025] ,

[0026] in, and These represent the electromyographic signal data before and after confidence compensation, respectively. Indicates the regulating factor. This represents the muscle activation vector in the standard action feature data corresponding to the highest similarity in the similarity matrix.

[0027] By cascading electromyographic signal data and pressure feature data to generate a real-time operation fingerprint, and performing similarity matching with a standard action feature database to determine the action type and corresponding threshold, a differentiated recognition strategy is achieved. When the matching fails, confidence compensation is performed through topological integrity analysis. If a match still cannot be found, a posture lock signal is output to control the device to automatically hover, effectively avoiding false triggering caused by abnormal operation or unexpected actions, and ensuring safety is prioritized when facing unknown operations.

[0028] Preferably, correcting the signal threshold according to the state correction coefficient includes the following steps:

[0029] The electromyographic signal data is subjected to time-frequency domain transformation to obtain frequency domain feature data;

[0030] The average power frequency of the electromyographic signal data is calculated based on the frequency domain feature data, and a state correction coefficient characterizing the operator's fatigue level is calculated based on the average power frequency; the calculation formula is as follows:

[0031] ,

[0032] in, Represents the state correction factor. This represents the average power frequency of the muscle group activation vector in the standard motion feature data corresponding to the real-time motion type. This represents the gain coefficient used to control the sensitivity to changes in fatigue. Indicates the bias number;

[0033] The signal threshold is corrected according to the state correction coefficient to obtain the corrected signal threshold; the correction formula is as follows:

[0034] ,

[0035] in, and These represent the signal thresholds before and after the correction, respectively. This represents the maximum correction factor.

[0036] Preferably, the calculation of the physiological intent score includes the following steps:

[0037] Determine whether the amplitude of the electromyographic signal data is greater than the corrected signal threshold;

[0038] If so, the first gating state is output as passed, and the first difference between the amplitude of the electromyographic signal data and the corrected signal threshold is calculated;

[0039] Calculate the second difference between the amplitude of the muscle group activation vector in the standard action feature data corresponding to the real-time action type and the corrected signal threshold;

[0040] Calculate the ratio of the first difference to the second difference to obtain the physiological intention score;

[0041] If not, the first gating state is output as cut off, the physiological intention score is set to zero, the surface controller is controlled to maintain the current form lock, and the drone is controlled to perform a preset no-signal standby operation.

[0042] By determining the state correction coefficient for the operator's fatigue level based on electromyography (EMG) signal data and dynamically correcting the signal threshold, the problem of intention recognition failure caused by the decrease in EMG signal amplitude under fatigue state is solved. When the corrected EMG signal amplitude is greater than the correction threshold, the first gating state is output as passed and the physiological intention score is calculated, which ensures the effective capture of the real operation intention under fatigue state and realizes stable recognition under different physiological states.

[0043] Preferably, the predicted pressure intensity value, which characterizes the physical pressure response generated by the physiological intention under a real-time action type, includes the following steps:

[0044] Acquire historical data corresponding to the real-time action types in several sets of tasks performed by drone operators to obtain a historical dataset; the historical data includes at least historical electromyography signal data, historical physiological intention scores, and historical standard pressure intensity values.

[0045] A final pressure intensity prediction model is constructed based on the historical dataset.

[0046] When the first gating state is passed, the electromyographic signal data and physiological intention score are input into the final pressure intensity prediction model to predict the standard pressure intensity value corresponding to the real-time action type performed by the drone operator, so as to obtain the expected pressure intensity value characterizing the physical pressure response generated by the physiological intention under the real-time action type.

[0047] Preferably, the calculation of the physical action score includes the following steps:

[0048] The real-time pressure intensity value is calculated based on the pressure characteristic data.

[0049] Determine whether the deviation between the real-time pressure intensity value and the expected pressure intensity value is less than the deviation threshold. If so, output the second gating state as passed, and calculate the physical action score based on the standard pressure distribution characteristics in the standard action characteristic data corresponding to the real-time action type. The calculation formula is as follows:

[0050] ,

[0051] in, Indicates the score for physical actions. and These represent the real-time pressure intensity value and the area of ​​the effective touch region extracted from the pressure distribution feature data in the pressure feature data, respectively. and These represent the standard pressure intensity value and standard touch area extracted from the standard pressure distribution features in the standard action feature data corresponding to the real-time action type, respectively. and These represent the pressure characteristic trade-off coefficient and the area characteristic trade-off coefficient, respectively.

[0052] If not, the second gating state is output as cut off, the physical action score is set to zero, the surface controller is controlled to maintain the current form lock, and the drone is controlled to perform a preset no-signal standby operation.

[0053] When the first gating state is passed, the second gating state is output by predicting the standard pressure characteristics corresponding to the physiological intention based on electromyographic signal data and judging whether the deviation between the actual pressure characteristics and the standard pressure characteristics is less than the threshold. This realizes closed-loop verification from physiological intention to physical execution. It effectively identifies and intercepts deviations in physical actions caused by external interference, ensuring that only operations with a high degree of consistency between physiological intention and physical action can trigger the instruction, thus reducing the risk of false triggering.

[0054] Preferably, the surface controller adjusts the current shape according to the real-time action type, including the following steps:

[0055] When the second gating state is passed, the total evaluation value is calculated based on the physiological intention score and physical action score combined with the real-time action type; the calculation formula is as follows:

[0056] ,

[0057] in, This represents the total evaluation value. and These represent the preference weights of the real-time action type for physiological intention and physical action, respectively.

[0058] Determine whether the total evaluation value is greater than the preset execution threshold. If not, control the surface controller to maintain the current form lock and control the drone to perform the preset no-signal standby operation.

[0059] If so, the real-time action type is matched with the preset surface controller shape adjustment instruction library using a depth-first search algorithm. The surface controller shape adjustment instruction that matches the real-time action type is searched from the preset surface controller shape adjustment instruction library and the data is labeled to obtain the real-time surface controller shape adjustment instruction.

[0060] The surface controller shape adjustment command includes at least surface curvature adjustment parameters, damping coefficient adjustment parameters, and surface stiffness adjustment parameters;

[0061] Based on the total evaluation value, the adjustment parameters in the real-time surface controller shape adjustment command are adaptively adjusted to obtain the adjusted real-time surface controller shape adjustment command.

[0062] The adjusted real-time surface controller shape adjustment command is pushed to the main control unit through the wireless communication network. The main control unit executes the surface control task according to the adjusted real-time surface controller shape adjustment command and adjusts the shape of the current surface controller.

[0063] By employing the above technical solution, the present invention provides an adaptive human-computer interaction surface control system based on multimodal perception, which has at least the following beneficial effects:

[0064] 1. This invention constructs a dual-gated verification mechanism using electromyographic and pressure signals, combining physiological intent recognition with physical action verification to output an operation validity judgment result. It also generates an adaptive judgment standard by combining standard thresholds from an action type feature database, enabling the controller to intelligently adjust based on the operator's state and action type. This effectively avoids the false triggering problem caused by traditional single physical contact signal judgment methods, making operation intent recognition and validity judgment more closely match the operator's actual operating characteristics under real working conditions, thus improving the operational accuracy and safety reliability of the curved surface controller in high-risk application scenarios.

[0065] 2. This invention generates a real-time operation fingerprint by cascading electromyographic signal data and pressure feature data, and determines the action type and corresponding threshold by similarity matching with a standard action feature database, thus realizing a differentiated recognition strategy. When the matching fails, confidence compensation is performed through topological integrity analysis. If the matching still fails, a posture lock signal is output to control the device to automatically hover, effectively avoiding the problem of false triggering caused by abnormal operation or unexpected action, and ensuring safety first when facing unknown operations.

[0066] 3. This invention solves the problem of intention recognition failure caused by the decrease in electromyographic signal amplitude under fatigue by determining the state correction coefficient of operator fatigue level based on electromyographic signal data and dynamically correcting the signal threshold. When the corrected electromyographic signal amplitude is greater than the correction threshold, the first gating state is output as passed and the physiological intention score is calculated, which ensures the effective capture of the real operation intention under fatigue and realizes stable recognition under different physiological states.

[0067] 4. When the first gating state is passed, the present invention predicts the standard pressure characteristics corresponding to the physiological intention based on electromyographic signal data, and determines whether the deviation between the actual pressure characteristics and the standard pressure characteristics is less than a threshold to output the second gating state, thereby realizing closed-loop verification from physiological intention to physical execution; effectively identifying and intercepting physical action deviations caused by external interference, ensuring that only operations with a high degree of consistency between physiological intention and physical action can trigger the instruction, reducing the risk of false triggering. Attached Figure Description

[0068] The above and other objects, features, and advantages of this application will become more apparent from the more detailed description of the embodiments of this application in conjunction with the accompanying drawings. The drawings are provided to further illustrate the embodiments of this application and form part of the specification. They are used together with the embodiments of this application to explain the application and do not constitute a limitation thereof. In the drawings, the same reference numerals generally represent the same components or steps.

[0069] Figure 1 This is a schematic diagram of the module of the adaptive human-computer interaction surface control system provided by the present invention. Detailed Implementation

[0070] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0071] Exemplary System

[0072] Existing technologies rely solely on a single physical contact signal to determine whether an operation command is triggered. This leads to errors such as accidental touches, sliding deviations, or unintentional presses when the operator is under stress, fatigue, or in complex environmental conditions, resulting in the unintended triggering of high-risk commands. This embodiment proposes an adaptive human-computer interaction surface control system based on multimodal perception. For example... Figure 1 As shown, it includes:

[0073] The data acquisition module is used to acquire electromyographic signal data and stress characteristic data of the drone operator;

[0074] The real-time motion determination module is used to determine the motion type corresponding to the current operation based on the electromyographic signal data and pressure feature data combined with a preset standard motion feature database, and to obtain the signal threshold and pressure threshold corresponding to the motion type.

[0075] The electromyography (EMG) signal analysis module is used to determine a state correction coefficient representing the operator's fatigue level based on the EMG signal data, correct the signal threshold based on the state correction coefficient, and determine whether the amplitude of the EMG signal data is greater than the corrected signal threshold. If so, the first gating state is output as passed, and the physiological intention score is calculated.

[0076] The pressure feature analysis module is used to predict the standard pressure feature of the physical action expected to represent the physiological intention based on the electromyographic signal data when the first gate state is passed, determine whether the deviation between the pressure feature data and the standard pressure feature is less than the pressure threshold, and if so, output the second gate state as passed and calculate the physical action score.

[0077] The curved surface control module is used to calculate the total evaluation value based on the physiological intention score and physical action score when the second gate state is passed. Only when the total evaluation value meets the preset conditions, the curved surface controller adjusts the shape according to the action type corresponding to the current operation.

[0078] The acquisition of electromyographic signal data and stress characteristic data of the drone operator includes the following steps:

[0079] A continuous sliding time window is constructed. Electromyography (EMG) signals are monitored based on the sliding time window. The root mean square (RMS) value of the EMG signals within the sliding time window is calculated. Only when the RMS value of the EMG signals within the window is within a preset effective trigger interval is the EMG signal determined to be an effective action signal containing subjective control intention. The current sliding time window is then locked, and the EMG signal data within the time window and the pressure feature data synchronized with its time sequence are extracted.

[0080] The pressure characteristic data includes at least pressure intensity data and pressure distribution characteristic data.

[0081] Determining the real-time action type corresponding to the current operation and obtaining the signal threshold and deviation threshold corresponding to the real-time action type includes the following steps:

[0082] The electromyography signal data and pressure feature data are cascaded to generate the real-time operation fingerprint of the current drone operator.

[0083] The similarity between each standard action feature data in the preset standard action feature database and the real-time operation fingerprint is calculated sequentially to obtain a similarity matrix;

[0084] Determine whether the maximum value of each similarity in the similarity matrix is ​​greater than a preset similarity threshold;

[0085] If so, it means that the current drone operator's operation has successfully matched the operation type corresponding to the standard action feature data with the similarity, and the real-time action type is obtained;

[0086] The minimum electromyographic amplitude and pressure deviation threshold corresponding to the standard action feature data corresponding to the similarity are labeled to obtain the signal threshold and deviation threshold corresponding to the real-time action type.

[0087] If not, calculate the topological completeness between the pressure distribution feature data in the pressure feature data and the standard pressure distribution feature in the standard action feature data corresponding to the highest similarity in the similarity matrix;

[0088] If the topological integrity is greater than the preset integrity threshold, then confidence compensation is performed on the electromyographic signal data based on the pressure feature data. A similarity matrix is ​​regenerated based on the compensated electromyographic signal data and the pressure feature data. If the maximum value of each similarity in the regenerated similarity matrix is ​​still less than or equal to the preset similarity threshold, it indicates that the action performed by the current UAV does not match the action type corresponding to each standard action feature data in the standard action feature database. An attitude lock signal is output to the surface controller to control the UAV to perform automatic hovering and wait for confirmation from the UAV operator.

[0089] If the topology completeness is less than or equal to the preset completeness threshold, it means that the action performed by the current UAV does not match the action type corresponding to each standard action feature data in the standard action feature database. The attitude lock signal is output to the surface controller to control the UAV to perform automatic hovering and wait for confirmation from the UAV operator.

[0090] The confidence compensation of electromyographic signal data based on pressure characteristic data includes the following steps:

[0091] The signal gain coefficient is calculated based on the topology integrity; the calculation formula is as follows:

[0092] ,

[0093] in, Represents the signal gain coefficient. Represents the sensitivity coefficient. Indicates topological completeness. This indicates the preset integrity threshold;

[0094] The electromyographic signal data is subjected to confidence compensation based on the signal gain coefficient to obtain compensated electromyographic signal data; the compensation formula is as follows:

[0095] ,

[0096] in, and These represent the electromyographic signal data before and after confidence compensation, respectively. Indicates the regulating factor. This represents the muscle activation vector in the standard action feature data corresponding to the highest similarity in the similarity matrix.

[0097] The standard motion feature database contains feature data corresponding to different motion types; the motion types include at least emergency hovering, speed switching, motor locking, and attitude fine-tuning.

[0098] The feature data includes at least the normalized muscle group activation vector, standard pressure distribution features, and the minimum electromyographic amplitude and pressure deviation threshold required for the corresponding action type under standard conditions.

[0099] The pressure deviation threshold represents the maximum permissible inconsistency between physiological intention and physical action under different action types.

[0100] The sensitivity coefficient represents the system's tolerance to the types of actions performed by the drone operator and is dynamically calibrated according to the different types of actions.

[0101] The adjustment factor represents the intensity control coefficient of electromyographic signal compensation, which is used to balance the weight distribution of electromyographic signal data before and after confidence compensation.

[0102] Preferably, the adjustment factor can be dynamically set according to the difference between the topological integrity and the preset integrity threshold. When the difference between the topological integrity and the preset integrity threshold is small, it indicates that the topological integrity of the pressure distribution just meets the standard but the quality is not high, and the adjustment factor takes a small value, such as 0.3-0.5. When the difference between the topological integrity and the preset integrity threshold is large, it indicates that the topological structure of the pressure distribution is complete and the quality is good, and the adjustment factor takes a large value, such as 0.6-0.8.

[0103] By cascading electromyographic signal data and pressure feature data to generate a real-time operation fingerprint, and performing similarity matching with a standard action feature database to determine the action type and corresponding threshold, a differentiated recognition strategy is achieved. When the matching fails, confidence compensation is performed through topological integrity analysis. If a match still cannot be found, a posture lock signal is output to control the device to automatically hover, effectively avoiding false triggering caused by abnormal operation or unexpected actions, and ensuring safety is prioritized when facing unknown operations.

[0104] The step of correcting the signal threshold according to the state correction coefficient includes the following steps:

[0105] The electromyographic signal data is subjected to time-frequency domain transformation to obtain frequency domain feature data;

[0106] The average power frequency of the electromyographic signal data is calculated based on the frequency domain feature data; the calculation formula is as follows:

[0107] ,

[0108] in, This represents the average power frequency of the electromyographic signal data. and These respectively represent the frequency domain feature data in the range of 0 to... The integral independent variable and the corresponding power spectral density within the range, Indicates the maximum bandwidth;

[0109] The state correction coefficient characterizing the operator's fatigue level is calculated based on the average power frequency; the calculation formula is as follows:

[0110] ,

[0111] in, Represents the state correction factor. This represents the average power frequency of the muscle group activation vector in the standard motion feature data corresponding to the real-time motion type. This represents the gain coefficient used to control the sensitivity to changes in fatigue. Indicates the bias number;

[0112] The signal threshold is corrected according to the state correction coefficient to obtain the corrected signal threshold; the correction formula is as follows:

[0113] ,

[0114] in, and These represent the signal thresholds before and after the correction, respectively. This represents the maximum correction factor.

[0115] The gain coefficient used to control the sensitivity to fatigue changes is set according to the operation time and fatigue accumulation rate of the application scenario. It is appropriately increased in long-term continuous operation scenarios and appropriately decreased in short-term intermittent operation scenarios.

[0116] The maximum correction factor represents the maximum reduction in the threshold of the limiting signal, preventing over-correction from causing false triggering, and is dynamically calibrated according to different action types.

[0117] The calculation of the physiological intent score includes the following steps:

[0118] Determine whether the amplitude of the electromyographic signal data is greater than the corrected signal threshold;

[0119] If so, the first gating state is output as passed, and the first difference between the amplitude of the electromyographic signal data and the corrected signal threshold is calculated;

[0120] Calculate the second difference between the amplitude of the muscle group activation vector in the standard action feature data corresponding to the real-time action type and the corrected signal threshold;

[0121] Calculate the ratio of the first difference to the second difference to obtain the physiological intention score;

[0122] If not, the first gating state is output as cut off, the physiological intention score is set to zero, the surface controller is controlled to maintain the current form lock, and the drone is controlled to perform a preset no-signal standby operation.

[0123] By determining the state correction coefficient for the operator's fatigue level based on electromyography (EMG) signal data and dynamically correcting the signal threshold, the problem of intention recognition failure caused by the decrease in EMG signal amplitude under fatigue state is solved. When the corrected EMG signal amplitude is greater than the correction threshold, the first gating state is output as passed and the physiological intention score is calculated, which ensures the effective capture of the real operation intention under fatigue state and realizes stable recognition under different physiological states.

[0124] The predicted stress intensity value, which characterizes the physical stress response generated by the physiological intention under a real-time action type, includes the following steps:

[0125] Acquire historical data corresponding to the real-time action types in several sets of tasks performed by drone operators to obtain a historical dataset; the historical data includes at least historical electromyography signal data, historical physiological intention scores, and historical standard pressure intensity values.

[0126] A final pressure intensity prediction model is constructed based on the historical dataset.

[0127] The final pressure intensity prediction model can be the Mamba model; the model construction process includes the following steps:

[0128] Construct an initial pressure intensity prediction model, set the training data ratio, such as 8:2 or 7:3, which can be reasonably adjusted according to the actual situation, and divide the historical dataset according to the training data ratio to obtain the training dataset and the test dataset.

[0129] Set a training error threshold, such as 5%-10%, which can be adjusted reasonably according to the actual situation. Input the training data in the training dataset into the initial stress intensity prediction model for training. Continuously adjust the parameters of the initial stress intensity prediction model according to the training results until the training error is less than the training error threshold, and obtain a well-trained stress intensity prediction model.

[0130] Set the test precision, such as 90%-95%, which can be adjusted reasonably according to the actual situation. Input the test data in the test dataset into the trained pressure intensity prediction model for testing, and calculate the accuracy of the test results. If the accuracy of the test results is greater than the test precision, the final pressure intensity prediction model is obtained; otherwise, retrain until the accuracy of the test results is greater than the test precision.

[0131] The structure of the initial pressure intensity prediction model can be seen in Table 1 below:

[0132]

[0133] Table 1

[0134] When the first gating state is passed, the electromyographic signal data and physiological intention score are input into the final pressure intensity prediction model to predict the standard pressure intensity value corresponding to the real-time action type performed by the drone operator, so as to obtain the expected pressure intensity value characterizing the physical pressure response generated by the physiological intention under the real-time action type.

[0135] The calculation of the physical action score includes the following steps:

[0136] The real-time pressure intensity value is calculated based on the pressure characteristic data.

[0137] Determine whether the deviation between the real-time pressure intensity value and the expected pressure intensity value is less than the deviation threshold. If so, output the second gating state as passed, and calculate the physical action score based on the standard pressure distribution characteristics in the standard action characteristic data corresponding to the real-time action type. The calculation formula is as follows:

[0138] ,

[0139] in, Indicates the score for physical actions. and These represent the real-time pressure intensity value and the area of ​​the effective touch region extracted from the pressure distribution feature data in the pressure feature data, respectively. and These represent the standard pressure intensity value and standard touch area extracted from the standard pressure distribution features in the standard action feature data corresponding to the real-time action type, respectively. and These represent the pressure characteristic trade-off coefficient and the area characteristic trade-off coefficient, respectively.

[0140] If not, the second gating state is output as cut off, the physical action score is set to zero, the surface controller is controlled to maintain the current form lock, and the drone is controlled to perform a preset no-signal standby operation.

[0141] The pressure characteristic trade-off coefficient and area characteristic trade-off coefficient are configured with parameters based on the varying degrees of dependence of each action type on pressure intensity and touch area. When the action relies more on the accuracy of pressure intensity, the weight of the pressure characteristic trade-off coefficient is increased, such as... It is 0.6. The value is 0.4, which can be adjusted reasonably according to the actual situation; when the action relies more on the trajectory of touch area changes, the weight of the area feature trade-off coefficient should be increased, such as... It is 0.4. The value is 0.6, but can be adjusted reasonably according to the actual situation.

[0142] Furthermore, the calculation of the real-time pressure intensity value includes the following steps:

[0143] Iterate through each sensing unit in the pressure distribution feature data of the pressure feature data and determine whether the pressure intensity value corresponding to each sensing unit is greater than the preset pressure intensity threshold.

[0144] If so, then the sensing unit is recorded as a valid sensing unit;

[0145] If not, then ignore the sensing unit;

[0146] After traversing all sensing units, all valid sensing units are summarized to obtain the set of valid contacts;

[0147] The root mean square value of the pressure intensity value corresponding to the sensing unit of each effective contact in the effective contact set is calculated to obtain the real-time pressure intensity value.

[0148] When the first gating state is passed, the second gating state is output by predicting the standard pressure characteristics corresponding to the physiological intention based on electromyographic signal data and judging whether the deviation between the actual pressure characteristics and the standard pressure characteristics is less than the threshold. This realizes closed-loop verification from physiological intention to physical execution. It effectively identifies and intercepts deviations in physical actions caused by external interference, ensuring that only operations with a high degree of consistency between physiological intention and physical action can trigger the instruction, thus reducing the risk of false triggering.

[0149] The surface controller adjusts the current shape according to the real-time action type, including the following steps:

[0150] When the second gating state is passed, the total evaluation value is calculated based on the physiological intention score and physical action score combined with the real-time action type; the calculation formula is as follows:

[0151] ,

[0152] in, This represents the total evaluation value. and These represent the preference weights of the real-time action type for physiological intention and physical action, respectively.

[0153] The preference weights of the real-time action type for physiological intentions and physical actions are dynamically calibrated according to the real-time action type.

[0154] Preferably, for high-risk command-type actions, such as emergency hovering and motor lock-up, because these actions involve critical changes in the drone's state and the consequences of mis-triggered actions are severe, the preference weight for physiological intent is set to 0.7-0.8, and the preference weight for physical actions is set to 0.2-0.3, emphasizing the explicit operational intent reflected by electromyographic signals; for precise control-type actions, such as attitude fine-tuning, because these actions require high operational stability and positional accuracy, the preference weight for physiological intent is set to 0.3-0.4, and the preference weight for physical actions is set to 0.6. -0.7, focusing on verifying the accuracy and positional precision of hand pressure distribution; for continuously adjustable actions, such as motor locking, since these actions involve continuous changes and are prone to slippage, the preference weight for physiological intention is set to 0.5-0.6, and the preference weight for physical action is set to 0.4-0.5, slightly emphasizing the verification of continuous adjustment intention, while tolerating the deviation of physical action caused by slippage; at the same time, when it is detected that the operator's pressure signal accuracy decreases due to wearing gloves, the weight of physiological intention is increased by 0.1-0.15 for all action types.

[0155] Determine whether the total evaluation value is greater than the preset execution threshold. If not, control the surface controller to maintain the current form lock and control the drone to perform the preset no-signal standby operation.

[0156] If so, the real-time action type is matched with the preset surface controller shape adjustment instruction library using a depth-first search algorithm. The surface controller shape adjustment instruction that matches the real-time action type is searched from the preset surface controller shape adjustment instruction library and the data is labeled to obtain the real-time surface controller shape adjustment instruction.

[0157] The surface controller shape adjustment command includes at least surface curvature adjustment parameters, damping coefficient adjustment parameters, and surface stiffness adjustment parameters;

[0158] Based on the total evaluation value, the adjustment parameters in the real-time surface controller shape adjustment command are adaptively adjusted to obtain the adjusted real-time surface controller shape adjustment command.

[0159] The adjusted real-time surface controller shape adjustment command is pushed to the main control unit through the wireless communication network. The main control unit executes the surface control task according to the adjusted real-time surface controller shape adjustment command and adjusts the shape of the current surface controller.

[0160] Furthermore, the adaptive adjustment of each adjustment parameter in the real-time surface controller shape adjustment command based on the total evaluation value includes the following steps:

[0161] The adaptive gain factor is calculated based on the total evaluation value and the execution threshold; the calculation formula is as follows:

[0162] ,

[0163] in, and These represent the total evaluation value and the execution threshold, respectively.

[0164] Based on the adaptive gain factor, the various adjustment parameters in the real-time surface controller shape adjustment command are adaptively adjusted to obtain the adjusted real-time surface controller shape adjustment command; the adjustment formula is as follows:

[0165] ,

[0166] in, and These represent the first and second parts of the real-time surface controller shape adjustment command before and after adaptive adjustment. Adjust parameters Indicates the first The compensation weights corresponding to the class adjustment parameters;

[0167] The compensation weights include at least curvature compensation weights, stiffness compensation weights, and damping compensation weights.

[0168] Preferably, the stiffness compensation weight has the most direct impact on the stability of the curved surface shape, and is set to 0.6 to 0.8; the curvature compensation weight affects the accuracy of the shape change, and is set to 0.3 to 0.5; the damping parameter mainly affects the dynamic response speed rather than the final shape, and is set to 0.2 to 0.4.

[0169] Exemplary computer-readable media

[0170] Embodiments of this application may also be computer-readable storage media storing computer program instructions thereon, which, when executed by a processor, cause the processor to perform the steps described in the "Exemplary Methods" section above according to the various embodiments of this application.

[0171] Computer-readable storage media may take the form of any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may, for example, include, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatuses, or devices, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0172] The basic principles of this application have been described above with reference to specific embodiments. However, it should be noted that the advantages, benefits, and effects mentioned in this application are merely examples and not limitations, and should not be considered as essential features of each embodiment of this application. Furthermore, the specific details disclosed above are for illustrative and facilitative purposes only, and are not limitations. These details do not limit the application to the necessity of employing the aforementioned specific details for implementation.

[0173] The block diagrams of devices, apparatuses, devices, and systems involved in this application are merely illustrative examples and are not intended to require or imply that they must be connected, arranged, or configured in the manner shown in the block diagrams. As those skilled in the art will recognize, these devices, apparatuses, devices, and systems can be connected, arranged, and configured in any manner. Words such as “comprising,” “including,” “having,” etc., are open-ended terms meaning “including but not limited to,” and are used interchangeably with them. The terms “or” and “and” as used herein refer to the terms “and / or,” and are used interchangeably with them unless the context clearly indicates otherwise. The term “such as” as used herein refers to the phrase “such as but not limited to,” and is used interchangeably with it.

[0174] It should also be noted that in the apparatus, equipment, and methods of this application, the components or steps can be disassembled and / or recombined. These disassemblies and / or recombinations should be considered as equivalent solutions of this application.

[0175] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use this application. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein can be applied to other aspects without departing from the scope of this application. Therefore, this application is not intended to be limited to the aspects shown herein, but rather to be accorded the widest scope consistent with the principles and novel features disclosed herein.

[0176] The above description has been given for purposes of illustration and description. Furthermore, this description is not intended to limit the embodiments of this application to the forms disclosed herein. Although numerous exemplary aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, alterations, additions, and sub-combinations thereof.

Claims

1. An adaptive human-computer interaction surface control system based on multimodal perception, characterized in that, include: The data acquisition module is used to acquire electromyographic signal data and stress characteristic data of the drone operator; The real-time motion determination module is used to determine the real-time motion type corresponding to the current operation based on the electromyographic signal data and pressure feature data combined with a preset standard motion feature database, and to obtain the signal threshold and deviation threshold corresponding to the real-time motion type. The electromyography (EMG) signal analysis module is used to determine a state correction coefficient representing the operator's fatigue level based on the EMG signal data, correct the signal threshold based on the state correction coefficient, and determine whether the amplitude of the EMG signal data is greater than the corrected signal threshold. If so, the first gating state is output as passed, and the physiological intention score is calculated. The pressure feature analysis module is used to predict the expected pressure intensity value, which characterizes the physical pressure response generated by the physiological intention under the real-time action type, based on the electromyographic signal data and the physiological intention score when the first gate state is passed, and to calculate the real-time pressure intensity value based on the pressure feature data. It then determines whether the deviation between the real-time pressure intensity value and the expected pressure intensity value is less than the deviation threshold. If so, it outputs that the second gate state is passed and calculates the physical action score. The curved surface control module is used to calculate the total evaluation value based on the physiological intention score and physical action score combined with the real-time action type when the second gate state is passed. Only when the total evaluation value meets the preset conditions, the curved surface controller adjusts the current form according to the real-time action type.

2. The adaptive human-computer interaction surface control system according to claim 1, characterized in that, Determining the real-time action type corresponding to the current operation and obtaining the signal threshold and deviation threshold corresponding to the real-time action type includes the following steps: The electromyography signal data and pressure feature data are cascaded to generate the real-time operation fingerprint of the current drone operator. The similarity between each standard action feature data in the preset standard action feature database and the real-time operation fingerprint is calculated sequentially to obtain a similarity matrix; Determine whether the maximum value of each similarity in the similarity matrix is ​​greater than a preset similarity threshold; If so, it means that the current drone operator's operation has successfully matched the operation type corresponding to the standard action feature data with the similarity, and the real-time action type is obtained; The minimum electromyographic amplitude and pressure deviation threshold corresponding to the standard action feature data corresponding to the similarity are labeled to obtain the signal threshold and deviation threshold corresponding to the real-time action type. If not, calculate the topological completeness between the pressure distribution feature data in the pressure feature data and the standard pressure distribution feature in the standard action feature data corresponding to the highest similarity in the similarity matrix; If the topological integrity is greater than the preset integrity threshold, then confidence compensation is performed on the electromyography signal data based on the pressure feature data. A similarity matrix is ​​regenerated based on the compensated electromyography signal data and the pressure feature data. If the maximum value of each similarity in the regenerated similarity matrix is ​​still less than or equal to the preset similarity threshold, then an attitude lock signal is output to the surface controller to control the drone to perform automatic hovering and wait for confirmation from the drone operator. If the topology integrity is less than or equal to a preset integrity threshold, an attitude lock signal is output to the surface controller to control the UAV to perform automatic hovering and wait for confirmation from the UAV operator.

3. The adaptive human-computer interaction surface control system according to claim 2, characterized in that, The confidence compensation of electromyographic signal data based on pressure characteristic data includes the following steps: The signal gain coefficient is calculated based on the topology integrity; the calculation formula is as follows: , in, Represents the signal gain coefficient. Represents the sensitivity coefficient. Indicates topological completeness. This indicates the preset integrity threshold; The electromyographic signal data is subjected to confidence compensation based on the signal gain coefficient to obtain compensated electromyographic signal data; the compensation formula is as follows: , in, and These represent the electromyographic signal data before and after confidence compensation, respectively. Indicates the regulating factor. This represents the muscle activation vector in the standard action feature data corresponding to the highest similarity in the similarity matrix.

4. The adaptive human-computer interaction surface control system according to claim 1, characterized in that, The step of correcting the signal threshold according to the state correction coefficient includes the following steps: The electromyographic signal data is subjected to time-frequency domain transformation to obtain frequency domain feature data; The average power frequency of the electromyographic signal data is calculated based on the frequency domain feature data, and the state correction coefficient characterizing the operator's fatigue level is calculated based on the average power frequency. The signal threshold is corrected according to the state correction coefficient to compensate for the natural attenuation of the electromyographic signal amplitude caused by muscle fatigue, thus obtaining the corrected signal threshold.

5. The adaptive human-computer interaction surface control system according to claim 1, characterized in that, The calculation of the physiological intent score includes the following steps: Determine whether the amplitude of the electromyographic signal data is greater than the corrected signal threshold; If so, the first gating state is output as passed, and the first difference between the amplitude of the electromyographic signal data and the corrected signal threshold is calculated; Calculate the second difference between the amplitude of the muscle group activation vector in the standard action feature data corresponding to the real-time action type and the corrected signal threshold; Calculate the ratio of the first difference to the second difference to obtain the physiological intention score; If not, the first gating state is output as cut off, the physiological intention score is set to zero, the surface controller is controlled to maintain the current form lock, and the drone is controlled to perform a preset no-signal standby operation.

6. The adaptive human-computer interaction surface control system according to claim 1, characterized in that, The calculation of the physical action score includes the following steps: Acquire historical data corresponding to the real-time action types in several sets of tasks performed by drone operators to obtain a historical dataset; the historical data includes at least historical electromyography signal data, historical physiological intention scores, and historical standard pressure intensity values. A final pressure intensity prediction model is constructed based on the historical dataset. When the first gating state is passed, the electromyographic signal data and physiological intention score are input into the final pressure intensity prediction model to predict the standard pressure intensity value corresponding to the real-time action type performed by the drone operator, so as to obtain the expected pressure intensity value characterizing the physical pressure response generated by the physiological intention under the real-time action type.

7. The adaptive human-computer interaction surface control system according to claim 1, characterized in that, The calculation of the physical action score includes the following steps: The real-time pressure intensity value is calculated based on the pressure characteristic data. Determine whether the deviation between the real-time pressure intensity value and the expected pressure intensity value is less than the deviation threshold. If so, output the second gating state as passed, and calculate the physical action score based on the standard pressure distribution characteristics in the standard action characteristic data corresponding to the real-time action type. The calculation formula is as follows: , in, Indicates the score for physical actions. and These represent the real-time pressure intensity value and the area of ​​the effective touch region extracted from the pressure distribution feature data in the pressure feature data, respectively. and These represent the standard pressure intensity value and standard touch area extracted from the standard pressure distribution features in the standard action feature data corresponding to the real-time action type, respectively. and These represent the pressure characteristic trade-off coefficient and the area characteristic trade-off coefficient, respectively. If not, the second gating state is output as cut off, the physical action score is set to zero, the surface controller is controlled to maintain the current form lock, and the drone is controlled to perform a preset no-signal standby operation.

8. The adaptive human-computer interaction surface control system according to claim 1, characterized in that, The surface controller adjusts the current shape according to the real-time action type, including the following steps: When the second gating state is passed, the total evaluation value is calculated based on the physiological intention score and physical action score combined with the real-time action type; the calculation formula is as follows: , in, This represents the total assessment value. and These represent the preference weights of the real-time action type for physiological intention and physical action, respectively. Determine whether the total evaluation value is greater than the preset execution threshold. If not, control the surface controller to maintain the current form lock and control the drone to perform the preset no-signal standby operation. If so, the real-time action type is matched with the preset surface controller shape adjustment instruction library using a depth-first search algorithm. The surface controller shape adjustment instruction that matches the real-time action type is searched from the preset surface controller shape adjustment instruction library and the data is labeled to obtain the real-time surface controller shape adjustment instruction. Based on the total evaluation value, the adjustment parameters in the real-time surface controller shape adjustment command are adaptively adjusted to obtain the adjusted real-time surface controller shape adjustment command. The adjusted real-time surface controller shape adjustment command is pushed to the main control unit through the wireless communication network. The main control unit executes the surface control task according to the adjusted real-time surface controller shape adjustment command and adjusts the shape of the current surface controller.

9. An electronic device comprising a memory and a processor, characterized in that: The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions, which, when executed by the processor, implement the system as described in any one of claims 1-7.

10. A computer storage medium storing computer-executable instructions thereon, characterized in that: When the computer-executable instructions are executed by a processor, they implement the system as described in any one of claims 1-7.

Images