A multi-modal human-machine interaction electric wheelchair control method and system

Abstract

This invention relates to the field of wheelchair control technology and proposes a multimodal human-computer interaction control method and system for electric wheelchairs. The method includes: extracting multimodal physiological state vectors and environmental risk perception vectors corresponding to the current user and their environment to construct an intent alignment matrix between the various input modalities of the electric wheelchair; extracting key scenario constraint indicators from the environmental risk perception vectors to determine the dynamic credibility weight of each input modality within the current decision-making cycle, combined with the intent alignment matrix; generating anti-disturbance drive commands for the electric wheelchair based on the dynamic credibility weights and the intent alignment matrix; and executing control actions of the electric wheelchair based on the anti-disturbance drive commands. This invention can effectively reduce the risk of loss of control caused by sudden environmental changes.

Description

Technical Field

[0001] This invention relates to a multimodal human-computer interaction electric wheelchair control method and system, belonging to the field of wheelchair control technology. Background Technology

[0002] With the increasing popularity of multimodal interaction technologies such as voice, vision, and haptic feedback in electric wheelchairs, the operational complexity of wheelchair control systems has been reduced, but they also face greater pressure in terms of information fusion and intent recognition. In order to more accurately understand user intent and achieve reliable control, it is necessary to further improve the system's comprehensive perception and intelligent decision-making capabilities for multi-source, asynchronous, and uncertain signals.

[0003] In existing technologies, the control of electric wheelchairs mostly adopts a static priority multimodal command fusion method. By assigning fixed priorities to input modalities such as voice, vision, and joysticks during the system initialization phase, command control of the wheelchair can be achieved. Although this method can maintain a relatively stable control effect in structured environments or low-interference scenarios, it cannot dynamically adjust the weights of each modality according to real-time environmental changes or user operation status, which makes electric wheelchairs prone to command misjudgment in complex or sudden scenarios. Summary of the Invention

[0004] This invention provides a multimodal human-computer interaction electric wheelchair control method and system, the main purpose of which is to effectively reduce the risk of loss of control caused by sudden environmental changes.

[0005] To achieve the above objectives, the present invention provides a multimodal human-computer interaction electric wheelchair control method, comprising: By deploying multi-source heterogeneous sensors on electric wheelchairs, multimodal physiological state vectors and environmental risk perception vectors corresponding to the current user and their environment are extracted. Based on the multimodal physiological state vector and the environmental risk perception vector, an intent alignment matrix is ​​constructed between the input modalities of the electric wheelchair. Key scenario constraint indicators are extracted from the environmental risk perception vector, and the dynamic credibility weight of each input modality in the current decision-making cycle is determined based on the key scenario constraint indicators and the intent alignment matrix. Based on the dynamic credibility weight and the intent alignment matrix, a multimodal interaction decision graph of the electric wheelchair is constructed, and based on the multimodal interaction decision graph, an anti-disturbance drive command for the electric wheelchair is generated. Based on the anti-disturbance drive command, the control actions of the electric wheelchair are executed.

[0006] Optionally, based on the key scenario constraint indicators and the intent alignment matrix, the dynamic credibility weight of each input modality within the current decision-making cycle is determined, including: Based on the aforementioned key scenario constraint indicators, calculate the constraint risk value for each input mode; The mean of the intent correlation between each input modality and all other modalities is extracted from the intent alignment matrix to obtain the intent of the corresponding input modality. Figure 1 Consistency support; The constraint risk value is reverse normalized to obtain the scene safety adaptability of the corresponding input mode; Based on the above intention Figure 1 Consistency support and scenario security adaptability are used to determine the dynamic credibility weight of each input modality in the current decision-making cycle.

[0007] Optionally, based on the key scenario constraint indicators, the constraint risk value of each input mode is calculated, including: Obtain the parsing instructions corresponding to each input mode within the current decision cycle, wherein the parsing instructions include control type and control parameters; Establish a mapping matrix between the control parameters and the key scenario constraint indicators to calculate the dynamic risk accumulation value of each input mode; Based on the dynamic risk accumulation value, the constraint risk value for each input mode is generated.

[0008] Optionally, a mapping matrix between the control parameters and the key scenario constraint indicators is established, including: The control parameters are projected onto a preset constraint parameter space to obtain the standardized constraint coordinates of each control parameter; Based on the key scenario constraint indicators, a dynamic feasible region is constructed in the constraint parameter space; Calculate the original safety margin from each of the standardized constraint coordinates to the boundary of the dynamic feasible region constraint; The original safety margin is weighted and scaled to generate a mapping matrix between the control parameters and the key scenario constraint indicators.

[0009] Optionally, based on the multimodal interaction decision graph, disturbance-resistant drive commands for the electric wheelchair are generated, including: The node weights and edge connection strengths of the multimodal interaction decision graph are analyzed. Based on the node weights and edge connection strengths, the core control intent and auxiliary control signals in the multimodal interaction decision graph are identified. Based on the core control intent, the basic driving action sequence of the electric wheelchair is determined; Based on the auxiliary control signal and the edge connection strength, calculate the dynamic adjustment coefficient corresponding to the basic driving action sequence; After correcting the basic drive action sequence using the dynamic adjustment coefficient, the disturbance-resistant drive command of the electric wheelchair is obtained.

[0010] Optionally, based on the core control intent, the basic driving action sequence of the electric wheelchair is determined, including: The core control intent is parsed into the basic control vector of the electric wheelchair; Retrieve candidate drive action sequences that match the basic control vector from a preset action instruction library; Based on the response priority of the core control intent, target candidate sequences are selected from the candidate driving action sequences; Define the security dynamic constraints of the core control intent; Based on the aforementioned safety dynamic constraints, the target candidate sequence is subjected to parameter optimization processing to obtain the basic driving action sequence.

[0011] Optionally, based on the auxiliary control signal and the edge connection strength, the dynamic adjustment coefficient corresponding to the basic driving action sequence is calculated, including: Extract the adjustment instruction components from the auxiliary control signal that correspond to each control dimension of the basic driving action sequence; Obtain the edge connection strength value between the modal node corresponding to the auxiliary control signal and the node corresponding to the core control intent; The dynamic adjustment coefficients corresponding to the basic driving action sequence are calculated based on the adjustment command components and the edge connection strength values.

[0012] Optionally, the input modalities include: voice modalities, visual gesture modalities, and joystick modalities.

[0013] Optionally, the control action is performed by the electric wheelchair according to the anti-disturbance drive command, driving at least one actuator among its hub motor, steering servo, and lifting mechanism.

[0014] To address the aforementioned problems, the present invention also provides a multimodal human-computer interaction electric wheelchair control system, the system comprising: The vector extraction module is used to extract multimodal physiological state vectors and environmental risk perception vectors corresponding to the current user and their environment through multi-source heterogeneous sensors deployed on the electric wheelchair. The matrix construction module is used to construct the intent alignment matrix between each input modality of the electric wheelchair based on the multimodal physiological state vector and the environmental risk perception vector. The weight calculation module is used to extract key scenario constraint indicators from the environmental risk perception vector, and determine the dynamic credibility weight of each input modality in the current decision-making cycle based on the key scenario constraint indicators and the intent alignment matrix. The instruction generation module is used to construct a multimodal interaction decision graph of the electric wheelchair based on the dynamic credibility weight and the intent alignment matrix, and generate anti-disturbance drive instructions for the electric wheelchair based on the multimodal interaction decision graph. The control execution module is used to execute the control actions of the electric wheelchair based on the anti-disturbance drive command.

[0015] Compared to the problems described in the background art, the embodiments of the present invention construct an intent alignment matrix between the various input modalities of the electric wheelchair based on the multimodal physiological state vector and the environmental risk perception vector. This matrix can transform the original physiological signals and environmental risk data into a reflection of the intent between the multimodal commands of the electric wheelchair. Figure 1 Furthermore, based on the key scenario constraint indicators and the intent alignment matrix, this embodiment of the invention determines the dynamic credibility weight of each input modality within the current decision-making cycle, ensuring that the electric wheelchair can make decisions based on environmental constraints and the intent alignment matrix. Figure 1 The consistency level is dynamically adjusted to regulate the decision influence of each input modality, thereby ensuring that the final generated control commands not only conform to the user's true intentions but also strictly meet real-time safety boundary conditions. Based on the dynamic credibility weights and the intention alignment matrix, this embodiment of the invention constructs a multimodal interaction decision graph for the electric wheelchair. This graph can dynamically parse the user's true intentions based on its global relational structure, significantly reducing the command misjudgment rate of the static priority fusion method. Furthermore, based on the multimodal interaction decision graph, this embodiment of the invention generates anti-disturbance drive commands for the electric wheelchair, which can reflect the multimodal intention relationships of the user. This is transformed into a reliable control signal capable of resisting environmental interference and execution errors during actual driving, thereby ensuring the safe driving of the wheelchair in complex situations and accurately responding to user intentions. Finally, by executing the control actions of the electric wheelchair based on the aforementioned anti-disturbance drive commands, the electric wheelchair can significantly improve its anti-interference ability and motion stability in disturbance scenarios such as sudden obstacles, road bumps, and strong crosswinds, effectively reducing the risk of loss of control caused by sudden environmental changes. At the same time, by outputting smooth and continuous motion commands, it avoids the abrupt jerks caused by control vibrations and sudden stops and starts, thereby greatly improving the naturalness and comfort of human-computer interaction. Therefore, this invention can effectively reduce the risk of loss of control caused by sudden environmental changes. Attached Figure Description

[0016] Figure 1 This is a flowchart illustrating a multimodal human-computer interaction electric wheelchair control method according to an embodiment of the present invention. Figure 2 A human-computer interaction model diagram for implementing a multimodal human-computer interaction electric wheelchair control method according to an embodiment of the present invention is shown. Figure 3 This is a schematic diagram of the modules for implementing a multimodal human-computer interaction electric wheelchair control method according to an embodiment of the present invention; Figure 4 A schematic diagram of a computer device for a multimodal human-computer interaction electric wheelchair control method provided in an embodiment of the present invention; The objectives, features, and advantages of this invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0017] It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0018] This application provides a multimodal human-computer interaction method for controlling an electric wheelchair. The executing entity of this multimodal human-computer interaction method includes, but is not limited to, at least one of the following electronic devices that can be configured to execute the method provided in this application: a server, a terminal, etc. In other words, the multimodal human-computer interaction method for controlling an electric wheelchair can be executed by software or hardware installed on a terminal device or a server device. The server includes, but is not limited to, a single server, a server cluster, a cloud server, or a cloud server cluster.

[0019] Reference Figure 1 The diagram shown is a flowchart illustrating a multimodal human-computer interaction electric wheelchair control method according to an embodiment of the present invention. In this embodiment, the multimodal human-computer interaction electric wheelchair control method includes: S1. By deploying multi-source heterogeneous sensors on electric wheelchairs, extract multimodal physiological state vectors and environmental risk perception vectors corresponding to the current user and their environment.

[0020] This invention utilizes multi-source heterogeneous sensors deployed on electric wheelchairs to extract multimodal physiological state vectors and environmental risk perception vectors corresponding to the current user and their environment. This enables the analysis of the relationship between the user's true control intention and the current environmental constraints within a unified semantic space, thereby providing a data foundation for dynamically adjusting the credibility weights of each input modality of the electric wheelchair.

[0021] In detail, the multi-source heterogeneous sensor refers to a combination of sensors deployed at different locations on the electric wheelchair, operating based on different physical principles, and capable of collecting various types of data. For example, it could be a skin conductivity sensor integrated on the armrest surface, a heart rate monitoring module built into the backrest, a miniature camera facing the user, a directional microphone array, etc.; the multimodal physiological state vector refers to a vector used to quantitatively characterize the user's current physiological and cognitive state; the environmental risk perception vector refers to a vector used to quantitatively characterize the potential threat to the user's driving safety posed by the external environment when using the electric wheelchair.

[0022] Optionally, the multimodal physiological state vector and environmental risk perception vector corresponding to the current user and their environment can be extracted using a general data processing procedure that combines feature engineering with machine learning models. For example, after dividing the physiological signals and environmental signals collected by the sensors into fixed-duration windows, the time-domain statistics within the calculated window can be used as basic features.

[0023] S2. Based on the multimodal physiological state vector and the environmental risk perception vector, construct the intent alignment matrix between each input modality of the electric wheelchair.

[0024] This invention constructs an intent alignment matrix between the various input modalities of the electric wheelchair based on the multimodal physiological state vector and the environmental risk perception vector. This matrix can transform the original physiological signals and environmental risk data into a representation of the intent between the multimodal commands of the electric wheelchair. Figure 1 The key indicator of consistency is the input modality, which refers to the different types of interaction methods used by the user when issuing control commands to the electric wheelchair, including voice modality, visual gesture modality, and joystick modality; the intent alignment matrix refers to a matrix formed by mapping information from different modalities to the same semantic space to describe the intent between the modalities. Figure 1 A structured matrix representing the degree of consistency.

[0025] As an embodiment of the present invention, an intent alignment matrix between the input modalities of the electric wheelchair is constructed based on the multimodal physiological state vector and the environmental risk perception vector, including: calculating the intent representation confidence of each input modality based on the multimodal physiological state vector; calculating the scene adaptation confidence of each input modality based on the environmental risk perception vector; generating dynamic alignment weights for each input modality based on the intent representation confidence and the scene adaptation confidence; extracting modal feature vectors from the original command signals of each input modality; calculating the intent correlation degree between each input modality based on the modal feature vectors and the dynamic alignment weights; and calculating the intent alignment matrix between each input modality based on the intent correlation degree.

[0026] The confidence level of intent representation refers to the reliability of the user intent conveyed by each input modality. This indicator reflects the probability that a specific input channel accurately conveys the user's true intent under the user's current physiological and cognitive state. For example, when the system detects a significant increase in "arm muscle fatigue" in the user's electromyography (EMG) signal, steering commands input via a joystick may be assigned a lower confidence score due to decreased operational precision; conversely, when the user's voice signal is clear and their heart rate is stable, voice commands may receive a higher confidence score. The scenario-fit reliability refers to the degree of executability and safety matching of the commands expressed by each input modality under the current environmental constraints. The dynamic alignment weight refers to the real-time normalized influence coefficient assigned to each input modality after combining the intent representation confidence and scenario-fit reliability. The original command signal refers to the sequence of underlying physical signals directly collected from each input device without semantic parsing, such as multi-channel audio waveforms collected from a microphone array or six-axis motion data streams collected from an inertial measurement unit. The modality feature vector refers to the standardized vector located in a unified intent semantic space generated after feature extraction and semantic embedding of the original command signal. The intent correlation refers to the degree of consistency between the user intents conveyed by two different input modalities.

[0027] Optionally, the confidence level of intent representation and the confidence level of scene adaptation for each input modality can be calculated based on a Bayesian inference model; the degree of intent correlation between each input modality can be calculated using the mutual information method.

[0028] S3. Extract key scenario constraint indicators from the environmental risk perception vector, and determine the dynamic credibility weight of each input modality in the current decision-making cycle based on the key scenario constraint indicators and the intent alignment matrix.

[0029] This invention, by extracting key scenario constraint indicators from the environmental risk perception vector, ensures that the subsequent multimodal interactive decision-making graph can adapt to changes in external risks in real time. These key scenario constraint indicators refer to environmental indicators directly affecting the safe driving and operability of electric wheelchairs, extracted from the environmental risk perception vector. For example, the environmental risk perception vector contains risk assessment values ​​in three dimensions: "distance to obstacles ahead," "ground slope angle," and "effective width of passage." From these values, the following can be extracted: minimum safe braking distance, maximum safe climbing angle, and minimum passage width for the wheelchair.

[0030] Alternatively, principal component analysis can be used to extract key scenario constraint indicators from the environmental risk perception vector.

[0031] Furthermore, in this embodiment of the invention, based on the key scenario constraint indicators and the intent alignment matrix, the dynamic credibility weight of each input modality is determined within the current decision-making cycle. This ensures that the electric wheelchair can make decisions based on the hard environmental constraints and the intent alignment between modalities. Figure 1 The consistency level is dynamically adjusted to regulate the decision-making influence of each input modality, thereby ensuring that the final generated control command not only conforms to the user's true intentions but also strictly meets real-time safety boundary conditions. The dynamic reliability weight refers to the weight based on current environmental constraints and multimodal intentions. Figure 1 Consistency relationship, the decision influence coefficient assigned in real time to each input modality.

[0032] As an embodiment of the present invention, determining the dynamic credibility weight of each input modality within the current decision-making cycle based on the key scenario constraint index and the intent alignment matrix includes: calculating the constraint risk value of each input modality based on the key scenario constraint index; extracting the average intent correlation between each input modality and all other modalities from the intent alignment matrix to obtain the intent of the corresponding input modality. Figure 1 Consistency support; the constraint risk value is inversely normalized to obtain the scene safety adaptability of the corresponding input modality; based on the intent Figure 1 Consistency support and scenario security adaptability are used to determine the dynamic credibility weight of each input modality in the current decision-making cycle.

[0033] The constraint risk value refers to the degree of likelihood that each input modality will cause a safety violation under the current environmental constraints. For example, under the environmental constraint of detecting "there is a staircase ahead," the voice command "go forward" has a constraint risk value of 0.9 because it may cause a fall; while the gesture command "stop" has a constraint risk value of 0.1. Figure 1 Consistency support refers to the strength of the collaborative verification of the intent conveyed by each input modality by other modalities; the scene safety adaptability refers to the degree to which the instructions conveyed by each input modality match the safety requirements of the current environment. For example, under the constraint of "the corridor width is only 0.9 meters", the constraint risk value of the voice instruction "go straight" is 0.2, and after inverse normalization, the scene safety adaptability is 0.8 (high adaptability); while the constraint risk value of the joystick instruction "turn at a large angle" is 0.8, and its scene safety adaptability is only 0.2.

[0034] Optionally, the constraint risk value can be reverse normalized by inversion calculation based on the Sigmoid function; the dynamic confidence weight of each input mode in the current decision period can be determined by probability distribution transformation method based on softmax function.

[0035] As an optional embodiment of the present invention, the constraint risk value of each input mode is calculated based on the key scenario constraint index, including: obtaining the parsing instruction corresponding to each input mode in the current decision cycle, wherein the parsing instruction includes a control type and control parameters; establishing a mapping relationship matrix between the control parameters and the key scenario constraint index to calculate the dynamic risk accumulation value of each input mode; and generating the constraint risk value of each input mode based on the dynamic risk accumulation value.

[0036] Further, a mapping matrix between the control parameters and the key scenario constraint indicators is established to calculate the dynamic risk accumulation value of each input mode, including: calculating the absolute deviation of each control parameter relative to the corresponding constraint threshold according to the mapping matrix; determining the coupling influence factor between the key scenario constraint indicators; performing correlation correction on the absolute deviation through the coupling influence factor to obtain the corrected deviation; identifying the constraint compliance of each input mode in the historical decision-making cycle; and calculating the dynamic risk accumulation value of each input mode based on the constraint compliance and the corrected deviation.

[0037] The parsed instructions refer to instructions with clear control semantics identified from the original signals of each input modality; the control parameters refer to quantifiable operational values ​​in the parsed instructions. For example, the target speed value (e.g., 5 km / h) corresponding to the "accelerate" instruction, and the steering angle value (e.g., 30 degrees) corresponding to the "turn left" instruction; the mapping relationship matrix refers to a matrix used to characterize the matching relationship between each control parameter and each key scenario constraint index. The rows of this matrix correspond to control parameters, the columns correspond to constraint indexes, and the element values ​​represent the degree of compliance of the parameter under the constraint; the dynamic risk accumulation value refers to the quantified result of the constraint violation risk accumulated by each input modality in a continuous decision-making cycle after combining the current deviation and historical behavior. The higher the value, the stronger the trend of the modality generating unsafe instructions in the near future; the constraint threshold refers to the safety boundary value corresponding to each key scenario constraint index; and the absolute deviation refers to the degree of difference between the actual value of the control parameter and the corresponding constraint threshold. For example, when the actual speed is 7 km / h and the "maximum safe speed" threshold is 6 km / h, the absolute deviation is |7-6| / 6 = 0.167; the coupling influence factor refers to the strength coefficient of the mutual influence between different constraint indicators. For example, the "slippery ground" constraint will enhance the risk weight of the "maximum braking distance" constraint; the constraint compliance refers to the stability of each input mode in complying with constraints in historical decision-making cycles. For example, if a mode complies with safety constraints 9 times in the past 10 cycles, its constraint compliance is 0.9.

[0038] Optionally, the coupling influence factors among the key scenario constraint indicators can be determined by the Pearson correlation coefficient method; the constraint compliance of each input mode in the historical decision-making cycle can be identified by the sliding window statistical method.

[0039] For example, the dynamic risk accumulation value is calculated using the following formula. It should be noted that this calculation method is only one possible method and does not affect the implementation of the basic scheme above: ; in, This represents the cumulative value of dynamic risk. Indicates the current correction deviation. Indicates the current degree of constraint compliance. This represents the cumulative risk value at the previous moment. This indicates the weight of the current deviation. This indicates the degree of attenuation.

[0040] It should be noted that the above formula introduces an exponential decay mechanism that follows degree modulation (...). This paper constructs a nonlinear relationship between historical behavior credibility and current risk, breaking through the limitations of traditional risk assessment where historical data only participates in linear weighted averages or static threshold comparisons. Simultaneously, this formula, through risk memory separation and weighted fusion structure, incorporates the immediate risk contribution of the current period. ) and the historical risk residue from the previous cycle ( ) are categorized into independently interpretable terms and weighted using coefficients. By dynamically adjusting the proportions of the two in the cumulative risk, the mathematical model can adapt to the decay law and memory strength of risk events in time-varying scenarios, thus providing a standardized quantitative basis for the subsequent allocation of dynamic credibility weights that combines timely response and historical behavioral memory.

[0041] Specifically, the current deviation impact weight is a coefficient that controls the proportion of newly generated deviations in the cumulative risk value within the current decision-making cycle, with a value range of α ∈ [0.3, 0.7]. This range balances the needs of historical memory and immediate response, maintaining the stability and sensitivity of risk assessment in most interactive scenarios. The compliance attenuation coefficient is a proportional coefficient that determines the strength of the suppression of current deviation by historical constraint compliance, with a value range of... [1,2.5] This range can effectively reflect the cognitive logic that "a good historical record should reduce the current risk weight" while avoiding excessive suppression that could lead to missed risk assessments.

[0042] As another optional embodiment of the present invention, establishing a mapping relationship matrix between the control parameters and the key scenario constraint indicators includes: projecting the control parameters onto a preset constraint parameter space to obtain standardized constraint coordinates of each control parameter; constructing a dynamic feasible region in the constraint parameter space based on the key scenario constraint indicators; calculating the original safety margin from each standardized constraint coordinate to the constraint boundary of the dynamic feasible region; and performing weight allocation and scaling transformation on the original safety margin to generate a mapping relationship matrix between the control parameters and the key scenario constraint indicators.

[0043] The constraint parameter space refers to a standardized mathematical space used to uniformly represent all control parameters and constraint indicators. For example, parameters with different dimensions, such as speed and angle, are normalized to a multi-dimensional space consisting of the interval [0,1]. The standardized constraint coordinates refer to the position coordinates of the control parameters in the constraint parameter space. For example, a speed of 5 km / h (normalized to 0.6) and a steering angle of 30 degrees (normalized to 0.5) form coordinate points (0.6, 0.5) in a two-dimensional space. The dynamic feasible region refers to the safe operating area defined by all key scenario constraint indicators within the constraint parameter space. For example, in the speed-steering angle two-dimensional space, it is a closed area enclosed by the constraints of "maximum safe speed" and "minimum turning radius". The constraint boundary refers to the geometric boundary line / surface of the dynamic feasible region, corresponding to the critical threshold of each constraint indicator. The original safety margin refers to the geometric distance from the standardized constraint coordinates to the nearest constraint boundary. For example, the vertical distance from the coordinate point (0.6, 0.5) to the speed boundary line (0.8) is 0.2, and this value is the original safety margin.

[0044] Optionally, a dynamic feasible region can be constructed in the constraint parameter space using the convex hull algorithm; the original safety margin from each of the standardized constraint coordinates to the constraint boundary of the dynamic feasible region can be calculated using the Euclidean distance formula; and the original safety margin can be weighted and scaled using the entropy weight method.

[0045] S4. Based on the dynamic credibility weight and the intent alignment matrix, construct the multimodal interaction decision graph of the electric wheelchair, and generate the anti-disturbance drive command of the electric wheelchair according to the multimodal interaction decision graph.

[0046] This invention constructs a multimodal interaction decision graph for the electric wheelchair based on the dynamic credibility weights and the intent alignment matrix. This graph dynamically parses the user's true intent based on the global association structure of the graph, significantly reducing the instruction misjudgment rate of static priority fusion methods. The multimodal interaction decision graph is a topological network structure used to describe the dynamic decision relationships between the various input modalities of the electric wheelchair. This graph uses each input modality as a node and the intent relationships between modalities as the basis for its structure. Figure 1The correlation strength is used as the edge, and the dynamic credibility weight of each modality is mapped to the attribute value of the node, thus forming a decision network that can be updated and reasoned in real time.

[0047] As an embodiment of the present invention, a multimodal interaction decision graph of the electric wheelchair is constructed based on the dynamic credibility weight and the intent alignment matrix, including: constructing a weighted directed graph for each input modality based on the intent relevance in the intent alignment matrix and the dynamic credibility weight; filtering out strong connections with edge weights exceeding a preset relevance threshold from the weighted directed graph; and constructing the multimodal interaction decision graph of the electric wheelchair based on the node weights of the weighted directed graph and the strong connections.

[0048] The weighted directed graph refers to a directed network graph with each input modality as a node, the intention correlation between modalities as edge weights, and the dynamic credibility weight of each modality as node weights; the correlation threshold refers to a critical value used to determine whether the intention correlation between modalities is significant; the strong connection relationship refers to the connection relationship between modalities in the weighted directed graph where the edge weight exceeds the correlation threshold; and the node weight refers to the dynamic credibility weight value attached to each modal node in the weighted directed graph.

[0049] Alternatively, weighted directed graphs of each input modality can be constructed using graph theory modeling tools, such as Python's NetworkX library.

[0050] Furthermore, in this embodiment of the invention, based on the multimodal interaction decision graph, an anti-disturbance drive command for the electric wheelchair is generated. This can transform the decision graph, which reflects the user's multimodal intent relationships, into a reliable control signal that can resist environmental interference and execution errors during actual driving. This ensures the safe driving of the wheelchair in complex situations and accurately responds to the user's intent. The anti-disturbance drive command refers to the final motor control signal generated after fusing multimodal intents and considering real-time environmental risks. This signal can actively cancel or adapt to external interference and internal noise.

[0051] As an embodiment of the present invention, generating an anti-disturbance driving command for the electric wheelchair based on the multimodal interaction decision graph includes: parsing the node weights and edge connection strengths of the multimodal interaction decision graph; identifying the core control intent and auxiliary control signals in the multimodal interaction decision graph based on the node weights and edge connection strengths; determining the basic driving action sequence of the electric wheelchair based on the core control intent; calculating the dynamic adjustment coefficients corresponding to the basic driving action sequence based on the auxiliary control signals and edge connection strengths; and obtaining the anti-disturbance driving command for the electric wheelchair after correcting the basic driving action sequence using the dynamic adjustment coefficients.

[0052] The core control intent refers to high-level semantic information inferred from the multimodal interaction decision graph that characterizes the user's current primary motion goal or desired state; the auxiliary control signal refers to additional information identified from the multimodal interaction decision graph that is used to refine and supplement the core control intent; the basic driving action sequence refers to a specific control command sequence planned based on the core control intent, considering the basic motion model and static constraints, without considering dynamic disturbances; and the dynamic adjustment coefficient refers to a parameter correction factor calculated based on the auxiliary control signal and its connection strength in the graph, used for online adjustment of the basic driving action sequence.

[0053] Optionally, the core control intent and auxiliary control signals in the multimodal interactive decision graph can be identified by node clustering algorithms, such as the DBSCAN algorithm; reinforcement learning models, such as the Q-learning model, can be used to correct the basic driving action sequence.

[0054] As an optional embodiment of the present invention, determining the basic driving action sequence of the electric wheelchair based on the core control intent includes: parsing the core control intent into a basic control vector of the electric wheelchair; retrieving candidate driving action sequences that match the basic control vector from a preset action instruction library; filtering out target candidate sequences from the candidate driving action sequences based on the response priority of the core control intent; defining safety dynamic constraints of the core control intent; and performing parameter optimization processing on the target candidate sequences according to the safety dynamic constraints to obtain the basic driving action sequence.

[0055] The basic control vector refers to the key motion parameter vector extracted from the core control intent and directly used for motion planning; the motion instruction library refers to a pre-set set of standardized control instructions that directly correspond to the underlying actuators (such as motors and servos) of the electric wheelchair. For example, the entries in the motion instruction library may be: Instruction ID: Turn_10deg_Left, Execution parameters: [Left wheel differential speed: +0.1m / s, Right wheel differential speed: -0.1m / s, Duration: 0.5s], Achieved effect: Turn left approximately 10 degrees in place]; the candidate drive motion sequence refers to a preliminary scheme that can achieve the motion effect described by the basic control vector, composed of multiple basic instructions from the motion instruction library arranged in chronological order; the response priority refers to the decision rule level at which the system prioritizes satisfying a target when multiple motion targets or constraints conflict; the target candidate sequence refers to the single execution scheme that prioritizes satisfying the core constraint, selected from all candidate drive motion sequences according to the response priority rule. For example, if the response priority requires "collision avoidance first", then a candidate sequence that needs to pass through a potential collision area will be eliminated, while another sequence that has a longer detour but is absolutely safe will be selected as the target candidate sequence; the safety dynamic constraint refers to the safety boundary conditions used to limit the execution parameters of the action sequence, which are calculated based on the real-time perceived user state (such as fatigue) and environmental risks (such as slippery road surface).

[0056] Optionally, an intent mapping algorithm, such as a nonlinear fitting algorithm, can be used to parse the core control intent into the basic control vector of the electric wheelchair; the safety dynamic constraints of the core control intent can be defined by a threshold adaptive algorithm, such as a sliding window threshold algorithm; and the particle swarm optimization algorithm (PSO) can be used to optimize the parameters of the target candidate sequence.

[0057] As another optional embodiment of the present invention, the dynamic adjustment coefficient corresponding to the basic driving action sequence is calculated based on the auxiliary control signal and the edge connection strength, including: extracting the adjustment instruction component in the auxiliary control signal that corresponds to each control dimension of the basic driving action sequence; obtaining the edge connection strength value between the modal node corresponding to the auxiliary control signal and the node corresponding to the core control intent; and calculating the dynamic adjustment coefficient corresponding to the basic driving action sequence according to the adjustment instruction component and the edge connection strength value.

[0058] The adjustment command component refers to the quantized correction value contained in the auxiliary control signal, used to fine-tune specific motion parameters in the basic driving action sequence. For example, if the basic driving action sequence contains the parameter "speed: 0.8m / s", and the auxiliary control signal is "slightly slower", then its adjustment command component may be quantized as "speed correction value: -0.1m / s". The edge connection strength value refers to the weight value of the edge connecting the modal node corresponding to the auxiliary control signal and the modal node corresponding to the core control intent in the multimodal interaction decision graph. For example, if the edge connection strength value between the voice node representing "slightly slower" and the core gesture node representing "straight ahead" is 0.3, it indicates that the auxiliary signal has a weak influence on the correction of the core intent.

[0059] Optionally, the adjustment command components in the auxiliary control signal corresponding to each control dimension of the basic driving action sequence can be extracted by an adaptive filtering algorithm; the dynamic adjustment coefficients corresponding to the basic driving action sequence can be calculated using a linear weighted fusion algorithm, such as using the expected value of the motion parameter correction as the adjustment benchmark and the edge connection strength value as the adjustment ratio factor, and calculating the dynamic adjustment coefficients by a linear weighted fusion algorithm.

[0060] S5. Based on the anti-disturbance drive command, execute the control actions of the electric wheelchair.

[0061] This invention, through the execution of control actions of the electric wheelchair based on the anti-disturbance drive command, significantly improves the anti-interference capability and motion stability of the electric wheelchair in disturbance scenarios such as sudden obstacles, road bumps, and strong crosswinds, effectively reducing the risk of loss of control caused by sudden environmental changes. Simultaneously, by outputting smooth and continuous motion commands, it avoids the abrupt jerks caused by control vibrations and sudden stops and starts, thereby greatly improving the naturalness and comfort of human-computer interaction. The control actions are completed by the electric wheelchair according to the anti-disturbance drive command, driving at least one actuator among its wheel hub motor, steering servo, and lifting mechanism. For example, when the anti-disturbance drive command is [left wheel speed: 1.0 m / s, right wheel speed: 0.8 m / s, duration: 2.0 s], the corresponding control action is for the wheelchair to complete a smooth left-turning arc motion within 2 seconds. If the command includes seat lifting parameters, the control action also includes the precise extension and retraction of the electric push rod to adjust the seat height.

[0062] See Figure 2 The diagram shown is a human-computer interaction model diagram of an electric wheelchair control method for multimodal human-computer interaction provided in an embodiment of the present invention. The diagram intuitively illustrates the collaborative process between user commands, environmental inputs and wheelchair responses, and clearly demonstrates how this solution combines human intent with machine intelligent control to achieve stable and safe motion control in complex disturbance environments, thereby improving the naturalness of the overall interaction and riding comfort.

[0063] like Figure 3 The diagram shown is a functional block diagram of an electric wheelchair control system with multimodal human-computer interaction according to the present invention.

[0064] The multimodal human-computer interaction electric wheelchair control system 200 described in this invention can be installed in an electronic device. Depending on the functions implemented, the multimodal human-computer interaction electric wheelchair control system includes a vector extraction module 201, a matrix construction module 202, a weight calculation module 203, an instruction generation module 204, and a control execution module 205. The modules described in this invention can also be referred to as units, which are a series of computer program segments that can be executed by the processor of an electronic device and can perform a fixed function, and are stored in the memory of the electronic device.

[0065] In this embodiment of the invention, the functions of each module / unit are as follows: The vector extraction module 201 is used to extract multimodal physiological state vectors and environmental risk perception vectors corresponding to the current user and his / her environment through multi-source heterogeneous sensors deployed on the electric wheelchair. The matrix construction module 202 is used to construct an intent alignment matrix between the input modalities of the electric wheelchair based on the multimodal physiological state vector and the environmental risk perception vector. The weight calculation module 203 is used to extract key scenario constraint indicators from the environmental risk perception vector, and determine the dynamic credibility weight of each input modality in the current decision-making cycle based on the key scenario constraint indicators and the intent alignment matrix. The instruction generation module 204 is used to construct a multimodal interaction decision graph of the electric wheelchair based on the dynamic credibility weight and the intent alignment matrix, and generate anti-disturbance drive instructions for the electric wheelchair according to the multimodal interaction decision graph. The control execution module 205 is used to execute the control actions of the electric wheelchair based on the anti-disturbance drive command.

[0066] In detail, the modules in the multimodal human-computer interaction electric wheelchair control system 200 described in this embodiment of the invention employ the same methods as described above during use. Figure 1 The method used is the same as the multimodal human-computer interaction electric wheelchair control method described in the article, and can produce the same technical effect, so it will not be repeated here.

[0067] In one embodiment, a computer device is provided, which may be a server or a client, and its internal structure diagram may be as follows: Figure 4As shown, the computer device includes a processor, memory, network interface, and database connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile and / or volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The network interface is used for communication with external clients via a network connection. When the computer program is executed by the processor, it implements the functions or steps of a multimodal human-computer interaction electric wheelchair control method on the server or client side.

[0068] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to perform the following steps: S1. By using multi-source heterogeneous sensors deployed on electric wheelchairs, extract multimodal physiological state vectors and environmental risk perception vectors corresponding to the current user and their environment; S2. Construct the intent alignment matrix between each input modality of the electric wheelchair based on the multimodal physiological state vector and the environmental risk perception vector; S3. Extract key scenario constraint indicators from the environmental risk perception vector, and determine the dynamic credibility weight of each input modality in the current decision-making cycle based on the key scenario constraint indicators and the intent alignment matrix. S4. Based on the dynamic credibility weight and the intent alignment matrix, construct the multimodal interaction decision graph of the electric wheelchair, and generate the anti-disturbance drive command of the electric wheelchair according to the multimodal interaction decision graph; S5. Based on the anti-disturbance drive command, execute the control actions of the electric wheelchair.

[0069] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program performing the following steps when executed by a processor: S1. By using multi-source heterogeneous sensors deployed on electric wheelchairs, extract multimodal physiological state vectors and environmental risk perception vectors corresponding to the current user and their environment; S2. Construct the intent alignment matrix between each input modality of the electric wheelchair based on the multimodal physiological state vector and the environmental risk perception vector; S3. Extract key scenario constraint indicators from the environmental risk perception vector, and determine the dynamic credibility weight of each input modality in the current decision-making cycle based on the key scenario constraint indicators and the intent alignment matrix. S4. Based on the dynamic credibility weight and the intent alignment matrix, construct the multimodal interaction decision graph of the electric wheelchair, and generate the anti-disturbance drive command of the electric wheelchair according to the multimodal interaction decision graph; S5. Based on the anti-disturbance drive command, execute the control actions of the electric wheelchair.

[0070] It should be noted that the functions or steps that can be implemented by the computer-readable storage medium or computer device described above can be referred to the relevant descriptions on the server side and client side in the foregoing method embodiments. To avoid repetition, they will not be described one by one here.

[0071] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0072] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0073] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention.

[0074] Finally, it should be noted that in the above embodiments, each embodiment can be combined with each other or independent. Deleting any one of them will not affect the technical implementation of other embodiments. The above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention.

Claims

1. A multimodal human-computer interaction control method for electric wheelchairs, characterized in that, The method includes: By deploying multi-source heterogeneous sensors on electric wheelchairs, multimodal physiological state vectors and environmental risk perception vectors corresponding to the current user and their environment are extracted. Based on the multimodal physiological state vector and the environmental risk perception vector, an intent alignment matrix is ​​constructed between the input modalities of the electric wheelchair. Key scenario constraint indicators are extracted from the environmental risk perception vector, and the dynamic credibility weight of each input modality in the current decision-making cycle is determined based on the key scenario constraint indicators and the intent alignment matrix. Based on the dynamic credibility weight and the intent alignment matrix, a multimodal interaction decision graph of the electric wheelchair is constructed, and based on the multimodal interaction decision graph, an anti-disturbance drive command for the electric wheelchair is generated. Based on the anti-disturbance drive command, the control actions of the electric wheelchair are executed.

2. The method for controlling an electric wheelchair with multimodal human-computer interaction as described in claim 1, characterized in that, Based on the key scenario constraint indicators and the intent alignment matrix, determine the dynamic credibility weight of each input modality within the current decision-making cycle, including: Based on the aforementioned key scenario constraint indicators, calculate the constraint risk value for each input mode; The mean of the intent correlation between each input modality and all other modalities is extracted from the intent alignment matrix to obtain the intent consistency support of the corresponding input modality. The constraint risk value is reverse normalized to obtain the scene safety adaptability of the corresponding input mode; Based on the intent consistency support and the scenario security adaptability, the dynamic credibility weight of each input modality is determined in the current decision cycle.

3. The method for controlling an electric wheelchair with multimodal human-computer interaction as described in claim 2, characterized in that, Based on the aforementioned key scenario constraint indicators, the constraint risk value for each input modality is calculated, including: Obtain the parsing instructions corresponding to each input mode within the current decision cycle, wherein the parsing instructions include control type and control parameters; Establish a mapping matrix between the control parameters and the key scenario constraint indicators to calculate the dynamic risk accumulation value of each input mode; Based on the dynamic risk accumulation value, the constraint risk value for each input mode is generated.

4. The multimodal human-computer interaction electric wheelchair control method as described in claim 3, characterized in that, Establishing a mapping matrix between the control parameters and the key scenario constraint indicators includes: The control parameters are projected onto a preset constraint parameter space to obtain the standardized constraint coordinates of each control parameter; Based on the key scenario constraint indicators, a dynamic feasible region is constructed in the constraint parameter space; Calculate the original safety margin from each of the standardized constraint coordinates to the boundary of the dynamic feasible region constraint; The original safety margin is weighted and scaled to generate a mapping matrix between the control parameters and the key scenario constraint indicators.

5. The multimodal human-computer interaction electric wheelchair control method as described in claim 1, characterized in that, Based on the multimodal interaction decision graph, disturbance-resistant drive commands for the electric wheelchair are generated, including: The node weights and edge connection strengths of the multimodal interaction decision graph are analyzed. Based on the node weights and edge connection strengths, the core control intent and auxiliary control signals in the multimodal interaction decision graph are identified. Based on the core control intent, the basic driving action sequence of the electric wheelchair is determined; Based on the auxiliary control signal and the edge connection strength, calculate the dynamic adjustment coefficient corresponding to the basic driving action sequence; After correcting the basic drive action sequence using the dynamic adjustment coefficient, the disturbance-resistant drive command of the electric wheelchair is obtained.

6. The method for controlling an electric wheelchair with multimodal human-computer interaction as described in claim 5, characterized in that, Based on the core control intent, the basic driving motion sequence of the electric wheelchair is determined, including: The core control intent is parsed into the basic control vector of the electric wheelchair; Retrieve candidate drive action sequences that match the basic control vector from a preset action instruction library; Based on the response priority of the core control intent, the target candidate sequence is selected from the candidate driving action sequences; Define the security dynamic constraints of the core control intent; Based on the safety dynamic constraints, the target candidate sequence is optimized to obtain the basic driving action sequence.

7. The multimodal human-computer interaction electric wheelchair control method as described in claim 5, characterized in that, Based on the auxiliary control signal and the edge connection strength, the dynamic adjustment coefficients corresponding to the basic driving action sequence are calculated, including: Extract the adjustment instruction components from the auxiliary control signal that correspond to each control dimension of the basic driving action sequence; Obtain the edge connection strength value between the modal node corresponding to the auxiliary control signal and the node corresponding to the core control intent; The dynamic adjustment coefficients corresponding to the basic driving action sequence are calculated based on the adjustment command components and the edge connection strength values.

8. The method for controlling an electric wheelchair with multimodal human-computer interaction as described in claim 1, characterized in that, The input modalities include: voice modal, visual gesture modal, and joystick modal.

9. The method for controlling an electric wheelchair with multimodal human-computer interaction as described in claim 1, characterized in that, The control action is performed by the electric wheelchair according to the anti-disturbance drive command, driving at least one actuator among its hub motor, steering servo, and lifting mechanism.

10. A multimodal human-computer interaction electric wheelchair control system, characterized in that, The system includes: The vector extraction module is used to extract multimodal physiological state vectors and environmental risk perception vectors corresponding to the current user and their environment through multi-source heterogeneous sensors deployed on the electric wheelchair. A matrix construction module is used to construct an intent alignment matrix between the input modalities of the electric wheelchair based on the multimodal physiological state vector and the environmental risk perception vector. The weight calculation module is used to extract key scenario constraint indicators from the environmental risk perception vector, and determine the dynamic credibility weight of each input modality in the current decision-making cycle based on the key scenario constraint indicators and the intent alignment matrix. The instruction generation module is used to construct a multimodal interaction decision graph of the electric wheelchair based on the dynamic credibility weight and the intent alignment matrix, and generate anti-disturbance drive instructions for the electric wheelchair according to the multimodal interaction decision graph. The control execution module is used to execute the control actions of the electric wheelchair based on the anti-disturbance drive command.

Images