An electric wheelchair control system and method based on a multimodal large model
By integrating information from a multimodal large model and combining information on the occupant status, vehicle operation, posture, and environment of the electric wheelchair, the system identifies the source of actions and outputs adaptive control commands, thus solving the problem of misoperation of electric wheelchairs in complex scenarios and improving control reliability and intelligence.
Patent Information
- Application Number
- CN202611298793.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-08-26
- Publication Date
- 2026-11-10
AI Technical Summary
Existing electric wheelchair control systems cannot comprehensively judge the source of movement and control risks based on the vehicle's real-time speed and changes in wheelchair posture, resulting in a high risk of misoperation. In particular, they cannot distinguish between user-initiated control actions, user-involuntary physiological movements, and passive swaying during wheelchair movement in complex scenarios.
Employing a multimodal large model, it integrates occupant status information, vehicle operation information, wheelchair posture information, driving environment information, and navigation and positioning information. By matching occupant action characteristics with preset control actions, and combining the direction and time of disturbance triggering, it identifies the type of action source and outputs corresponding control commands based on terminal access area and obstacle approach information, distinguishing between active control, passive disturbance, and abnormal risk actions.
It improves the reliability and intelligence of electric wheelchair control, avoids misjudgments caused by a single control signal, and ensures safe passage for occupants in complex environments.
Smart Images

Figure CN123067459A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of wheelchair control technology, and more specifically, to an electric wheelchair control system and method based on a multimodal large model. Background Technology
[0002] With the improvement of civil aviation passenger service quality and the advancement of smart airport construction, ground services for special passengers (such as elderly and disabled passengers) have become a key focus of airport service evaluation. In airport terminals, wheelchairs need to pass through densely populated areas such as passageways, security checkpoints, and boarding gates. Users may make involuntary movements due to head swaying, unstable posture, or nervous reactions. The control process of electric wheelchairs cannot rely solely on a single control signal or simple action matching. It also requires a comprehensive judgment of the current driving risk by combining the passenger's status, vehicle movement status, environmental perception information, and safety supervision information, so as to avoid the risk of misinterpreting involuntary movements as control commands.
[0003] Existing patent CN106909106A discloses a motion control method and device for an electric wheelchair. This method collects posture signals and determines actions based on posture and motion trajectory data to control the electric wheelchair. However, directly mapping posture and motion recognition results to wheelchair control signals fails to comprehensively assess the source of the action and control risks based on the vehicle's real-time speed and changes in wheelchair posture. Furthermore, it fails to distinguish between user-initiated actions, involuntary physiological movements, and passive swaying caused by road bumps, slope changes, and turning inertia during wheelchair movement. This leads to the risk of misoperation in complex scenarios, resulting in insufficient control reliability for the electric wheelchair.
[0004] Therefore, it is necessary to design an electric wheelchair control system and method based on a multimodal large model to solve the problems existing in the current technology. Summary of the Invention
[0005] In view of this, the present invention proposes an electric wheelchair control system and method based on a multimodal large model, which aims to solve the problem that the existing technology cannot comprehensively judge the source of motion and control risks based on the real-time speed of the vehicle and the changes in the wheelchair posture, and does not distinguish between the user's active control actions, the user's physiological involuntary actions and the passive swaying during the wheelchair's movement, resulting in insufficient reliability of electric wheelchair control.
[0006] In one aspect, this invention proposes a control method for electric wheelchairs based on a multimodal large model, comprising: Collect information on the occupant status, vehicle operation, wheelchair posture, driving environment, and navigation and positioning of the electric wheelchair; Based on the occupant status information, occupant action features are extracted and matched with preset control actions to determine candidate control actions. Based on the vehicle operation information and wheelchair posture information, the disturbance trigger direction and disturbance occurrence time are determined. The disturbance trigger direction and disturbance occurrence time are correlated and compared with the candidate control actions to determine the action source type of the candidate control actions. Based on the driving environment information and navigation positioning information, the terminal passage area, path deviation information and obstacle approach information of the electric wheelchair are identified, and the scene constraint state is determined. Based on the action source type and scene constraint state, the control command of the electric wheelchair is determined. When the action source type is an active control action, the crew operation capability level is determined based on the crew status information, and the passage safety boundary is determined based on the terminal passage area and path deviation information. Based on the matching result of the crew operation capability level and the passage safety boundary, the driving adjustment command, speed limit driving command or deceleration and steering command is output. When the action source type is a passive disturbance action, the candidate control action is masked based on the disturbance triggering direction, disturbance occurrence time, and scene constraint state, and a driving maintenance command or a driving deceleration command is output based on the scene constraint state. When the action source type is an abnormal risk action, a parking control command / remote request command / emergency stop control command is output based on the occupant's operational ability level, obstacle approach information, and terminal access area.
[0007] Furthermore, when determining candidate control actions, the process includes: determining the occupant's head turning direction, upper body tilting direction, movement amplitude, and facial orientation information based on the occupant's state information; determining the head turning direction and upper body tilting direction as the action direction; determining the movement amplitude as the action intensity information; determining the occupant's action characteristics based on the action direction, action intensity information, and facial orientation information; and determining the corresponding preset control action as the candidate control action when the occupant's action characteristics are consistent with the direction and amplitude conditions in the preset control action.
[0008] Furthermore, in determining the disturbance triggering direction and disturbance occurrence time, the process includes: determining the vehicle speed change, drive wheel speed difference, and drive current change based on the vehicle operation information; determining the vehicle body pitch change, vehicle body yaw change, and vehicle body roll change based on the wheelchair posture information; and when the vehicle speed change, drive wheel speed difference, drive current change, vehicle body pitch change, vehicle body yaw change, and / or vehicle body roll change are synchronous disturbances, the synchronization start time is determined as the disturbance occurrence time, and the front-to-back disturbance direction and / or left-to-right disturbance direction and / or lateral disturbance direction are determined based on the direction of the synchronous disturbance.
[0009] Furthermore, in determining the action source type of the candidate control action, the following steps are included: when the action occurrence time of the candidate control action is earlier than the disturbance occurrence time, and the action direction of the candidate control action is consistent with the control direction corresponding to the preset control action, the action source type is determined to be the active control action; when the action occurrence time of the candidate control action is later than or equal to the disturbance occurrence time, and the action direction of the candidate control action is consistent with or opposite to the disturbance triggering direction, the action source type is determined to be the passive disturbance action; when the candidate control action matches several preset control actions, and the action direction is inconsistent with both the current driving direction and the path direction, the action source type is determined to be the abnormal risk action.
[0010] Furthermore, when determining the scene constraint state, the process includes: determining the current position of the electric wheelchair in the terminal based on the navigation and positioning information, matching the current position with the area boundary to determine the terminal access area, determining the path deviation information based on the navigation and positioning information and the target access path, determining obstacle approach information based on the driving environment information, and determining the scene constraint state as a passage access state when the terminal access area is a passage area, the path deviation information is within the deviation range, and the obstacle approach information does not meet the approach conditions; otherwise, it is a restricted access state.
[0011] Furthermore, when determining the scene constraint state, the following are also included: the restricted access state includes an area restricted state, a path restricted state, and an obstacle restricted state. When the terminal access area is a security check area / boarding gate area / elevator area / ramp area / turning area / human confluence area, the restricted access state is determined to be the area restricted state; when the path deviation information is that the electric wheelchair deviates from the target access path and the deviation direction is not the target access area, the restricted access state is determined to be the path restricted state; when the obstacle approach information is that there is an obstacle in the forward / lateral / turning direction, the restricted access state is determined to be the obstacle restricted state.
[0012] Furthermore, determining the occupant's operational ability level includes: determining the occupant's age group, sitting posture, head movement, upper body posture, and gaze state based on the occupant's status information; when the occupant's age group is a low-risk age group, and the sitting posture, head movement, and gaze state all meet the stability condition, it is determined to be a first operational ability level; when the occupant's age group is an elderly age group, or when any of the sitting posture, head movement, and upper body swaying states does not meet the stability condition, it is determined to be a second operational ability level; when the occupant's gaze state or upper body posture is abnormal, it is determined to be a third operational ability level.
[0013] Furthermore, in determining the passage safety boundary, the process includes: determining an area control boundary based on the terminal passage area, determining an occupant control boundary based on the occupant's operational capability level, and superimposing the area control boundary and the occupant control boundary to determine the passage safety boundary.
[0014] Furthermore, when outputting wheelchair control commands, the following steps are included: when the action source type is the active control action, and the speed increment, steering angle, and action response time corresponding to the candidate control action are all within the traffic safety boundary, then the driving adjustment command is output; when the action source type is the active control action, and the corresponding speed increment / steering angle / action response time exceeds the traffic safety boundary, then the excess portion is truncated, and the speed limit driving command or deceleration steering command is output based on the scenario constraint state; when the action source type is the abnormal risk action, and there is an obstacle-restricted state / path-restricted state / third operational capability level, then a parking control command, a remote request command, or an emergency stop control command is output.
[0015] Compared with existing technologies, the advantages of this invention are as follows: By collecting occupant status information, vehicle operation information, wheelchair posture information, driving environment information, and navigation positioning information, the control basis of the electric wheelchair is no longer limited to a single posture action. Instead, it incorporates occupant status, vehicle movement, wheelchair posture, surrounding environment, and current position into the judgment, avoiding misjudgment of control due to a single control signal. Matching occupant action characteristics with preset control actions can filter out actions corresponding to the control intent. By combining vehicle operation information and wheelchair posture information, the direction of disturbance triggering and the time of disturbance occurrence can be determined, distinguishing between active control actions, passive disturbance actions, and abnormal risk actions. This avoids misjudging road bumps, slope changes, turning inertia, or involuntary occupant swaying as valid control commands. By defining the constraints of the scenario, the control commands can be adjusted according to different terminal environments such as passageways, security checkpoints, boarding gates, and densely populated areas. Furthermore, the control can be restricted based on the passenger's own condition and the area they are in, reducing erroneous actions caused by vehicle movement. Based on the passenger's operational ability level, obstacle proximity information, and the terminal's access area, corresponding control commands are output, improving the reliability and intelligence of electric wheelchair control.
[0016] On the other hand, this application also provides an electric wheelchair control system based on a multimodal large model, for applying the above-mentioned electric wheelchair control method based on a multimodal large model, including: The data acquisition unit is configured to collect information on the occupant status of the electric wheelchair, vehicle operation information, wheelchair posture information, driving environment information, and navigation and positioning information. The analysis unit is configured to extract occupant action features based on the occupant status information, match the occupant action features with preset control actions to determine candidate control actions, determine the disturbance triggering direction and disturbance occurrence time based on the vehicle operation information and wheelchair posture information, and correlate and compare the disturbance triggering direction and disturbance occurrence time with the candidate control actions to determine the action source type of the candidate control actions. The control unit is configured to identify the terminal access area, path deviation information and obstacle approach information of the electric wheelchair based on the driving environment information and navigation positioning information, determine the scene constraint state, and determine the control command of the electric wheelchair based on the action source type and scene constraint state; The control unit is further configured to, when the action source type is an active control action, determine the occupant's operational capability level based on the occupant status information, determine the passage safety boundary based on the terminal passage area and path deviation information, and output a driving adjustment command, a speed limit driving command, or a deceleration and steering command based on the matching result of the occupant's operational capability level and the passage safety boundary. The control unit is further configured to, when the action source type is a passive disturbance action, to shield the candidate control action based on the disturbance triggering direction, disturbance occurrence time, and scene constraint state, and to output a driving maintenance command or a driving deceleration command based on the scene constraint state. The control unit is also configured to output a stop control command / remote request command / emergency stop control command based on the occupant's operational capability level, obstacle proximity information, and terminal access area when the action source type is an abnormal risk action.
[0017] It is understandable that the above-mentioned electric wheelchair control system and method based on a multimodal large model have the same beneficial effects, and will not be elaborated further here. Attached Figure Description
[0018] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0019] Figure 1 A flowchart illustrating an electric wheelchair control method based on a multimodal large model, provided for an embodiment of the present invention; Figure 2 A flowchart of an electric wheelchair control based on a multimodal large model is provided for an embodiment of the present invention; Figure 3 This is a functional block diagram of an electric wheelchair control system based on a multimodal large model, provided for an embodiment of the present invention. Detailed Implementation
[0020] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0021] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.
[0022] See Figure 1-2 As shown in some embodiments of this application, an electric wheelchair control method based on a multimodal large model includes: S100: Collects information on the occupant status, vehicle operation, wheelchair posture, driving environment, and navigation and positioning of the electric wheelchair. S200: Extract occupant action features based on occupant status information, match occupant action features with preset control actions to determine candidate control actions, determine disturbance trigger direction and disturbance occurrence time based on vehicle operation information and wheelchair posture information, correlate and compare disturbance trigger direction and disturbance occurrence time with candidate control actions to determine the action source type of candidate control actions; S300: Based on driving environment information and navigation positioning information, identify the terminal passage area, path deviation information and obstacle approach information of electric wheelchairs, determine the scene constraint state, and determine the control commands of electric wheelchairs based on the action source type and scene constraint state; S400: When the action source type is active control action, determine the crew's operational capability level based on the crew status information, determine the passage safety boundary based on the terminal passage area and path deviation information, and output driving adjustment command, speed limit driving command or deceleration and turning command based on the matching result of crew operational capability level and passage safety boundary. S500: When the action source type is a passive disturbance action, the candidate control action is masked based on the disturbance trigger direction, disturbance occurrence time, and scene constraint state, and the driving command or deceleration driving command is output based on the scene constraint state. S600: When the action source type is an abnormal risk action, the stop control command / remote request command / emergency stop control command is output based on the crew's operational ability level, obstacle approach information and terminal access area.
[0023] Specifically, existing motion control systems tend to directly interpret involuntary occupant movements and passive swaying caused by vehicle bumps as control commands. Sufficient state data is needed before generating control commands to avoid the risk of erroneous commands. Occupant state information can be collected through devices such as occupant-facing cameras, posture sensors, seat pressure sensors, armrest contact detectors, voice or interactive terminals, etc. This information includes data such as head turn, upper body tilt, range of motion, facial orientation, sitting posture, and gaze direction. Vehicle operation information can be collected through devices such as motor drivers, encoders, and current detectors, including current vehicle speed, acceleration / deceleration changes, left and right drive wheel speeds, drive current, braking status, and driving speed. Information such as direction and wheelchair posture data can be collected through devices such as inertial measurement units, gyroscopes, and accelerometers. Wheelchair posture data includes pitch, yaw, and roll changes. Driving environment information can be collected through devices such as forward-facing cameras, side cameras, depth cameras, LiDAR, ultrasonic sensors, or millimeter-wave radar. Driving environment information includes information such as passageway conditions, personnel distribution, obstacle locations, obstacle distances, passage width, and turning space. Navigation and positioning information can be collected through devices such as GPS positioning, laser SLAM, and BeiDou positioning. Navigation and positioning information includes the electric wheelchair's current position, target path, target boarding gate or service point location, current driving direction, and path deviation distance. Features related to control intentions, such as direction of movement, amplitude of movement, duration of movement, and facial orientation, are extracted from the occupant's status information. Examples include: head turning to the left, upper body tilting to the right, amplitude of movement reaching preset conditions, and facial orientation consistent with the driving direction. The system matches occupant movement characteristics with preset control actions. These preset control actions are pre-defined action types that correspond to the control intentions of the electric wheelchair, such as forward, backward, left turn, right turn, deceleration, and stopping. Preset control actions can be set using VLA and VLN large models. When an occupant's movement characteristics match the conditions of a preset control action, a candidate control action is determined. A candidate control action does not necessarily represent a control intention actively issued by the occupant; its source must be determined by combining vehicle operation information and wheelchair posture information. The disturbance trigger direction indicates the direction of swaying caused by changes in the vehicle's own motion state or body posture, such as pitching forward or backward due to crossing a threshold. The disturbance occurrence time indicates the point in time when significant changes begin to occur in the vehicle operation information or wheelchair posture information, such as a sudden decrease in vehicle speed or a large change in the speed difference between the drive wheels. By using information from both the vehicle and wheelchair sides to track whether there is vehicle disturbance before or after the occupant's actions, if the candidate control action appears immediately after the vehicle swaying, the action may be caused by the occupant's passive swaying rather than an active control action. If the candidate control action appears before the vehicle disturbance, it may be an active control intention issued by the occupant.By correlating and comparing the disturbance trigger direction and occurrence time with candidate control actions, the source type of the candidate control actions is determined. Action source types include active control actions, passive disturbance actions, and abnormal risk actions. Active control actions are those initiated by the occupant; passive disturbance actions are involuntary actions caused by road bumps, slope changes, turning inertia, or vehicle swaying; and abnormal risk actions are those whose direction, amplitude, combination, or occurrence conditions are inconsistent with the control logic, and are related to abnormal occupant posture, stress response, or erroneous actions. By separating the processing of whether an action resembles a control action from whether the action truly originates from the occupant's active intention, the risk of directly mapping posture and action recognition results to control signals is avoided, ensuring the reliability and accuracy of electric wheelchair control.
[0024] Specifically, even if a candidate control action is confirmed as an active control action, it cannot be executed using the same control method in all terminal scenarios. For example, while passageways allow for driving adjustments, security checkpoints, boarding gates, elevator entrances, ramps, turning areas, and pedestrian confluence areas require speed reduction and steering control. The terminal passage area refers to the functional area where the electric wheelchair is located, which is determined through navigation and positioning information, such as passageway areas and security checkpoints. Path deviation information indicates the deviation relationship between the electric wheelchair's current position and the target passageway, including deviation distance and direction. Obstacle approach information indicates the proximity of obstacles to the electric wheelchair in the driving environment, including forward obstacle distance, lateral obstacle distance, obstacle position inside the turn, and the direction of approach for the person. The scene constraint state represents the current control restriction state determined by the terminal access area, path deviation information, and obstacle approach information. After determining the scene constraint state, the action source and scene constraints are used together as the control basis. When the action source type is an active control action, the occupant's operational ability level is determined based on the occupant's status information. The occupant's operational ability level represents the stability of the occupant when controlling the electric wheelchair, which is determined based on the occupant's age group, sitting posture, etc. The passage safety boundary is determined based on the terminal access area and path deviation information. The passage safety boundary represents the control range that the wheelchair is allowed to perform in the current area and current path state. Based on the matching results of the occupant's operational ability level and the passage safety boundary, the system outputs driving adjustment instructions, speed limit driving instructions, or deceleration and steering instructions. Among them, the driving adjustment instruction indicates that the candidate control action can be converted into forward movement, steering, or speed adjustment within the safety boundary. The speed limit driving instruction indicates that driving is allowed to continue but the speed increment or maximum speed is limited. The deceleration and steering instruction indicates that the speed is reduced and the steering range is limited before the steering is executed. Even if the action issued by the occupant is an active control action, the control intensity will be limited and adjusted according to the occupant's own condition and the terminal scene to avoid elderly occupants or occupants with unstable postures triggering too fast or too large control actions in complex areas.
[0025] Understandably, when the source of the action is a passive disturbance, candidate control actions are masked. This means that instead of converting them into active control commands like steering or acceleration, the system outputs a maintain-driving command or a deceleration command based on the scenario constraints. The maintain-driving command maintains the current direction and speed control, while the deceleration command reduces the current speed but does not change direction according to the candidate control actions. When the electric wheelchair passes over thresholds, ramps, or crowded areas, the occupant is prone to head or upper body swaying due to vehicle movement. If candidate control actions are not masked, this could lead to accidental steering or acceleration. When the source of the action is an abnormal risk action, parking control commands, remote request commands, or emergency stop control commands are output based on the occupant's operational ability level, obstacle proximity information, and the terminal's passageway. Parking control commands cause the electric wheelchair to decelerate and stop under control. Remote request commands send a takeover request to a remote monitoring terminal or ground staff, allowing remote personnel to confirm or take over based on video, location, and status. Emergency stop control commands immediately cut off the driving output and trigger braking when there is a high risk. In other words, if the passenger's operational ability level is low but the surrounding obstacles have not reached the level of emergency approach, a stop control command or a remote request command will be output first. If the obstacle is already approaching, or the electric wheelchair is in a motion-sensitive area such as a security checkpoint, boarding gate, or a pedestrian confluence area, a remote request command or an emergency stop control command will be output. If the passenger's condition is abnormal and an obstacle is approaching, an emergency stop control command will be output directly. This transforms the control of the electric wheelchair from a single action matching to a continuous process of passenger action recognition, disturbance source judgment, terminal scene constraints, passenger operational ability matching, and hierarchical output of control commands. This preserves the passenger's ability to actively control the wheelchair while also enabling adaptive control of the electric wheelchair in passive shaking and complex terminal passage scenarios. This ensures both the level of intelligence in controlling the electric wheelchair and the safety of special passengers when passing through the terminal.
[0026] In some embodiments of this application, determining candidate control actions includes: determining the occupant's head turning direction, upper body tilting direction, action amplitude, and facial orientation information based on occupant state information, and determining the head turning direction and upper body tilting direction as action direction, and the action amplitude as action intensity information; determining occupant action characteristics based on action direction, action intensity information, and facial orientation information; and determining the corresponding preset control action as a candidate control action when the occupant action characteristics are consistent with the direction and amplitude conditions in the preset control action.
[0027] Specifically, the direction of head turn is the direction in which the occupant's head turns to the left, right, forward, or backward relative to the seat reference direction. For example, if the occupant's head turns from facing forward to the left, the direction of head turn is leftward. The direction of upper body tilt is the direction in which the occupant's shoulders, torso, or back tilt relative to the seat back or the vehicle's centerline. For example, if the occupant's upper body shifts to the right, the direction of upper body tilt is rightward. The range of motion is the degree to which the occupant's head or upper body deviates from the seat reference position. For example, the range of motion formed by the head turning to the left at a certain angle or the upper body tilting forward at a certain angle. Facial orientation information refers to the direction the occupant's face is facing or their line of sight. The direction of head turn and upper body tilt are defined as the movement direction. Movement direction refers to the primary direction of change in the occupant's body movement in space, such as left, right, forward, or backward. Individual head movements are easily influenced by the occupant's observation of the environment, communication with others, tension reactions, or unconscious swaying. Upper body tilt direction provides auxiliary information for judging the overall posture change of the occupant. It is more suitable to judge candidate control actions when the head turn and upper body tilt directions are consistent or conform to a preset combination of control actions. The amplitude of the movement is defined as the movement intensity information, which is the degree to which the occupant's movement control is triggered. Occupant movement characteristics are movement judgments formed by combining movement direction, movement intensity information, and facial orientation information. For example, a left-turn-related occupant movement characteristic can be formed by: head turning to the left, slight upper body tilt to the left, movement intensity reaching the left-turn trigger condition, and facial orientation consistent with the left-side observation direction. When the occupant's movement characteristics match the direction and amplitude conditions in the preset control actions, the corresponding preset control action is determined as a candidate control action. Preset control actions can be set using VLA and VLN large models combined with software such as Vicon Nexus, Blender, and MediaPipe, and specifically include forward, backward, left turn, and right turn actions. The direction condition is the required direction of the action in the preset control action. For example, a left turn requires the occupant's head to turn or upper body to tilt to meet the left-direction condition, and a right turn requires the right-direction condition. The amplitude condition is the required amplitude of the action to reach the corresponding trigger range. For example, if the amplitude is below the trigger range, no candidate control action is generated; only when the amplitude reaches the trigger range is it allowed to proceed to the next judgment. The trigger range can be dynamically adjusted according to different users' limbs and body postures, and is not specifically limited here. Candidate control actions retain the ability for occupants to control the electric wheelchair through posture movements, while avoiding directly converting single, occasional, slight, or ambiguous movements into control commands, further ensuring the level of intelligence in controlling the electric wheelchair.
[0028] In some embodiments of this application, determining the disturbance triggering direction and the disturbance occurrence time includes: determining the vehicle speed change, the difference in drive wheel speeds, and the change in drive current based on vehicle operation information; determining the vehicle pitch change, the vehicle yaw change, and the vehicle roll change based on wheelchair posture information; when the vehicle speed change, the difference in drive wheel speeds, the change in drive current, the vehicle pitch change, the vehicle yaw change, and / or the vehicle roll change are synchronous disturbances, the synchronization start time is determined as the disturbance occurrence time, and the front-to-back disturbance direction and / or the left-to-right disturbance direction and / or the lateral disturbance direction are determined based on the direction of the synchronous disturbance.
[0029] Specifically, the speed change is the magnitude of the speed change of the electric wheelchair between adjacent sampling moments. The drive wheel speed difference is the difference in speed between the left and right drive wheels. For example, when the electric wheelchair turns, encounters obstruction on one side of the wheel, or traverses uneven ground on one side, the speeds of the left and right drive wheels will be inconsistent. The drive current change is the magnitude of the change in drive motor current per unit time. For example, when the wheelchair goes uphill, crosses a small obstacle, encounters wheel obstruction, or experiences a sudden change in road friction, the drive motor needs to output greater driving force, resulting in a significant increase in drive current. Vehicle operation information only indicates changes in the wheelchair's speed, motor, and wheel status. It is necessary to use wheelchair posture information to confirm whether the vehicle itself has undergone a change in posture, thereby determining whether the changes in the electric wheelchair's operation have been transmitted to the point of affecting the occupant's body sway. The vehicle pitch change is the magnitude of the vehicle's pitching or tilting along the forward and backward direction. For example, when the electric wheelchair goes uphill, downhill, crosses a threshold, or encounters a bump with its front wheels, the vehicle will pitch forward and backward. The yaw rate is the magnitude of the electric wheelchair's left-right swaying or steering yaw around the vertical direction. For example, when the electric wheelchair turns, avoids pedestrians, or experiences a sudden increase in the speed difference between the left and right wheels, the wheelchair will yaw. The tilt rate is the magnitude of the electric wheelchair's tilting along the left and right directions. For example, when one wheel traverses uneven ground, a laterally uneven slope, or when the inertia is significant during a turn, the wheelchair will tilt. When the changes in vehicle speed, drive wheel speed difference, drive current, vehicle pitch, yaw, and / or tilt are synchronous disturbances, the synchronization start time is determined as the disturbance occurrence time. Synchronous disturbances occur simultaneously in both vehicle operation information and wheelchair posture information, with a corresponding relationship between the changes. For example, a decrease in vehicle speed coupled with an increase in vehicle pitch indicates the wheelchair may be traversing a threshold, ramp, or obstacle protrusion; an increase in the difference in drive wheel speeds coupled with an increase in vehicle yaw indicates the wheelchair may be turning, experiencing unilateral obstruction, or needing to avoid an obstacle; an increase in drive current coupled with an increase in vehicle roll indicates the wheelchair may be traveling on a slope, on a partially uneven surface, or in an environment with varying lateral resistance. Identifying synchronous disturbances based on multiple changes, rather than a single data change, helps avoid control errors. For example, slight fluctuations in vehicle speed may simply be normal speed adjustment, a short-term increase in drive current may be merely motor response, and slight changes in vehicle posture may simply be sensor noise. Only when vehicle operation information and wheelchair posture information change simultaneously in time, and the directions of change correspond, will it be considered a disturbance affecting the occupant's posture. The synchronization start time is the point in time when the state of the synchronous disturbance changes significantly. For example, when the front wheel of the wheelchair just touches the threshold, the drive current first increases, and then the speed change and the pitch change of the vehicle body occur simultaneously. The time when the synchronous changes start together is determined as the disturbance occurrence time.The forward / backward disturbance direction refers to the disturbance primarily acting along the wheelchair's forward or backward direction; the left / right disturbance direction refers to the disturbance primarily acting along the wheelchair's left / right swaying direction; and the lateral disturbance direction refers to the disturbance primarily acting along the wheelchair's left / right tilting direction. If the synchronous disturbance is mainly formed by changes in vehicle speed and vehicle pitch, it can be determined as a forward / backward disturbance direction. If the synchronous disturbance is mainly formed by the difference in drive wheel speeds and changes in vehicle yaw, it can be determined as a left / right disturbance direction. If the synchronous disturbance is mainly formed by changes in drive current, changes in vehicle tilt, and changes in resistance on one side of the wheel, it can be determined as a lateral disturbance direction. If changes in multiple directions exist simultaneously, it can be determined as a combination of forward / backward disturbance direction and / or left / right disturbance direction and / or lateral disturbance direction. Establishing vehicle disturbance criteria from vehicle operation information and wheelchair posture information separates road bumps, slope changes, turning inertia, and passive swaying caused by vehicle resistance from the active control of the occupant, further ensuring the level of intelligence in controlling the electric wheelchair.
[0030] In some embodiments of this application, determining the action source type of a candidate control action includes: when the occurrence time of the candidate control action is earlier than the occurrence time of the disturbance, and the direction of the candidate control action is consistent with the control direction corresponding to the preset control action, the action source type is determined to be an active control action; when the occurrence time of the candidate control action is later than or equal to the occurrence time of the disturbance, and the direction of the candidate control action is consistent with or opposite to the direction of the disturbance triggering, the action source type is determined to be a passive disturbance action; when the candidate control action matches several preset control actions, and the direction of the action is inconsistent with both the current driving direction and the path direction, the action source type is determined to be an abnormal risk action.
[0031] Specifically, the time of action occurrence is the time when the preset directional and amplitude conditions of the control action are first met. The control direction is the wheelchair movement direction corresponding to the preset control action; for example, a left turn corresponds to a left turn control direction, and a forward movement corresponds to a forward movement control direction. If the occupant's action occurs first, and the vehicle disturbance occurs afterward, it indicates that the action was not caused by vehicle bumps or body swaying. Furthermore, if the direction of the action is consistent with the control direction corresponding to the preset control action, it indicates that the action has a clear control intent and is thus confirmed as an active control action. Conversely, if the direction of the action is consistent with the disturbance trigger direction, it means that the vehicle experiences a front-back, left-right, or lateral disturbance first, and the occupant's body sways in the same direction as the disturbance. For example, after a wheelchair experiences a front-back disturbance while traversing a ramp, the occupant's upper body leans forward or backward. If the direction of the action is opposite to the disturbance trigger direction, it means that the occupant sways in the opposite direction due to inertia or self-balancing after the vehicle sways. For example, after a wheelchair experiences a left-right disturbance while turning, the occupant's body shifts to the outside of the turn. If a vehicle disturbance occurs first, followed by an occupant's action, and the two actions correspond mechanically in the same or opposite directions, then although the candidate control action formally matches the preset control action, it is a passive sway caused by road bumps, turning inertia, slope changes, or vehicle obstruction, and therefore cannot directly control the electric wheelchair. Matching several preset control actions means that the occupant's actions simultaneously meet the triggering conditions for multiple control actions. For example, satisfying both steering and deceleration actions, or simultaneously triggering multiple action rules for forward tilting, side tilting, and facial turning. The current travel direction is the actual current direction of movement of the electric wheelchair, and the path direction is the forward or turning direction required by the target travel path in the navigation and positioning information. Normal active control should be interpretable in relation to the current driving direction or path direction. However, if an action triggers multiple preset control actions simultaneously, and the direction of the action does not conform to either the current driving direction or the path direction, then the action originates from situations such as passenger tension, physical instability, or sudden discomfort. Before the action causes the wheelchair to move erroneously, it is identified as an abnormal risk action. This forms a pre-control mechanism from action matching to action source determination, avoiding control risks caused by the inability to identify active and inactive actions, and further ensuring the level of intelligence in controlling electric wheelchairs.
[0032] In some embodiments of this application, determining the scene constraint state includes: determining the current position of the electric wheelchair in the terminal based on navigation and positioning information, matching the current position with the area boundary to determine the terminal access area, determining path deviation information based on navigation and positioning information and target access path, determining obstacle approach information based on driving environment information, and determining the scene constraint state as a passage access state when the terminal access area is a passage area, the path deviation information is within the deviation range, and the obstacle approach information does not meet the approach conditions; otherwise, it is a restricted access state.
[0033] Specifically, the current location is the real-time location of the electric wheelchair in the terminal. The current location is matched with the area boundary, which is the boundary line or coordinate range of different functional areas in the terminal in advance, such as the boundary of the passage area, the boundary of the security check area, the boundary of the boarding gate area, and the boundary of the elevator entrance area. The terminal access area is determined by judging which area boundary the current location falls into. The terminal access area refers to the specific functional area of the terminal where the electric wheelchair is currently passing. Passageways allow the wheelchair to travel along the target path. Security checkpoints, boarding gates, elevator entrances, and ramps are areas where queuing, stopping, turning, passenger confluence, or space narrowing occur. The target path is the route the electric wheelchair should take based on the passenger's service tasks, current location, and target boarding gate or service point. Path deviation information indicates the deviation of the electric wheelchair's current position from the target path. For example, if the electric wheelchair deviates towards the security checkpoint barrier, boarding gate seating area, wall, queuing crowd, or a non-target passageway, it means the path deviation information has exceeded the acceptable range, resulting in a path deviation error. Obstacle approach information is determined based on the driving environment. Obstacle approach information refers to the proximity of an obstacle to the electric wheelchair in front, to the side, or to the inside of a turn. For example, a passenger suddenly stopping ahead, a suitcase approaching from the side, or the presence of a pillar or seat on the inside of a turn all constitute obstacle approach information. The proximity range can be dynamically adjusted based on the electric wheelchair's braking distance and size. Scene constraint states refer to the restrictions imposed on wheelchair control actions by the current terminal scene. Deviation range refers to the safe deviation range allowed for the electric wheelchair on both sides of the target passageway. This range is determined based on the width of the electric wheelchair and the width of the terminal passageway. For example, a distance is reserved on each side of the centerline of the target passageway. If the lateral distance between the current position of the electric wheelchair and the centerline of the target passageway does not exceed this distance, the path deviation information is considered to be within the deviation range. If the lateral distance exceeds this distance, or if the deviation direction points to non-target passageways such as barriers, walls, seating areas, security equipment, or boarding gate queuing areas, the path deviation information is considered to be outside the deviation range. Approach conditions are when the distance to obstacles, the direction of approach, or the relative motion state meets the conditions requiring restrictions on wheelchair control. Passageway status indicates that the current area, path, and obstacle conditions all meet normal driving conditions, and driving adjustment commands can be executed within the safe passage boundary. If the terminal access area is not a passageway, or the path deviation information exceeds the deviation range, or the obstacle approach information meets the approach conditions, it is determined to be a restricted access state. The restricted access state indicates that there are area, pedestrian flow, path or obstacle restrictions in the current control environment. This avoids directly controlling the wheelchair based solely on the occupant's actions, thus enabling differentiated control of electric wheelchairs in different scenarios such as passageways, security checkpoints, boarding gates, pedestrian intersections and obstacle approach, thereby ensuring the level of intelligence in controlling electric wheelchairs.
[0034] In some embodiments of this application, when determining the scene constraint state, the following further applies: the restricted access state includes an area restricted state, a path restricted state, and an obstacle restricted state. When the terminal access area is a security check area / boarding gate area / elevator area / ramp area / turning area / human confluence area, the restricted access state is determined to be an area restricted state. When the path deviation information is that the electric wheelchair deviates from the target access path and the deviation direction is not the target access area, the restricted access state is determined to be a path restricted state. When the obstacle approach information is that there is an obstacle in the forward / lateral / turning direction, the restricted access state is determined to be an obstacle restricted state.
[0035] Specifically, restricted access is a state in which an electric wheelchair is not suitable for regular driving adjustments according to the normal passage conditions. Restricted access is not the result of a single source of risk. In the terminal, an electric wheelchair may need to reduce control response due to entering a special area, or it may need to make corresponding control corrections due to deviating from the target passage path. It may also need to be restricted from continuing to move due to obstacles in front or to the side. Area restricted access is a situation in which an electric wheelchair is in an area of the terminal with strict requirements for speed, turning or stopping. Path restricted access is a situation in which the electric wheelchair's current position has deviated from the target passage path, and the deviation may cause the wheelchair to enter an area that should not be entered. Obstacle restricted access is a situation in which there are people, luggage, isolation facilities, seats, walls or equipment that need to be avoided in the current direction of travel, to the side or inside the turning direction of the electric wheelchair. When the terminal's access areas are security checkpoints, boarding gate areas, elevator areas, ramp areas, turning areas, or areas where people converge, the restricted access status is defined as an area restricted status. Among these, the security checkpoint is an area with densely distributed passenger queues, security equipment, and barriers; the boarding gate area is an area where passengers wait, queue to board, and staff frequently operate; the elevator area is an area where people's entry and exit directions are complex and there are many short stops; the ramp area is an area where electric wheelchairs are prone to speed changes and body pitch changes; the turning area is an area where wheelchairs need to change direction and turning inward can easily create blind spots; and the area where people converge is an area where passengers cross laterally, make temporary stops, or move a lot of luggage. These areas are designated as restricted areas. Even if the candidate control action is an active control action, it will not be executed directly according to the control method of ordinary channels. Instead, it provides the data basis for subsequent execution of speed limit driving commands, deceleration and steering commands, or remote request commands. Secondly, when the path deviation information indicates that the electric wheelchair has deviated from the target passage path, and the deviation direction is a non-target passage area, the restricted passage state is determined to be a path restricted state. Non-target passage areas are areas that do not belong to the current service task's planned route, such as the inside of the barrier, the seating area, the wall, or other areas not corresponding to the current passage path. When the occupant actively issues a steering action, but the steering direction will cause the electric wheelchair to deviate into a non-target passage area, the steering action will not be executed directly. Instead, the risk will be reduced by deceleration and steering, stopping, or remote request.When the obstacle approach information indicates an obstacle is present in the forward, lateral, or turning direction, the restricted passage state is determined to be an obstacle-restricted state. A forward obstacle is a passenger, luggage, wall, seat, queuing railing, or service facility in front of the electric wheelchair in its current direction of travel. A lateral obstacle is a person or object within a safe distance on either side of the electric wheelchair. A turning obstacle is an obstacle that may be swept or squeezed by the vehicle body when the electric wheelchair is preparing to turn left or right. When the obstacle has already affected the travel space, further acceleration or large-angle turning is restricted. The system incorporates area restrictions, path deviation, and obstacle approach in the complex terminal environment into the control judgment, so that the electric wheelchair no longer simply executes control based on the occupant's actions, but first determines the cause of the restriction, and then selects different control methods such as speed limit, deceleration, and turning based on the cause of the restriction. This ensures that the control of the electric wheelchair meets the safety needs of special passengers in the terminal and further improves the intelligence level of electric wheelchair control.
[0036] In some embodiments of this application, determining the occupant's operational capability level includes: determining the occupant's age group, sitting posture, head movement, upper body posture, and gaze state based on occupant status information; when the occupant's age group is a low-risk age group, and the sitting posture, head movement, and gaze state all meet the stability condition, it is determined to be a first operational capability level; when the occupant's age group is an elderly age group, or when any of the sitting posture, head movement, and upper body swaying states does not meet the stability condition, it is determined to be a second operational capability level; when the occupant's gaze state or upper body posture is abnormal, it is determined to be a third operational capability level.
[0037] Specifically, since occupants of electric wheelchairs are typically elderly, disabled, or have mobility issues, the same candidate control action may not have the same safety implications for different occupants. For example, a young occupant with a stable posture making a slight turning motion will respond normally, while an elderly occupant, an occupant with an unstable posture, or an occupant with abnormal gaze may not be making a specific control intention, or their tolerance for changes in speed or direction may be weaker. Therefore, the occupant's operational ability level must be determined before issuing wheelchair control commands. The age range for occupants is determined based on occupant service registration information and identity information. The low-risk age range is set at 5 to 59 years old, where occupants typically have good reaction and posture maintenance abilities. The advanced age range is set at 60 years old and above, where occupants are older and require a more conservative control strategy. The sitting posture refers to the positional relationship between the occupant's buttocks, back, shoulders, and the seat, including whether the occupant is sitting upright, whether they are leaning to one side, whether their back is against the backrest, and whether their body is sliding. The head movement status refers to whether the occupant's head movements are stable and continuous, which can be determined based on the direction of head turning, the range of motion, the speed of change in motion, and the recovery of motion. The upper body status refers to the posture of the occupant's shoulders, torso, and back, including whether the upper body is leaning forward, backward, sideways, or continuously swaying. Among these, upper body swaying is one type of upper body status, indicating that the occupant's upper body repeatedly deviates from the normal sitting posture within a short period of time. The specific time limit can be dynamically adjusted according to the occupant's body condition; generally, the shorter the time span, the better the occupant's physical discomfort. The occupant's gaze state is the general direction in which the occupant's face or line of sight is directed. When the occupant's age group is in the low-risk age group, and the sitting posture, head movement, and occupant gaze state all meet the stability conditions, it is determined to be the first level of operational ability. The stability conditions are that the sitting posture does not show any deviation or slippage, the head movement does not show any continuous shaking or large swinging, and the occupant gaze state does not show any continuous deviation from the driving direction or cannot be recognized. When the occupant is elderly, or if any of their sitting posture, head movement, or upper body swaying does not meet the stability criteria, they are classified as having a second operational ability level. This level indicates that the occupant has age-related risks or localized postural instability, but it has not yet reached a significant degree of abnormality. When the occupant's gaze or upper body posture is abnormal, they are classified as having a third operational ability level. This level indicates that the occupant is currently experiencing attention deviance, physical instability, or an inability to reliably control the wheelchair. The first and second operational ability levels can be converted to the third operational ability level, or vice versa. This ensures that the electric wheelchair does not have a fixed control intensity based on a single identity or action, but rather dynamically adjusts the control boundaries according to the occupant's current physical state. This allows active control actions, passive disturbance actions, and abnormal risk actions to receive different control responses in different occupant states, further enhancing the intelligence of electric wheelchair control.
[0038] In some embodiments of this application, determining the passage safety boundary includes: determining the area control boundary based on the terminal passage area, determining the crew control boundary based on the crew's operational capability level, and superimposing the area control boundary and the crew control boundary to determine the passage safety boundary.
[0039] Specifically, the area control boundary is the control limit range determined by the current passage area of the terminal where the electric wheelchair is located. This includes data such as maximum permissible speed, permissible acceleration, single turning angle, turning response speed, and minimum stopping distance. For example, when the terminal passage area is a corridor, the passage space is relatively continuous, and the flow of people is relatively stable. The area control boundary allows for relatively smooth normal driving and moderate turning. When the terminal passage area is a security checkpoint, boarding gate area, elevator area, or area where people converge, there are more instances of queuing, stopping, lateral movement, and temporary avoidance. In this case, the area control boundary tightens the maximum permissible speed and reduces the single turning angle. When the terminal passage area is a ramp area, wheelchairs are prone to pitch and speed changes. The area control boundary limits acceleration and prioritizes deceleration. When the terminal passage area is a turning area, there may be blind spots on the inside of the wheelchair's turning radius. The area control boundary restricts the amount of wheelchair control based on the terminal passage area. Even within the same terminal's access area, different occupants have varying abilities to adapt to changes in speed, steering, and control response. For example, occupants at the first operational ability level have stable postures, head movements, and normal gaze, and can tolerate relatively normal driving adjustments. Occupants at the second operational ability level are elderly, have slightly unstable postures, or unstable head movements, requiring reduced speed increments, smaller steering angles, or longer response times. Occupants at the third operational ability level have abnormal gaze or upper body conditions, making them unsuitable for further drastic driving adjustments; in this case, parking control commands, remote request commands, or emergency stop control commands will be prioritized. The occupant control boundary is the control limit range determined by the occupant's operational ability level, specifically including the maximum permissible speed, the maximum permissible steering angle, and the response time. For example, at the first operational capability level, the occupant control boundary allows candidate control actions to be converted into driving adjustment commands within the area control boundary. At the second operational capability level, the occupant control boundary reduces the speed increment corresponding to the candidate control actions and compresses the steering angle, extending the action response time to prevent the electric wheelchair from suddenly moving due to slight swaying or slow reaction of the occupant. At the third operational capability level, the occupant control boundary prohibits acceleration and large-angle steering, and limits the control results to parking control commands, remote request commands, or emergency stop control commands.The area control boundary and the occupant control boundary are superimposed. For example, if the area control boundary allows a lower maximum speed, but the occupant control boundary requires a further reduction in speed due to the occupant's second operational capability level, then the passage safety boundary uses a lower maximum speed; if the area control boundary allows normal turning, but the occupant control boundary requires a limit on the turning angle due to unstable head movements, then the passage safety boundary uses a smaller single turning angle; if the area control boundary requires remote requests for elevator access areas, but the occupant control boundary does not require remote requests because the occupant is at the first operational capability level, then the passage safety boundary still retains the remote request trigger condition. The passage safety boundary is a control range that simultaneously satisfies the restrictions of the terminal area and the restrictions of the passenger status. The control of electric wheelchairs no longer depends solely on the passenger's actions or the terminal area itself, but also on whether the area is suitable for performing the action and whether the current passenger is suitable to withstand the action. When both are permissible, the candidate control action is executed normally. This avoids the need for strong control in complex areas such as security checkpoints, boarding gates, and areas where crowds converge due to the large range of passenger movements. At the same time, it also avoids executing wheelchair actions according to the control parameters of a normal passenger when the passenger's posture is unstable, head movements are unstable, or gaze is abnormal, further improving the intelligence of controlling electric wheelchairs.
[0040] In some embodiments of this application, when outputting wheelchair control commands, the following steps are included: when the action source type is an active control action, and the speed increment, steering angle, and action response time corresponding to the candidate control action are all within the traffic safety boundary, a driving adjustment command is output; when the action source type is an active control action, and the corresponding speed increment / steering angle / action response time exceeds the traffic safety boundary, the excess portion is truncated, and a speed limit driving command or a deceleration steering command is output based on the scene constraint state; when the action source type is an abnormal risk action, and there is an obstacle-restricted state / path-restricted state / third operational capability level, a parking control command, a remote request command, or an emergency stop control command is output.
[0041] Specifically, speed increment is the amount of speed change that a candidate control action would increase or adjust in an electric wheelchair. For example, after an occupant makes a forward movement, the wheelchair doesn't immediately reach a certain speed; instead, the speed increase corresponding to that action is first determined. Steering angle is the angle at which a candidate control action would cause the wheelchair to turn left or right. For example, after an occupant makes a left turn, the angle or steering range of the wheelchair's single left turn needs to be determined. Action response time is the response time between the confirmation of a candidate control action and the actual execution of the control action by the wheelchair. Small speed increments may be safe in ordinary passageways, but in boarding gate areas or areas where people converge, the same speed increment may be too large. In elevator areas or turning areas, it may collide with pedestrians, luggage, or walls. When the source of the action is an active control action, and the speed increment, steering angle, and action response time corresponding to the candidate control action are all within the safe passage boundary, it means that the action comes from the active intention of the passenger and does not exceed the final control range formed by the passage area of the terminal and the passenger's operational ability level. In this case, a driving adjustment command is output, allowing the electric wheelchair to perform corresponding driving control according to the candidate control action, such as: performing forward, backward, left turn, right turn, or speed adjustment according to the candidate control action. When the action source type is an active control action, and any one of the corresponding speed increment, steering angle, or action response time exceeds the traffic safety boundary, the excess portion will be truncated. Truncating the excess portion does not completely negate the occupant's active control action, but rather compresses the control quantity exceeding the traffic safety boundary in the candidate control action to the allowable range. For example, if the speed increment corresponding to the candidate control action is greater than the maximum speed increment allowed by the current traffic safety boundary, only the speed increment within the allowable range will be retained. If the steering angle corresponding to the candidate control action is greater than the single steering angle allowed by the current traffic safety boundary, only the allowable steering angle will be executed. Active control actions themselves have the occupant's control intention. If the electric wheelchair directly refuses to execute, it may affect the occupant's normal passage. However, if it is fully executed, it may cause excessively fast, abrupt, or large wheelchair movements under restricted passage conditions. Truncating is equivalent to converting the action that the occupant wants to do into an action that is allowed to be executed in the current area and the current occupant state. The speed limit driving command limits the maximum speed or speed increment while allowing the wheelchair to continue driving. It is suitable for situations where the candidate control action mainly involves forward movement, acceleration, or speed adjustment, but the current scenario constraints are not suitable for direct acceleration. The deceleration and steering command reduces the vehicle speed and limits the steering angle or speed before or during a steering maneuver. It is applicable when the candidate control action involves a left turn, right turn, or path correction, but the current area presents risks related to curves, pedestrian traffic, ramps, path deviation, or obstacle approach. When the action source type is an abnormal risk action, and there are obstacle-restricted, path-restricted, or third-level operational capability conditions, a parking control command, a remote request command, or an emergency stop control command is output.An abnormal risk action occurs when the candidate control action deviates from the normal control logic. This may stem from sudden discomfort, instability, stress reactions, misoperation, or the simultaneous triggering of multiple control actions by the occupant. When an abnormal risk action is accompanied by any risk condition, it indicates that continuing to adjust the vehicle's movement according to the candidate control action is no longer suitable, nor is it appropriate to handle the situation simply by cutting off control inputs. In this case, a stop control command, a remote request command, or an emergency stop control command is output based on the risk level. The stop control command controls the electric wheelchair to stop using a controlled deceleration method, applicable when an abnormal risk action has occurred but the obstacle distance has not yet reached an emergency level, or when path deviation requires correction. The remote request command sends a takeover request to a remote monitoring terminal or ground staff, allowing personnel to confirm or take over based on video, location, occupant status, and the driving environment. This is applicable to scenarios requiring manual intervention, such as boarding gate areas, security checkpoints, and areas with high pedestrian traffic. The emergency stop control command directly triggers the electric wheelchair's emergency braking or cuts off the driving output, applicable when an abnormal risk action is accompanied by an obstacle-limited state or a third level of operational capability. This not only avoids the risk of directly controlling the wheelchair as soon as a movement is matched, as is the case with traditional motion control, but also enables electric wheelchairs to be controlled in a graded manner according to the source of the movement, the constraints of the scene, the safety boundaries of passage, and the level of the occupant's operating ability, further improving the intelligence of controlling electric wheelchairs.
[0042] In summary, the beneficial effects of this invention are as follows: By collecting occupant status information, vehicle operation information, wheelchair posture information, driving environment information, and navigation positioning information, the control basis of the electric wheelchair is no longer limited to a single posture action. Instead, it incorporates occupant status, vehicle movement, wheelchair posture, surrounding environment, and current position into the judgment, avoiding misjudgments of control due to a single control signal. Matching occupant action characteristics with preset control actions can filter out actions corresponding to the control intent. By combining vehicle operation information and wheelchair posture information, the direction of disturbance triggering and the time of disturbance occurrence can be determined, distinguishing between active control actions, passive disturbance actions, and abnormal risk actions. This avoids misjudging road bumps, slope changes, turning inertia, or involuntary occupant swaying as valid control commands. By defining the constraints of the scenario, the control commands can be adjusted according to different terminal environments such as passageways, security checkpoints, boarding gates, and densely populated areas. Furthermore, the control can be restricted based on the passenger's own condition and the area they are in, reducing erroneous actions caused by vehicle movement. Based on the passenger's operational ability level, obstacle proximity information, and the terminal's access area, corresponding control commands are output, improving the reliability and intelligence of electric wheelchair control.
[0043] In another preferred embodiment based on the above embodiments, see [reference] Figure 3 As shown, this embodiment provides an electric wheelchair control system based on a multimodal large model, used to apply the above-described electric wheelchair control method based on a multimodal large model, including: The data acquisition unit is configured to collect information on the occupant status of the electric wheelchair, vehicle operation information, wheelchair posture information, driving environment information, and navigation and positioning information. The analysis unit is configured to extract occupant action features based on occupant status information, match the occupant action features with preset control actions to determine candidate control actions, determine the disturbance trigger direction and disturbance occurrence time based on vehicle operation information and wheelchair posture information, and compare the disturbance trigger direction and disturbance occurrence time with candidate control actions to determine the action source type of candidate control actions. The control unit is configured to identify the terminal access area, path deviation information and obstacle approach information of the electric wheelchair based on driving environment information and navigation positioning information, and determine the scene constraint state, and determine the control command of the electric wheelchair based on the action source type and scene constraint state; The control unit is also configured to determine the crew's operational capability level based on the crew status information and determine the passage safety boundary based on the terminal passage area and path deviation information when the action source type is an active control action. Based on the matching result of the crew's operational capability level and the passage safety boundary, it outputs driving adjustment command, speed limit driving command or deceleration and steering command. The control unit is also configured to, when the action source type is a passive disturbance action, mask candidate control actions based on the disturbance triggering direction, disturbance occurrence time, and scene constraint state, and output a driving-maintaining command or a driving-decelerating command based on the scene constraint state; The control unit is also configured to output a stop control command / remote request command / emergency stop control command based on the crew's operational competence level, obstacle proximity information, and terminal access area when the source of the action is an abnormal risk action.
[0044] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program goods according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.
[0045] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the specific implementation of the present invention. Any modifications or equivalent substitutions that do not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.
Claims
1. A control method for an electric wheelchair based on a multimodal large model, characterized in that, include: Collect information on the occupant status, vehicle operation, wheelchair posture, driving environment, and navigation and positioning of the electric wheelchair; Based on the occupant status information, occupant action features are extracted and matched with preset control actions to determine candidate control actions. Based on the vehicle operation information and wheelchair posture information, the disturbance trigger direction and disturbance occurrence time are determined. The disturbance trigger direction and disturbance occurrence time are correlated and compared with the candidate control actions to determine the action source type of the candidate control actions. Based on the driving environment information and navigation positioning information, the terminal passage area, path deviation information and obstacle approach information of the electric wheelchair are identified, and the scene constraint state is determined. Based on the action source type and scene constraint state, the control command of the electric wheelchair is determined. When the action source type is an active control action, the crew operation capability level is determined based on the crew status information, and the passage safety boundary is determined based on the terminal passage area and path deviation information. Based on the matching result of the crew operation capability level and the passage safety boundary, the driving adjustment command, speed limit driving command or deceleration and steering command is output. When the action source type is a passive disturbance action, the candidate control action is masked based on the disturbance triggering direction, disturbance occurrence time, and scene constraint state, and a driving maintenance command or a driving deceleration command is output based on the scene constraint state. When the action source type is an abnormal risk action, a parking control command / remote request command / emergency stop control command is output based on the occupant's operational ability level, obstacle approach information, and terminal access area.
2. The electric wheelchair control method based on a multimodal large model according to claim 1, characterized in that, When determining candidate control actions, the process includes: determining the occupant's head turning direction, upper body tilting direction, movement amplitude, and facial orientation information based on the occupant's state information; determining the head turning direction and upper body tilting direction as the action direction; determining the movement amplitude as the action intensity information; determining the occupant's action characteristics based on the action direction, action intensity information, and facial orientation information; and determining the corresponding preset control action as the candidate control action when the occupant's action characteristics are consistent with the direction and amplitude conditions in the preset control action.
3. The electric wheelchair control method based on a multimodal large model according to claim 2, characterized in that, When determining the disturbance triggering direction and disturbance occurrence time, the process includes: determining the vehicle speed change, drive wheel speed difference, and drive current change based on the vehicle operation information; determining the vehicle body pitch change, vehicle body yaw change, and vehicle body roll change based on the wheelchair posture information; when the vehicle speed change, drive wheel speed difference, drive current change, vehicle body pitch change, vehicle body yaw change, and / or vehicle body roll change are synchronous disturbances, the synchronization start time is determined as the disturbance occurrence time, and the front-to-back disturbance direction and / or left-to-right disturbance direction and / or lateral disturbance direction are determined based on the direction of the synchronous disturbance.
4. The electric wheelchair control method based on a multimodal large model according to claim 3, characterized in that, When determining the action source type of the candidate control action, the following steps are taken: if the action occurrence time of the candidate control action is earlier than the disturbance occurrence time, and the action direction of the candidate control action is consistent with the control direction corresponding to the preset control action, then the action source type is determined to be the active control action; if the action occurrence time of the candidate control action is later than or equal to the disturbance occurrence time, and the action direction of the candidate control action is consistent with or opposite to the disturbance triggering direction, then the action source type is determined to be the passive disturbance action; if the candidate control action matches several preset control actions, and the action direction is inconsistent with both the current driving direction and the path direction, then the action source type is determined to be the abnormal risk action.
5. The electric wheelchair control method based on a multimodal large model according to claim 1, characterized in that, When determining the scene constraint state, the process includes: determining the current position of the electric wheelchair in the terminal based on the navigation and positioning information, matching the current position with the area boundary to determine the terminal access area, determining the path deviation information based on the navigation and positioning information and the target access path, determining obstacle approach information based on the driving environment information, and determining the scene constraint state as a passage access state when the terminal access area is a passage area, the path deviation information is within the deviation range, and the obstacle approach information does not meet the approach conditions; otherwise, it is a restricted access state.
6. The electric wheelchair control method based on a multimodal large model according to claim 5, characterized in that, When determining the scene constraint state, the following are also included: the restricted access state includes area restricted state, path restricted state, and obstacle restricted state. When the terminal access area is a security check area / boarding gate area / elevator area / ramp area / turning area / human confluence area, the restricted access state is determined to be the area restricted state; when the path deviation information is that the electric wheelchair deviates from the target access path and the deviation direction is not the target access area, the restricted access state is determined to be the path restricted state; when the obstacle approach information is that there is an obstacle in the forward / lateral / turning direction, the restricted access state is determined to be the obstacle restricted state.
7. The electric wheelchair control method based on a multimodal large model according to claim 1, characterized in that, Determining the occupant's operational ability level includes: determining the occupant's age group, sitting posture, head movement, upper body posture, and gaze state based on the occupant's status information; when the occupant's age group is a low-risk age group, and the sitting posture, head movement, and gaze state all meet the stability condition, it is determined to be at the first operational ability level; when the occupant's age group is an elderly age group, or when any of the sitting posture, head movement, and upper body swaying states does not meet the stability condition, it is determined to be at the second operational ability level; when the occupant's gaze state or upper body posture is abnormal, it is determined to be at the third operational ability level.
8. The electric wheelchair control method based on a multimodal large model according to claim 7, characterized in that, Determining the passage safety boundary includes: determining the area control boundary based on the terminal passage area, determining the crew control boundary based on the crew operation capability level, and superimposing the area control boundary and the crew control boundary to determine the passage safety boundary.
9. The electric wheelchair control method based on a multimodal large model according to claim 8, characterized in that, When outputting wheelchair control commands, the following steps are taken: when the action source type is the active control action, and the speed increment, steering angle, and action response time corresponding to the candidate control action are all within the traffic safety boundary, then the driving adjustment command is output; when the action source type is the active control action, and the corresponding speed increment / steering angle / action response time exceeds the traffic safety boundary, then the excess portion is truncated, and the speed limit driving command or deceleration steering command is output based on the scenario constraint state; when the action source type is the abnormal risk action, and there is an obstacle-restricted state / path-restricted state / third operation capability level, then a parking control command, a remote request command, or an emergency stop control command is output.
10. An electric wheelchair control system based on a multimodal large model, used to apply the electric wheelchair control method based on a multimodal large model as described in any one of claims 1-9, characterized in that, include: The data acquisition unit is configured to collect information on the occupant status of the electric wheelchair, vehicle operation information, wheelchair posture information, driving environment information, and navigation and positioning information. The analysis unit is configured to extract occupant action features based on the occupant status information, match the occupant action features with preset control actions to determine candidate control actions, determine the disturbance triggering direction and disturbance occurrence time based on the vehicle operation information and wheelchair posture information, and correlate and compare the disturbance triggering direction and disturbance occurrence time with the candidate control actions to determine the action source type of the candidate control actions. The control unit is configured to identify the terminal access area, path deviation information and obstacle approach information of the electric wheelchair based on the driving environment information and navigation positioning information, determine the scene constraint state, and determine the control command of the electric wheelchair based on the action source type and scene constraint state; The control unit is further configured to, when the action source type is an active control action, determine the occupant's operational capability level based on the occupant status information, determine the passage safety boundary based on the terminal passage area and path deviation information, and output a driving adjustment command, a speed limit driving command, or a deceleration and steering command based on the matching result of the occupant's operational capability level and the passage safety boundary. The control unit is further configured to, when the action source type is a passive disturbance action, to shield the candidate control action based on the disturbance triggering direction, disturbance occurrence time, and scene constraint state, and to output a driving maintenance command or a driving deceleration command based on the scene constraint state. The control unit is also configured to output a stop control command / remote request command / emergency stop control command based on the occupant's operational capability level, obstacle proximity information, and terminal access area when the action source type is an abnormal risk action.
Citation Information
Patent Citations
Body feeling control method and device of electric wheelchair
CN106909106A
