Autonomous mobile platform system based on VLN and multi-mode perception and control method

Abstract

The invention discloses an autonomous mobile platform system based on VLN and multi-modal perception and a control method, the system allows a user to input a destination instruction through a natural language, a visual perception unit dynamically constructs a topological map containing semantic information of furniture, porches and the like in a continuous indoor environment, and plans an obstacle avoidance safety path in real time; meanwhile, key indexes such as the blood pressure, the heart rate and the facial expression of the user are monitored in real time through a multi-mode physiological sensor, a grading response mechanism is automatically triggered when health or emotion abnormity is detected, and the running speed of the wheelchair is intelligently limited or emergency braking is executed. In addition, the system is deeply cooperated with the smart home equipment through an Internet of Things communication protocol, and the preparation action of the environment equipment is automatically triggered according to the navigation path prediction. According to the invention, the passing efficiency and safety of people with mobility difficulties in an indoor complex environment are remarkably improved, and a set of highly integrated intelligent mobile solution is provided for intelligent old-age care communities and medical rehabilitation institutions.

Description

Technical Field

[0001] This invention relates to the fields of rehabilitation aids, service robots, and human-computer interaction technologies. It relates to an autonomous mobile platform system and control method based on VLN (Vision-Language Navigation) and multimodal perception. In particular, it relates to an autonomous mobile platform system and control method that integrates Vision-Language Navigation (VLN-CE), multimodal physiological-emotion perception, smart home environment collaboration, path planning, and dynamic control. Background Technology

[0002] With the accelerating aging of the global population, the demand for mobility assistance among the elderly and people with physical disabilities is rising sharply. The "World Population Aging Report 2024" predicts that by 2030, there will be more than 1.4 billion people aged 60 and over worldwide, a significant proportion of whom will experience varying degrees of mobility impairment. The ability of this population to achieve independent mobility, safe care, and a comfortable experience in residential, nursing home, and medical environments is becoming a core pain point for the rehabilitation assistive device industry.

[0003] Most existing electric wheelchairs (autonomous mobility platforms) still use joysticks or buttons for control, and the user interface is not user-friendly for users with limb weakness or limited cognitive abilities. Even if some products add voice control, they usually rely on fixed keyword commands and cannot understand natural language goals such as "take me to the sofa by the window in the living room." In terms of navigation, most products are only equipped with ultrasonic or infrared obstacle avoidance sensors, passively avoiding obstacles and unable to plan a global path, so users still need to frequently manually adjust the direction.

[0004] For autonomous navigation, the academic community has successively proposed laser SLAM, visual SLAM, and depth camera fusion solutions. However, these technologies often require pre-mapped environments or operation in stable lighting and open spaces, making them difficult to adapt to the complex conditions of residential scenarios with dense furniture, large lighting variations, and narrow spaces. Visual-language navigation (VLN), which has emerged in recent years, allows robots to infer the location of targets in continuous environments based on natural language. Unfortunately, it is still mainly used in indoor inspection and service robot research platforms, and commercial wheelchairs (autonomous mobility platforms) have not yet integrated this approach.

[0005] On the other end of the spectrum, disconnected from navigation functionality, is the monitoring of the user's physiological and emotional state. Devices on the market, such as pressure ulcer alarm cushions and independent wristband heart rate monitors, can only provide single physiological indicators; they are neither integrated into the control logic of the autonomous mobility platform nor can they adjust driving parameters or trigger emergency stops in real time when high blood pressure or pain is detected, posing safety hazards.

[0006] Meanwhile, smart home and Internet of Things (IoT) technologies are rapidly becoming widespread—door locks, lights, curtains, elevators, and even care robots now have networked control capabilities. However, the interaction between existing wheelchairs (autonomous mobility platforms) and these devices mostly relies on mobile apps or independent remote controls. The operation process is decoupled from the wheelchair's (autonomous mobility platform's) movement, leading to user experience issues such as waiting at the door and glare from nighttime lighting. This further highlights that if wheelchairs (autonomous mobility platforms) could directly collaborate with home systems, it would significantly improve mobility and living comfort.

[0007] In summary, the common technical shortcomings in the current field of intelligent wheelchairs (autonomous mobility platforms) include limited control methods, insufficient navigation robustness, lack of health protection, and lack of environmental coordination.

[0008] Therefore, developing an autonomous mobile platform solution that can simultaneously understand natural language targets, navigate autonomously in continuous indoor environments, perceive multimodal physiological and emotional information in real time, and seamlessly coordinate with smart homes is of great practical significance. Summary of the Invention

[0009] Due to the aforementioned deficiencies in existing technologies, this invention provides an autonomous mobile platform solution and its control method that can simultaneously understand natural language targets, navigate autonomously in continuous indoor environments, perceive multimodal physiological and emotional information in real time, and seamlessly coordinate with smart homes. This solution can meet the urgent needs of disabled and elderly users for safe, efficient, and low-burden travel, and overcomes the shortcomings of current autonomous mobile platforms, such as poor autonomous navigation capabilities in continuous indoor environments, inability to perceive and coordinate with users' physiological and emotional information, and inability to seamlessly coordinate with smart homes.

[0010] To achieve the above objectives, the present invention provides the following technical solution:

[0011] An autonomous mobile platform system based on VLN and multimodal perception includes:

[0012] A) Visual-language navigation module, used to fuse user natural language or gesture target commands with real-time visual data to generate navigation paths;

[0013] B) Multimodal physiological-emotion perception module, used to collect and analyze user blood pressure, heart rate and facial expression information;

[0014] C) Dynamics control module, used to output motion commands for the autonomous mobile platform according to the navigation path, and to perform deceleration or emergency stop when an abnormal state is detected;

[0015] D) Interactive panel and annotation map module, used to graphically display paths and environmental elements and receive user intervention;

[0016] E) Smart home collaboration module, used to communicate with various smart home devices and trigger corresponding actions;

[0017] F) Emergency stop safety module, used to brake quickly and log when a collision risk or abnormal user condition is detected.

[0018] The aforementioned visual-language navigation module can be designed as follows: continuous images are acquired through a forward-facing RGB-D and a top-facing depth camera, and natural language target instructions are combined with fine-tuned BERT encoding to complete semantic-visual feature fusion in a cross-modal gated Transformer-GRU network; then Anytime D* Lite global planning and MPC-DWA local obstacle avoidance are run on a dynamically generated semantic topology map to achieve robust navigation without the need for pre-built maps;

[0019] The aforementioned multimodal physiological-emotion perception module can be designed as follows: integrating PPG, cuff blood pressure monitor and facial expression camera, using Kalman filtering and AU-RCNN network to output heart rate, blood pressure and emotional state, and coupling with navigation decision through hierarchical finite state machine to realize speed limit, emergency stop or nursing call.

[0020] The aforementioned smart home collaboration module can be designed as follows: Unified access via Matter, Wi-Fi, and ZigBee is implemented on the Thread Border Router; the relative position of the autonomous mobile platform and points of interest (POIs) is predicted using UWB-RTLS; a YAML-DSL script is generated through a semantic-temporal scene scheduling unit to trigger pre-opening / pre-closing 3 seconds before the autonomous mobile platform is about to pass a door lock, light, or elevator; and it automatically downgrades to local Mesh rules when the network is disconnected, ensuring that core functions do not fail.

[0021] The above-mentioned emergency stop safety module can be designed as follows: the hardware E-stop directly cuts off the main power supply, and the software pulls the motor PWM linearly down to zero and records a 2-second high-frequency log when the IMU detects an acceleration >2 g or a physiological "critical" alarm, in order to meet the requirements of ISO7176-14 and GB 12996-2020.

[0022] This invention presents an autonomous mobility platform system based on VLN and multimodal perception. Its rational structural design allows for seamless collaboration between modules, enabling effective coordination between VLN and user status monitoring. This provides reliable safety protection for users, and the addition of a smart home collaboration module significantly enhances the user experience. It is particularly suitable for use by the elderly and those with severe disabilities. Through natural language / gesture interaction, continuous environmental semantic navigation, real-time physiological and emotional monitoring, dual safety braking, and IoT collaborative control, this system constructs an integrated solution of "autonomous mobility—health protection—environmental interconnection," providing users with a low-load, highly safe, and highly comfortable indoor travel experience, demonstrating promising application prospects.

[0023] This invention also provides a control method for an autonomous mobile platform system based on VLN and multimodal sensing as described above, comprising the following steps:

[0024] S1. Users input their destination using natural language or gestures;

[0025] S2, the visual-language navigation module combines visual data to construct a semantic-topology map and plan global-local paths;

[0026] S3. The dynamics control module drives the autonomous mobile platform to move according to the global-local path.

[0027] S4. The multimodal physiological-emotion perception module monitors the user's status in real time and sends alarms to the dynamic control module and smart home collaboration module when the threshold is exceeded.

[0028] S5. The interactive panel and annotation map module display the annotation map and allow users to modify the path or destination;

[0029] S6. The smart home collaboration module can trigger the actions of home devices (door locks, lights, air conditioners or elevators) in advance based on the next point of interest in the path.

[0030] S7. In the event of a network outage, switch to local automation rules to maintain the minimum available control for home appliances (door locks and lighting) and save fault information to the log module.

[0031] Calibrate and perform health checks on all sensors of the autonomous mobile platform system before operation to improve the user experience.

[0032] The aforementioned control method features a well-designed sequence. It not only enables real-time monitoring of users through a multimodal physiological-emotion perception module and rapid control of the autonomous mobile platform to ensure user safety and prevent secondary harm, but also significantly enhances the user experience through seamless home collaboration. Furthermore, the system possesses resilience and security redundancy during network outages, ensuring device availability and reliability, and demonstrating promising application prospects.

[0033] As a preferred technical solution:

[0034] As described above, in the control method, the visual-language navigation module generates a semantic anchor sequence based on the depth-color image acquired by the RGB-D camera and uses a finely tuned Transformer-GRU cross-modal fusion network.

[0035] The visual-language navigation module generates semantic nodes for static objects such as furniture, doors, and stairs through an incremental semantic-topology map construction unit, and updates passable edges for dynamic obstacles in real time.

[0036] The vision-language navigation module uses deep reinforcement learning to fine-tune the planning-execution error online, thereby reducing the cumulative positioning deviation.

[0037] As described above, the dynamic control module adaptively adjusts the maximum linear velocity and angular velocity according to the path curvature and the load of the autonomous mobile platform, and controls the braking distance of the autonomous mobile platform to within 0.20 m when the emergency stop safety module is triggered.

[0038] As described in the control method above, the multimodal physiological-emotion perception module includes a PPG sensor, a cuff-type blood pressure monitor, and an expression recognition camera based on a Facial Action Coding System;

[0039] The multimodal physiological-emotion perception module identifies anomalies using a trend-threshold joint algorithm with time windowing.

[0040] As described above, in the control method, when the multimodal physiological-emotion perception module detects a heart rate higher than 120 bpm and recognizes the expression as pain or anger, it sends an emergency stop command to the dynamic control module and pops up a nursing call interface on the interactive panel. In conjunction with the multimodal physiological-emotion perception module's real-time assessment of the user's state, various safety strategies can be set, such as a two-level safety strategy of "minor danger - speed limit" and "serious danger - emergency stop." Switching between these two levels of safety strategies can improve the user experience.

[0041] As described above, the smart home collaboration module includes a semantic-temporal scene scheduling unit. The semantic-temporal scene scheduling unit generates control instructions for home devices based on the next point of interest predicted by the visual-language navigation module, and cancels unexecuted instructions when the user's path changes.

[0042] As described above, the smart home collaboration module uses a communication unit that conforms to the Matter, Thread, Wi-Fi or ZigBee protocol, and establishes an end-to-end encrypted channel through OSCORE or DTLS.

[0043] When the smart home collaboration module detects an interruption in Internet connection, it automatically switches to local ThreadMesh automation mode and synchronizes the device status after the network is restored. In the event of a fault or emergency stop, it outputs the circular log buffer to the diagnostic interface to provide a basis for subsequent maintenance.

[0044] The smart home collaboration module dynamically adjusts smart home devices using a predicted travel time window to improve user comfort. Specifically, it can adjust the air conditioner fan speed and light brightness to maintain the user's skin temperature fluctuations within ±0.5℃.

[0045] As described above, the autonomous mobile platform system based on VLN and multimodal perception also includes a UWB-RTLS positioning base station and a UWB-Tag installed on the autonomous mobile platform, which is used to predict the distance between the autonomous mobile platform and home appliances, and send home control commands 3 seconds in advance when the distance is less than a preset threshold.

[0046] As described above, the interactive panel of the interactive panel and the annotation map module includes a touch screen, a voice recognition device and a gesture recognition input device, and simultaneously displays a semantic-topological map, the current pose of the autonomous mobile platform and a summary of the user's vital signs on the interface.

[0047] Within one second after an emergency stop event, the system uploads the fault log and sensor data from 500ms before and after the event to the cloud-based nursing platform via cellular or Wi-Fi network for remote diagnosis.

[0048] The above technical solution is only one feasible technical solution of the present invention. The scope of protection of the present invention is not limited thereto. Those skilled in the art can reasonably adjust the specific design according to actual needs.

[0049] The above invention has the following advantages or beneficial effects:

[0050] (1) The autonomous mobile platform system based on VLN and multimodal perception of the present invention has a high robust semantic navigation capability. In the experiment of residential and hospital scenes with dense furniture and large changes in lighting, the navigation success rate increased from 79.2% of the traditional DWA to 96.4%, and the average positioning error converged to 0.13 m.

[0051] (2) The autonomous mobile platform system based on VLN and multimodal perception of the present invention has the ability to actively protect health. The recall rate of abnormal heart rate by multimodal monitoring reaches 94.6%, and the average response time of emergency stop is 0.21 s, which significantly reduces the risk of secondary injury.

[0052] (3) The autonomous mobile platform system based on VLN and multimodal perception of the present invention has seamless home collaboration capabilities. The pre-action mechanism reduces the waiting time for door locks / elevators by 65% ​​and the peak value of nighttime light illuminance jump by 43%, thereby improving traffic efficiency and visual comfort.

[0053] (4) The autonomous mobile platform system based on VLN and multimodal perception of the present invention has network outage resilience and security redundancy. Local Mesh rules ensure that key functions are 100% available during a 30-minute network outage. Dual-path emergency stop and cyclic log meet the requirements of medical device safety regulations.

[0054] (5) The autonomous mobile platform system based on VLN and multimodal perception of the present invention has the advantage of easy industrialization. The modular hardware design and ROS 2 Foxy software architecture can directly connect to existing autonomous mobile platform chassis and smart home platforms, reducing mass production and maintenance costs.

[0055] (6) The autonomous mobile platform system based on VLN and multimodal perception of the present invention has a reasonable structural design, and the modules can cooperate with each other. It can achieve good coordination between VLN and user status monitoring, and can provide reliable security protection for users. At the same time, the addition of the smart home collaboration module can significantly improve the user experience. It is particularly suitable for use in scenarios involving the elderly and severely disabled, and has good application prospects. Attached Figure Description

[0056] The invention, its features, shape, and advantages will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings. Like reference numerals denote like parts throughout the drawings. The drawings are not drawn to scale; their purpose is to illustrate the gist of the invention.

[0057] Figure 1 This is an overall flowchart of the control method for the autonomous mobile platform system based on vision-language navigation and multimodal perception according to the present invention;

[0058] Figure 2 This is a schematic diagram of the modules of the autonomous mobile platform system based on vision-language navigation and multimodal perception of the present invention;

[0059] Figure 3 This is a flowchart of the multimodal perception-obstacle avoidance fusion process.

[0060] Figure 4 Flowchart of the user control recovery mechanism. Detailed Implementation

[0061] To make the objectives, technical solutions, beneficial effects, and significant advancements of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the embodiments of the present invention.

[0062] Obviously, all the embodiments described are only some embodiments of the present invention, and not all embodiments; based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0063] Example 1

[0064] An autonomous mobile platform system based on VLN and multimodal perception, such as Figure 2 As shown, it includes:

[0065] A) Visual-language navigation module, used to fuse user natural language or gesture target commands with real-time visual data to generate navigation paths;

[0066] B) Multimodal physiological-emotion perception module, used to collect and analyze user blood pressure, heart rate and facial expression information;

[0067] C) Dynamics control module, used to output motion commands for the autonomous mobile platform according to the navigation path, and to perform deceleration or emergency stop when an abnormal state is detected;

[0068] D) Interactive panel and annotation map module, used to graphically display paths and environmental elements and receive user intervention;

[0069] E) Smart home collaboration module, used to communicate with various smart home devices and trigger corresponding actions;

[0070] F) Emergency stop safety module, used to brake quickly and log when a collision risk or abnormal user condition is detected.

[0071] The control method for the autonomous mobile platform system based on VLN and multimodal perception mentioned above includes the following steps (such as...). Figure 1 (as shown)

[0072] S1. Users input their destination using natural language or gestures;

[0073] S2, the visual-language navigation module combines visual data to construct a semantic-topology map and plan global-local paths. The visual-language navigation module is based on depth-color images acquired by an RGB-D camera and uses a fine-tuned Transformer-GRU cross-modal fusion network to generate semantic anchor sequence. The visual-language navigation module generates semantic nodes for static objects through an incremental semantic-topology map construction unit and updates passable edges for dynamic obstacles in real time. The visual-language navigation module uses deep reinforcement learning to fine-tune the planning-execution error online.

[0074] S3. The dynamic control module drives the autonomous mobile platform to move according to the global-local path. The dynamic control module adaptively adjusts the maximum linear velocity and angular velocity according to the path curvature and the load of the autonomous mobile platform, and controls the braking distance of the autonomous mobile platform to within 0.20 m when the emergency stop safety module is triggered.

[0075] S4. The multimodal physiological-emotion perception module monitors the user's status in real time and sends alarms to the dynamic control module and smart home collaboration module when the threshold is exceeded. The multimodal physiological-emotion perception module includes a PPG sensor, a cuff blood pressure monitor, and an expression recognition camera based on the Facial Action Coding System. The multimodal physiological-emotion perception module uses a trend-threshold joint algorithm with time windowing to identify abnormalities. When the multimodal physiological-emotion perception module detects a heart rate higher than 120 bpm and the expression is identified as pain or anger, it sends an emergency stop command to the dynamic control module and pops up a nursing call interface through the interactive panel.

[0076] S5. The interactive panel and annotation map module displays the annotation map and allows users to modify the path or destination. The interactive panel of the interactive panel and annotation map module includes a touch screen, a voice recognition device and a gesture recognition input device, and simultaneously displays the semantic-topology map, the current pose of the autonomous mobile platform and the user's vital signs summary on the interface.

[0077] S6. The smart home collaboration module triggers home device actions in advance based on the next point of interest (POI) along the path. This module includes a semantic-temporal scene scheduling unit, which generates home device control commands based on the POI predicted by the visual-language navigation module and cancels unexecuted commands when the user's path changes. The smart home collaboration module uses communication units conforming to Matter, Thread, Wi-Fi, or ZigBee protocols and establishes an end-to-end encrypted channel via OSCORE or DTLS. When an Internet connection interruption is detected, the smart home collaboration module automatically switches to local Thread Mesh automation mode and synchronizes device status after network recovery. The smart home collaboration module dynamically adjusts smart home devices using predicted travel time windows to improve user comfort. The autonomous mobile platform system based on VLN and multimodal perception also includes a UWB-RTLS positioning base station and a UWB-Tag installed on the autonomous mobile platform to predict the distance between the autonomous mobile platform and home devices, and sends home control commands in advance when the distance is less than a preset threshold.

[0078] S7. When the network is interrupted, switch to local automation rules to maintain the minimum available control of home devices and save fault information to the log module. The system uploads the fault log and sensor data before and after 500 ms to the cloud care platform via cellular or Wi-Fi network within a certain period after the emergency stop event.

[0079] The obstacle avoidance methods of the vision-language navigation module involved in the above control method are as follows (e.g.) Figure 3 (as shown)

[0080] (1) Security domain construction;

[0081] (1.1) Definition of safe distance, safe distance S safe Take half of the maximum outer width of the autonomous mobile platform as a reference value;

[0082] (1.2) The boundary of the safety region, in the local coordinate system XY:

[0083] Upper boundary: The left wall is shifted inward by a distance equal to S. safe The distance;

[0084] Lower boundary: When there are no obstacles in front, the right wall also moves inward by S. safe ;

[0085] If an obstacle is detected, its X-coordinate position is recorded as X. obs The width of the obstacle itself is denoted as W. obs Then, upon reaching X on the autonomous mobile platform... obs Previously, the lower boundary was still shifted inward by S.safe When the X coordinate of the autonomous mobile platform exceeds X obs Subsequently, to ensure that the outline of the autonomous mobile platform does not contact the outline of the obstacle, the total distance the lower boundary is translated inward is equal to S. safe With W obs The value after addition;

[0086] (1.3) Gradual boundary, the starting point of the turning is denoted as X. str To avoid abrupt boundary changes during turning, the boundary within a buffer zone (length δ) in front of the starting point is smoothly transitioned using linear interpolation. It is also stipulated that throughout the transition and subsequent regions, the shortest distance from the boundary to the outer edge of the obstacle must be no less than a safety distance S. safe With the set safety margin D safe The total value

[0087] (2) Spatial safety factor field;

[0088] (2.1) Reference safety track

[0089] Straight-line phase (X coordinate less than X str ): Take the current driving centerline of the autonomous mobile platform as the reference;

[0090] Obstacle avoidance phase (X coordinate between X...) str With X obs Between): Starting from the turning point P1 (x-coordinate X) str , ordinate Y str ) and reference lateral displacement point P2 (x-coordinate X) obs , ordinate Y trg The line connecting the two points serves as the reference.

[0091] Beyond stage (X coordinate greater than X) obs ): Take the ordinate as Y trg A horizontal line is used as a reference;

[0092] (2.2) Definition of safety factor at any x-axis position:

[0093] When the lateral displacement of the target point of the autonomous mobile platform is equal to the baseline safe lateral displacement, the safety factor is recorded as 1;

[0094] As the lateral displacement gradually deviates from the reference until it approaches the boundary of the safety zone, the safety factor decreases from 1 to 0 according to a linear law.

[0095] When the lateral displacement reaches or exceeds the boundary of the safety zone, the safety factor is 0.

[0096] (3) Dynamic stability safe region

[0097] (3.1) The two indicators of the center of mass sideslip angle and yaw rate are selected to characterize the lateral dynamics of the autonomous mobile platform.

[0098] (3.2) In the phase plane with the centroid sideslip angle as the horizontal axis and the centroid sideslip angle change rate as the vertical axis, give the dynamic stability boundary defined by two straight lines.

[0099] (3.3) The boundary intersects with the aforementioned spatial safety domain to form a comprehensive reference safety domain, which serves as the feasible state range for lateral obstacle avoidance control.

[0100] (4) Control mode determination

[0101] Calculate the estimated collision time T col and the latest braking time T LB Compare and simultaneously detect the steering wheel torque T applied by the driver. d :

[0102] If T col Longer than T LB If the system detects that the driver has a clear intention to steer, it will enter the cooperative assistance mode (see Part (5)).

[0103] If T col Not longer than T LB If the driver has not yet performed sufficient obstacle avoidance maneuvers, the system will enter emergency assistance mode (see Part (6)).

[0104] (5) Cooperative Assisted Control Algorithm

[0105] (5.1) State and disturbance quantity

[0106] The discrete state vector includes lateral displacement and heading angle, the control quantity is the auxiliary torque acting on the steering wheel, and the disturbance quantity is composed of the driver's steering torque and the electric power steering output.

[0107] (5.2) Model Establishment

[0108] Construct a discrete state-space incremental model to describe the relationship between the current state and the changes in control variables, and to facilitate subsequent constraint settings.

[0109] (5.3) Objective function, setting three objectives:

[0110] The goal is to make the predicted lateral displacement and heading angle closely match the reference trajectory; to make the changes in control quantities smooth and avoid sudden changes; and to suppress excessive auxiliary torque to reduce interference with the driver's control experience.

[0111] (5.4) Constraints: Set two levels of constraints:

[0112] The control quantity itself and its variation range must not exceed the limits that the assist motor can achieve;

[0113] The output quantities such as lateral displacement and heading angle must not exceed the corresponding safety domain boundary.

[0114] (5.5) Human-machine weight allocation, introducing two safety factors:

[0115] Obstacle avoidance space safety factor σ s With driving behavior safety factor σ d The obstacle avoidance safety level index η is obtained by averaging the two values. The lower the η value, the higher the risk, and the controller automatically increases its own weight; conversely, the driver takes more control.

[0116] (6) Emergency Assist Control Algorithm

[0117] (6.1) In emergency mode, the system directly controls the differential torque of the left and right drive wheels to achieve efficient lateral displacement.

[0118] (6.2) Control the target, quickly generate the necessary lateral displacement and heading angle, and try to keep the steering wheel output changes smooth to prevent impact on the driver.

[0119] (6.3) Constraints, which simultaneously limit wheel speed saturation, lateral acceleration and torque output change rate, to ensure both safety and comfort.

[0120] (7) Exit mechanism such as Figure 4 As shown;

[0121] (7.1) Driver actively terminates: During obstacle avoidance, if the direction of the steering wheel torque applied by the driver is opposite to that of the auxiliary torque, and the difference between the two reaches five Newtons and lasts for more than half a second, the system immediately stops outputting and returns all control to the driver.

[0122] (7.2) When the driver takes over and obstacle avoidance has been completed, but the driver wants to take control of the autonomous mobile platform immediately, the system will immediately disengage as long as the difference between the steering torque and the system auxiliary torque reaches five Nm and remains there.

[0123] (7.3) Prompt Exit If the driver does not make any obvious operation after the obstacle avoidance is completed, as long as the vehicle's lateral displacement meets the minimum requirements, the longitudinal direction has completely passed the obstacle, the lateral acceleration is less than one meter per second squared, the heading angle deviation is less than ten degrees and is maintained for two seconds, the system will prompt the driver to take over via voice and then automatically exit.

[0124] The autonomous mobile platform system based on VLN and multimodal perception of this invention has been successfully applied in the following three scenarios:

[0125] I) Indoor obstacle avoidance around the box: In a scenario where the aisle width is 1200 mm and the obstacle width is 400 mm, the system sets a safety distance of 300 mm, the comprehensive human-machine safety level index is 0.35, the system provides a maximum steering wheel assist torque of 2.1 Nm, successfully avoids the obstacle and prompts to exit;

[0126] II) Outdoor emergency obstacle avoidance: When the autonomous mobile platform is traveling at a speed of 1.2 meters per second, a child suddenly crosses the road in front. The estimated collision time is 600 milliseconds, which is earlier than the latest braking time of 800 milliseconds. Due to insufficient driver operation, the system immediately enters emergency mode and applies 18 Nm of dynamic torque to the difference between the left and right wheels, quickly completing a lateral displacement of about 350 millimeters to avoid the collision.

[0127] III) Multimodal physiological linkage: When the system detects that the rider's heart rate has risen to 120 beats per minute and the facial expression analysis shows anxiety, the system automatically increases the trajectory tracking weight and decreases the torque smoothing weight to enhance the assist effect, and prompts the user to stay relaxed on the interactive panel.

[0128] Verification has shown that the autonomous mobile platform system based on VLN and multimodal perception of the present invention has a reasonable structural design, and the modules can cooperate with each other. It can achieve good coordination between VLN and user status monitoring, and can provide reliable security protection for users. At the same time, the addition of the smart home collaboration module can significantly improve the user experience. It is particularly suitable for use in scenarios involving the elderly and severely disabled, and has good application prospects.

[0129] Those skilled in the art should understand that variations can be implemented by combining existing technology with the above embodiments, which will not be elaborated here. Such variations do not affect the essence of the present invention, and will not be elaborated here either.

[0130] The preferred embodiments of the present invention have been described above. It should be understood that the present invention is not limited to the specific embodiments described above, and the devices and structures not described in detail should be understood as being implemented in a conventional manner in the art. Any person skilled in the art can make many possible variations and modifications to the technical solutions of the present invention using the methods and techniques disclosed above, or modify them into equivalent embodiments with equivalent changes, without departing from the scope of the present invention. This does not affect the essential content of the present invention. Therefore, any simple modifications, equivalent changes, and modifications made to the above embodiments based on the technical essence of the present invention without departing from the content of the present invention's technical solutions still fall within the protection scope of the present invention.

Claims

1. An autonomous mobile platform system based on VLN and multimodal perception, characterized in that: include: A) Visual-language navigation module, used to fuse user natural language or gesture target commands with real-time visual data to generate navigation paths; B) Multimodal physiological-emotion perception module, used to collect and analyze user blood pressure, heart rate and facial expression information; C) Dynamics control module, used to output motion commands for the autonomous mobile platform according to the navigation path, and to perform deceleration or emergency stop when an abnormal state is detected; D) Interactive panel and annotation map module, used to graphically display paths and environmental elements and receive user intervention; E) Smart home collaboration module, used to communicate with various smart home devices and trigger corresponding actions; F) Emergency stop safety module, used to brake quickly and log when a collision risk or abnormal user condition is detected.

2. The control method for an autonomous mobile platform system based on VLN and multimodal sensing as described in claim 1, characterized in that, Includes the following steps: S1. Users input their destination using natural language or gestures; S2, the visual-language navigation module combines visual data to construct a semantic-topology map and plan global-local paths; S3. The dynamics control module drives the autonomous mobile platform to move according to the global-local path. S4. The multimodal physiological-emotion perception module monitors the user's status in real time and sends alarms to the dynamic control module and smart home collaboration module when the threshold is exceeded. S5. The interactive panel and annotation map module display the annotation map and allow users to modify the path or destination; S6, the smart home collaboration module triggers home device actions in advance based on the next point of interest in the path; S7. When the network is interrupted, switch to local automation rules to maintain the minimum available control of home devices and save fault information to the log module.

3. The control method according to claim 2, characterized in that, The visual-language navigation module is based on depth-color images acquired by an RGB-D camera and uses a finely tuned Transformer-GRU cross-modal fusion network to generate semantic anchor sequence. The visual-language navigation module generates semantic nodes for static objects and updates passable edges for dynamic obstacles in real time through the incremental semantic-topology map construction unit. The vision-language navigation module uses deep reinforcement learning to fine-tune the planning-execution error online.

4. The control method according to claim 2, characterized in that, The dynamic control module adaptively adjusts the maximum linear velocity and angular velocity based on the path curvature and the load of the autonomous mobile platform, and controls the braking distance of the autonomous mobile platform to within 0.20 m when the emergency stop safety module is triggered.

5. The control method according to claim 2, characterized in that, The multimodal physiological-emotion perception module includes a PPG sensor, a cuff-type blood pressure monitor, and an expression recognition camera based on the Facial Action Coding System. The multimodal physiological-emotion perception module identifies anomalies using a trend-threshold joint algorithm with time windowing.

6. The control method according to claim 2, characterized in that, When the multimodal physiological-emotion perception module detects a heart rate higher than 120 bpm and recognizes the expression as pain or anger, it sends an emergency stop command to the dynamic control module and pops up a nursing call interface through the interactive panel.

7. The control method according to claim 2, characterized in that, The smart home collaboration module includes a semantic-temporal scene scheduling unit, which generates home device control instructions based on the next point of interest predicted by the visual-language navigation module, and cancels unexecuted instructions when the user's path changes.

8. The control method according to claim 2, characterized in that, The smart home collaboration module uses a communication unit that conforms to the Matter, Thread, Wi-Fi or ZigBee protocol, and establishes an end-to-end encrypted channel through OSCORE or DTLS; When the smart home collaboration module detects an interruption in the Internet connection, it automatically switches to the local Thread Mesh automation mode and synchronizes the device status after the network is restored. The smart home collaboration module dynamically adjusts smart home devices using a predicted travel time window.

9. The control method according to claim 2, characterized in that, The autonomous mobile platform system based on VLN and multimodal perception also includes a UWB-RTLS positioning base station and a UWB-Tag installed on the autonomous mobile platform, which is used to predict the distance between the autonomous mobile platform and home devices, and send home control commands in advance when the distance is less than a preset threshold.

10. The control method according to claim 2, characterized in that, The interactive panel and the annotation map module's interactive panel include a touch screen, a voice recognition device, and a gesture recognition input device, and simultaneously display a semantic-topological map, the current pose of the autonomous mobile platform, and a summary of the user's vital signs on the interface; The system uploads fault logs and sensor data from 500 ms before and after the emergency stop event to the cloud-based nursing platform via cellular or Wi-Fi network within a certain period of time.

Images