Multi-modal motion control system, method and equipment for cleaning robot and medium

Abstract

The invention provides a cleaning robot multi-mode motion control system, method and equipment and a medium, and belongs to the technical field of intelligent robot control. The system comprises a multi-modal data acquisition module used for performing multi-source synchronous acquisition on an environment image, distance information, a voice signal and a collision signal and outputting original sensing data; the data fusion and processing module is used for receiving the original sensing data, sequentially executing space-time alignment correction, multi-source feature fusion and instruction semantic analysis, and outputting an environment sensing result and a structured control instruction; the motion planning and decision-making module is used for executing global path planning, dynamic obstacle trajectory prediction and motion parameter decision-making based on the environment perception result and the structured control instruction, and outputting a motion control instruction containing a motion mode, a speed and an angle; and the motion control execution module is used for receiving the motion control instruction, converting the motion control instruction into an execution mechanism driving signal, realizing closed-loop control through real-time state feedback and outputting a motion execution result.

Description

Technical Field

[0001] This invention belongs to the field of intelligent robot control technology, and more specifically relates to a multimodal motion control system, method, device and medium for a cleaning robot. Background Technology

[0002] With the rapid development of intelligent equipment technology, cleaning robots, as important automated equipment in home services and industrial operations and maintenance, have expanded their application scenarios from simple indoor environments to diverse scenarios such as complex homes, large office spaces, and industrial workshops. However, the current mainstream cleaning robot technology solutions still have significant limitations. The core problems lie in the limited environmental perception dimension, weak data processing capabilities, and rigid motion control strategies, making it difficult to meet the needs of efficient operation in complex dynamic environments. In everyday life scenarios such as homes and offices, the environmental adaptability deficiencies of traditional cleaning robots are particularly prominent. Existing products often rely on a single sensor to build their perception system, such as using only LiDAR for map building and obstacle detection, or simply relying on visual sensors for target recognition. This single-modal perception approach is susceptible to environmental interference: LiDAR is prone to ranging errors in strong light or specular reflection environments, while visual sensors are affected by changes in lighting, leading to misjudgments of targets and causing problems such as furniture collisions and missed cleaning areas. More importantly, when facing dynamic obstacles such as people walking or pets, traditional systems lack effective trajectory prediction and real-time obstacle avoidance mechanisms, often resulting in "following collisions" or "stalling and waiting to avoid," severely impacting cleaning efficiency and user experience. Industrial settings place higher demands on the motion control precision and environmental robustness of cleaning robots, but existing technologies have significant shortcomings. In factory workshops, warehouses, and logistics environments, the ground may contain various contaminants such as oil stains and metal shavings. Traditional robots' single cleaning modes cannot adjust operating parameters accordingly. Simultaneously, large equipment and temporarily stacked materials in industrial environments create complex static obstacles, while mobile devices such as forklifts and AGVs constitute high-dynamic interference sources. Existing path planning algorithms struggle to balance global efficiency with local obstacle avoidance requirements, often resulting in excessively long paths or delayed obstacle avoidance responses. Furthermore, the voice interaction requirements in industrial settings are more complex; traditional robots only support simple command recognition and cannot parse complex task commands that include area division and priority settings. From a technical perspective, the core bottleneck of existing cleaning robots lies in the lack of deep fusion capabilities for multi-source data and dynamic decision-making mechanisms. Sensor data lacks spatiotemporal calibration and feature complementation processing, resulting in blind spots and noise in environmental perception. The lack of real-time closed-loop feedback between motion control commands and environmental changes causes the robot to exhibit "mechanical execution" characteristics in complex scenarios. These technical deficiencies directly lead to insufficient intelligence and reliability in cleaning robots, limiting their further application in high-end service and industrial fields. Therefore, there is an urgent need to construct a new motion control system that integrates multimodal perception, intelligent fusion, and dynamic control. Summary of the Invention

[0003] To address the above problems, the present invention aims to provide a multimodal motion control system, method, device, and medium for cleaning robots, which can integrate multiple sensor data and control commands to achieve intelligent, efficient, and safe motion control of cleaning robots in complex environments, thereby improving the adaptability of cleaning robots to different scenarios and the quality of cleaning task completion.

[0004] To achieve the above objectives, the present invention employs the following technical solution: In a first aspect, embodiments of this application provide a multimodal motion control system for a cleaning robot, comprising: a multimodal data acquisition module, a data fusion and processing module, a motion planning and decision-making module, and a motion control execution module; The multimodal data acquisition module is used to synchronously acquire environmental images, distance information, voice signals and collision signals from multiple sources and output raw sensor data. The data fusion and processing module is used to receive raw sensor data, sequentially perform spatiotemporal alignment correction, multi-source feature fusion and instruction semantic parsing, and output environmental perception results and structured control instructions. The motion planning and decision-making module is used to perform global path planning, dynamic obstacle trajectory prediction and motion parameter decision-making based on environmental perception results and structured control commands, and output motion control commands including motion modes, speed and angle. The motion control execution module is used to receive motion control commands, convert them into actuator drive signals, realize closed-loop control through real-time status feedback, and output motion execution results.

[0005] In one optional implementation, the multimodal data acquisition module includes a visual sensor, a lidar, an ultrasonic sensor, a microphone, and a contact sensor. The vision sensor is mounted on the top center of the robot and uses a wide-angle camera to acquire environmental images at a first preset frequency. After Gaussian filtering and distortion correction, the environmental images are input into the YOLOv8 target detection model, which outputs the target category and 2D pixel coordinates, and determines the target's motion state variables. The target category includes obstacle targets and clean targets, and the motion state variables include target position and target velocity. The lidar is installed at the center of the robot and performs a 360° rotation scan at a second preset frequency to acquire distance data within a first preset range. After statistical filtering, the distance data within the first preset range is converted into a three-dimensional point cloud with the robot center as the origin and the forward direction as the X-axis. The ultrasonic sensors consist of six units, evenly distributed around the robot, and are used to collect distance data within a second preset range at a third preset frequency. After performing sliding window averaging on the distance data within the second preset range, the smoothed obstacle distance value is output. The microphone is a 4-microphone array, installed at the front of the robot, used to collect voice signals at a preset sampling rate, and convert the voice signals into a mono audio stream after noise suppression and echo cancellation. The contact sensors consist of eight piezoelectric contact sensors distributed along the edge of the robot's shell. They are used to output a 3.3V high-level signal when a collision force exceeding a preset threshold is detected.

[0006] In an optional implementation, the data fusion and processing module is specifically used for: Based on the robot system clock, timestamps are added to the original sensor data, and the 2D pixel coordinates, 3D point cloud and obstacle distance values ​​are uniformly converted to the robot body coordinate system through a coordinate transformation matrix. Multi-sensor data fusion: The Kalman filter algorithm is used to fuse visual and lidar data. The three-dimensional point cloud is used as the observation value and the motion state of the target is used as the state variable. A uniform motion model state transition matrix F is established to obtain the optimal estimated position of the target. A Bayesian network is used to fuse LiDAR and ultrasonic data. Distance data within a first preset range and distance data within a second preset range are used as evidence nodes. The posterior probability of the existence of an obstacle is calculated using a pre-trained conditional probability table. When the posterior probability is greater than the probability threshold, it is determined to be a valid obstacle. The mono audio stream is input into the GPT-4O speech recognition model and converted into text commands. The BERT semantic analysis model is then used to perform intent recognition on the text commands, and structured control commands are generated by combining them with a pre-set scene dictionary.

[0007] In an optional implementation, the motion planning and decision-making module is specifically used for: Using a LiDAR point cloud grid map constructed based on 3D point cloud as input, the target cleaning area's geometric center in the structured control command as the endpoint, and the robot's current pose as the starting point, the A* algorithm is used to plan the optimal path. The heuristic function of the algorithm is set to the Manhattan distance, and the cost function is set to F=0.6L+0.4D; where L is the comprehensive path length, D is the obstacle safety distance, and the safety distance D≥30cm; the A* algorithm outputs the coordinate sequence of the globally optimal path and the theoretical motion time. When the speed of the effective obstacle is greater than 0.5 m / s, a motion state equation including position, velocity and acceleration is established based on the Kalman filter algorithm to predict the trajectory of the effective obstacle in the next 2 seconds, which is then used as the predicted trajectory. The minimum distance between the predicted trajectory and the global optimal path is calculated using a collision detection algorithm. When the minimum distance is greater than or equal to 30 cm, the original path is maintained. When the minimum distance is less than 30 cm, local path replanning is triggered, and the Dijkstra algorithm is used to generate a detour path with a safe distance greater than or equal to 30 cm. The robot's motion modes are divided into straight lines, curves, and rotations in real time based on the robot's global path curvature and obstacle density. The speed parameters are dynamically adjusted in conjunction with the task priority, and corresponding speed and angle commands are output.

[0008] In one optional implementation, the real-time classification of the robot's motion modes into straight lines, curves, and rotations based on the robot's global path curvature and obstacle density, and the dynamic adjustment of the robot's speed parameters in conjunction with task priority, includes: If there are no obstacles in the path segment of 5 consecutive grids on the robot's current movement path and the distance between obstacles on both sides is greater than or equal to 1m, then control the robot to move in a straight line at a speed of 0.3m / s. If the path width on the robot's current movement path is 0.5-1m, then control the robot to move in a curved path with a turning radius of not less than 0.3m at a speed of 0.15m / s. If the robot is currently in a turning or fixed-point cleaning scenario, control the robot to rotate at an angular velocity of 0.5 rad / s; Combined with the task priority in the structured control instructions, the parameters are dynamically adjusted. If the robot is currently in the key cleaning mode, the robot's movement speed is set to 0.15m / s. If the robot is currently in the fast cleaning mode, and there are no obstacles in the path of 5 consecutive grids on the robot's current movement path and the distance between obstacles on both sides is greater than or equal to 1m, the robot is controlled to move in a straight line at a speed of 0.4m / s.

[0009] In an optional implementation, the motion control execution module is specifically used for: The motion control commands for speed and angle are converted into motor control signals, and the PWM duty cycle of the robot's built-in motor is proportional to the target speed. The robot's drive wheel encoders provide feedback on the actual rotation speed and steering angle at a frequency of 100Hz, and a PID controller is used in real time to adjust the robot's speed and angle to control the error between the actual rotation speed and steering angle and the corresponding target values. When the contact sensor triggers and outputs a 3.3V high-level signal, the motor output is cut off within 100ms, the PWM duty cycle is set to 0, and a replanning signal is sent to the motion planning module. The motion is restarted after the obstacle is removed.

[0010] In one optional implementation, the first preset frequency is 30fps, the second preset frequency is 30fps, and the third preset frequency is 16kHz.

[0011] Secondly, embodiments of this application also provide a multimodal motion control method for a cleaning robot, including: Simultaneous acquisition of environmental images, distance information, voice signals, and collision signals from multiple sources, and output of raw sensor data; It receives raw sensor data, sequentially performs spatiotemporal alignment correction, multi-source feature fusion, and instruction semantic parsing, and outputs environmental perception results and structured control instructions; Based on environmental perception results and structured control commands, global path planning, dynamic obstacle trajectory prediction and motion parameter decision-making are performed, and motion control commands including motion modes, speed and angle are output. It receives motion control commands, converts them into actuator drive signals, and achieves closed-loop control through real-time status feedback, outputting motion execution results.

[0012] Thirdly, embodiments of this application also provide an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the multimodal motion control method for cleaning robots as described in any of the above claims.

[0013] Fourthly, embodiments of this application also provide a storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the multimodal motion control method for a cleaning robot as described in any of the above claims.

[0014] As can be seen from the above technical solutions, the present invention has the following advantages: The multimodal motion control system for the cleaning robot provided in this application achieves multi-source synchronous acquisition of environmental images, distance, voice, and collision signals through a multimodal data acquisition module. Combined with the spatiotemporal alignment, Kalman filtering, and Bayesian network fusion algorithm of the data fusion and processing module, it significantly improves the comprehensiveness and accuracy of environmental perception, effectively distinguishes obstacles from cleaning targets, and achieves higher accuracy in voice command recognition and parsing. Through the motion planning and decision-making module, based on global path planning using the A* algorithm, dynamic obstacle trajectory prediction, and multimodal motion decision-making, and combined with task priority, it dynamically adjusts speed parameters to achieve optimal path planning and efficient dynamic obstacle avoidance in complex environments, greatly reducing the risk of collisions. Through the motion control execution module, it ensures motion accuracy and safety through PID closed-loop control and rapid collision emergency handling. The overall system significantly improves the environmental adaptability, cleaning efficiency, human-machine interaction flexibility, and operational stability of the cleaning robot in complex scenarios such as homes and industries.

[0015] This application enables the cleaning robot to fully perceive complex environments and accurately identify and respond to various static and dynamic obstacles and cleaning targets through multimodal data acquisition and fusion processing, greatly improving its adaptability in different scenarios. Whether in indoor environments such as homes and offices, or industrial environments such as factories and warehouses, it can efficiently complete cleaning tasks.

[0016] This application enables the cleaning robot to plan the optimal cleaning path through intelligent motion planning and decision-making, avoiding repeated cleaning and omissions, and improving cleaning efficiency. Simultaneously, it selects appropriate motion modes and cleaning intensities based on different cleaning tasks and surface materials to ensure cleaning quality.

[0017] This application enables users to interact with cleaning robots more conveniently by receiving and processing voice commands. Users can control the cleaning robot to perform specific tasks without a complicated operating interface, thus improving the user experience.

[0018] This application utilizes the collaborative work of multiple sensors and real-time monitoring and avoidance of dynamic obstacles to effectively prevent collisions during the movement of the cleaning robot, thus ensuring the safety of the robot itself and surrounding personnel and objects. Attached Figure Description

[0019] To more clearly illustrate the technical solution of the present invention, the accompanying drawings used in the description will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 This is a schematic diagram of the structure of the multimodal motion control system for the cleaning robot provided in this application.

[0021] Figure 2 This is a flowchart illustrating the multimodal motion control method for the cleaning robot provided in this application.

[0022] Figure 3 A schematic diagram of the structure of the electronic device provided in this application. Detailed Implementation

[0023] The various embodiments of this disclosure will be described more fully in the detailed system architecture and functions of the multimodal motion control system for the cleaning robot described below. This disclosure may have various embodiments, and adjustments and changes may be made therein. However, it should be understood that there is no intention to limit the various embodiments of this disclosure to the specific embodiments disclosed herein, but rather this disclosure should be understood to cover all adjustments, equivalents, and / or alternatives falling within the spirit and scope of the various embodiments of this disclosure.

[0024] In the following, the terms “comprising” or “may include”, which may be used in various embodiments of this disclosure, indicate the presence of the disclosed functions, operations, or elements, and do not limit the addition of one or more functions, operations, or elements. Furthermore, as used in various embodiments of this disclosure, the terms “comprising,” “having,” and their cognates are intended only to indicate a particular feature, number, step, operation, element, component, or combination of the foregoing, and should not be construed as primarily excluding the presence of one or more other features, numbers, steps, operations, elements, components, or combinations of the foregoing, or the possibility of adding one or more combinations of the foregoing.

[0025] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0026] Please see Figure 1 The diagram shown is a structural schematic of a multimodal motion control system for a cleaning robot in a specific embodiment. The system includes: a multimodal data acquisition module, a data fusion and processing module, a motion planning and decision-making module, and a motion control execution module. Each module achieves real-time data interaction through a data bus.

[0027] The multimodal data acquisition module is used to synchronously acquire environmental images, distance information, voice signals and collision signals from multiple sources and output raw sensor data.

[0028] In a specific implementation, the multimodal data acquisition module includes a visual sensor, a lidar sensor, an ultrasonic sensor, a microphone, and a contact sensor.

[0029] The vision sensor uses a wide-angle camera with a resolution of ≥20 megapixels and a field of view of ≥120°, mounted at the center of the top of the robot. The vision sensor is used to acquire environmental images in real time at 30fps. The acquired raw images are first denoised using a 5×5 Gaussian filter kernel, and then distortion correction is performed based on a pre-calibrated camera intrinsic parameter matrix. The corrected images are input into the YOLOv8 target detection model (input size 640×640, confidence threshold 0.7), which outputs category labels and 2D pixel coordinates for obstacles (such as furniture and walls) and cleaning targets (such as paper scraps and dust). Furthermore, the motion state variables (position and velocity) of the targets are extracted through continuous frame analysis. The LiDAR uses a 16-line LiDAR, installed at the center of the robot to ensure 360° scanning without blind spots. The LiDAR performs a rotating scan at 10Hz over a range of 0.1-10m, acquiring raw distance data with an accuracy ≤2cm. Statistical filtering is applied to the raw data to remove noise points within ±3 standard deviations of the mean distance, converting it into 3D point cloud data with the robot's center as the origin and the forward direction as the X-axis. This data is used for environmental mapping and obstacle localization. Six ultrasonic sensors are evenly distributed around the robot's outer shell at 60° intervals to collect near-range data within a range of 0.05-3m at 50Hz. The raw distance values ​​are smoothed using a sliding window averaging algorithm with a window size of 5 to output stable obstacle distance values, assisting in near-range collision warning. The microphone array consists of four microphones mounted on the robot's front end in the sensitive area for voice reception. These microphones acquire user voice signals at a 16kHz sampling rate, first performing noise suppression and echo cancellation through spectral subtraction, then converting the signal into a mono audio stream to provide clear input for subsequent voice command parsing. Eight piezoelectric contact sensors are mounted on the front, rear, left, and right edges of the robot's shell (two on each side). When a collision force ≥5N is detected, the sensors immediately output a 3.3V high-level signal with a response time ≤10ms, enabling real-time feedback of physical collisions.

[0030] The data fusion and processing module receives raw sensor data, sequentially performs spatiotemporal alignment correction, multi-source feature fusion, and instruction semantic parsing, and outputs environmental perception results and structured control instructions.

[0031] In a specific implementation, the data fusion and processing module is used to perform the following steps: 1. Spatiotemporal alignment correction: Based on the robot system clock (accuracy ±1ms), a unified timestamp is added to the visual image, laser point cloud, ultrasonic distance and voice signal to achieve temporal dimension alignment; through a pre-calibrated coordinate transformation matrix, the 2D pixel coordinates of the visual sensor, the 3D point cloud coordinates of the LiDAR and the distance data of the ultrasonic sensor are uniformly transformed to the robot body coordinate system (X-axis forward direction, Y-axis horizontal, Z-axis vertical upward), to complete the spatial dimension alignment. 2. Multi-sensor data fusion: Visual and LiDAR data fusion: The Kalman filter algorithm is used, with the 3D coordinates of the LiDAR point cloud as the observation value (observation noise covariance R=diag[0.02²,0.02²,0.05²]) and the motion state variables (position, velocity) of the visually recognized target as the state variables. A state transition matrix F based on a uniform motion model is established. The optimal estimated position of the target is calculated through prediction-update iteration, and the fusion error is ≤5cm. LiDAR and ultrasonic data fusion: Using a Bayesian network algorithm, the 0.1-10m ranging data of the LiDAR and the 0.05-3m ranging data of the ultrasonic sensor are used as evidence nodes. The posterior probability of the existence of the obstacle is calculated by calling the pre-trained conditional probability table. When the probability is ≥0.9, it is judged as a valid obstacle, thus eliminating the detection blind spot of a single sensor. 3. Voice command semantic parsing: The mono audio stream output from the microphone is input into the GPT-4O speech recognition model and converted into text commands (recognition accuracy ≥95%). The BERT semantic analysis model is used to perform intent recognition on the text commands. Combined with a preset scene dictionary (including room name, furniture type, and task type), structured control commands are generated, including the coordinates of the target cleaning area (e.g., "living room X:0-5m, Y:0-3m"), obstacle avoidance priority (e.g., "pet > furniture"), and task mode (e.g., "key cleaning" or "quick cleaning").

[0032] The motion planning and decision-making module is used to perform global path planning, dynamic obstacle trajectory prediction, and motion parameter decision-making based on environmental perception results and structured control commands, and output motion control commands including motion modes, speed, and angle.

[0033] In a specific implementation, the motion planning and decision-making module generates the optimal motion strategy based on environmental perception results and task instructions. The specific implementation steps are as follows: 1. Global Path Planning: Based on a 10cm×10cm grid map constructed from LiDAR point clouds, the static obstacle locations output by the data fusion module are marked as impassable areas. Starting from the robot's current pose (X0, Y0, θ0) and ending at the geometric center of the target cleaning area in the structured control command, the A* algorithm is used to plan the path. The algorithm's heuristic function is set to the Manhattan distance, and the cost function is set to F=0.6L+0.4D (L is the path length, and D is the safe distance from obstacles), where the safe distance D≥30cm. The output is the coordinate sequence of the globally optimal path (one path point every 10cm) and the theoretical motion time. 2. Dynamic Obstacle Trajectory Prediction and Avoidance: Real-time monitoring of the movement speed of effective obstacles. When a dynamic obstacle with a speed ≥ 0.5 m / s is detected, a motion state equation containing position (x, y), velocity (vx, vy), and acceleration (ax, ay) is established based on the Kalman filter algorithm to predict its trajectory within the next 2 seconds. The minimum distance between the predicted trajectory and the global path is calculated using a collision detection algorithm. If the minimum distance is ≥ 30 cm, the original path is maintained; if it is < 30 cm, local path replanning is triggered. The Dijkstra algorithm is used to generate a detour path with a safe distance ≥ 30 cm, ensuring that the detour time is ≤ 1 second. 3. Motion mode decision-making and velocity adjustment: Linear motion mode: When there are no obstacles in 5 consecutive grids on the current path segment of the robot and the distance between obstacles on both sides is ≥1m, it is determined to be an open area, and a linear motion command of 0.3m / s is output. Curved motion mode: When the path width is 0.5-1m (narrow passage), output curved motion command with a turning radius ≥0.3m, and the speed drops to 0.15m / s. Rotational motion mode: When in a turning or fixed-point cleaning scenario, output a rotational motion command with an angular velocity of 0.5 rad / s, and control the rotation angle error to ≤ ±2°. Priority dynamic adjustment: If the structured instruction is "key cleaning mode", the movement speed will be uniformly adjusted to 0.15m / s; if it is "fast cleaning mode" and the current area is open, the linear movement speed will be increased to 0.4m / s (not exceeding the safety threshold).

[0034] The motion control execution module receives motion control commands, converts them into actuator drive signals, implements closed-loop control through real-time status feedback, and outputs motion execution results.

[0035] In a specific implementation, the motion control execution module converts the planning instructions into mechanical actions and achieves precise control. The specific implementation steps are as follows: 1. Command Conversion and Motor Drive: Receives speed and angle commands from the motion planning module and converts them into PWM control signals through the motor drive circuit. The PWM duty cycle of the drive motor is proportional to the target speed (proportional coefficient 0.8), and the steering motor's rotation angle is proportional to the target angle (proportional coefficient 1.2), achieving precise control of wheel speed and steering angle. 2. Closed-loop feedback adjustment: An encoder with a resolution of 1024 lines / revolution is installed on the drive wheel to provide real-time feedback on the actual rotational speed (accuracy ±1%) and steering angle (accuracy ±1°) at a frequency of 100Hz. A PID controller is used to dynamically adjust the speed and angle, with the speed loop parameters set to Kp=0.5, Ki=0.1, Kd=0.05, and the angle loop parameters set to Kp=1.0, Ki=0.2, Kd=0.1, ensuring that the deviation between the actual value and the target value is ≤5%. 3. Collision emergency handling: When the contact sensor outputs a 3.3V high-level signal (collision force ≥5N), the system immediately triggers the emergency mechanism, cutting off the motor output within 100ms (PWM duty cycle set to 0), and simultaneously sending a collision signal to the motion planning and decision-making module; after the obstacle is removed and a replanning command is received, the motor is restarted and a new path is executed.

[0036] As can be seen, through the coordinated implementation of the above modules, cleaning robots can achieve precise environmental perception, dynamic path planning, and precise motion control in complex scenarios such as homes, offices, and industries, significantly improving cleaning efficiency and operational safety.

[0037] In this embodiment, a multimodal data acquisition module enables simultaneous acquisition of environmental images, distance, voice, and collision signals from multiple sources. The spatiotemporal alignment, Kalman filtering, and Bayesian network fusion algorithm of the data fusion and processing module improve the accuracy of environmental perception. Combined with GPT-4O and BERT models, it achieves accurate parsing of complex voice commands. The motion planning and decision-making module is based on global path planning using the A* algorithm, dynamic obstacle trajectory prediction, and multimodal motion decision-making. It dynamically adjusts speed parameters based on task priority to achieve efficient obstacle avoidance and path optimization in complex environments. The motion control execution module ensures motion accuracy and safety through PID closed-loop control and rapid collision emergency handling. Overall, it significantly improves the robot's environmental adaptability, cleaning efficiency, human-machine interaction flexibility, and operational stability in complex scenarios such as homes and industries.

[0038] like Figure 2 As shown, the following are embodiments of the multimodal motion control method for cleaning robots provided in this disclosure. This method and the multimodal motion control system for cleaning robots in the above embodiments belong to the same inventive concept. For details not described in detail in the embodiments of the multimodal motion control method for cleaning robots, please refer to the embodiments of the multimodal motion control system for cleaning robots described above.

[0039] A multimodal motion control method for a cleaning robot includes the following steps: S1: Simultaneously acquires environmental images, distance information, voice signals, and collision signals from multiple sources and outputs raw sensor data.

[0040] S2: Receives raw sensor data, sequentially performs spatiotemporal alignment correction, multi-source feature fusion, and instruction semantic parsing, and outputs environmental perception results and structured control instructions.

[0041] S3: Based on environmental perception results and structured control commands, it performs global path planning, dynamic obstacle trajectory prediction, and motion parameter decision-making, and outputs motion control commands including motion modes, speed, and angle.

[0042] S4: Receives motion control commands, converts them into actuator drive signals, and achieves closed-loop control through real-time status feedback, outputting motion execution results.

[0043] The multimodal motion control method for cleaning robots provided in this embodiment achieves a unified approach to cleaning robots, including comprehensive and accurate environmental perception, efficient and safe path planning, stable and accurate motion execution, and convenient and user-friendly human-machine interaction, through multi-source perception fusion, intelligent path planning, and dynamic motion control. This effectively improves cleaning efficiency and effectiveness.

[0044] Figure 3 A schematic diagram of the hardware structure of an electronic device for implementing various embodiments of the present invention.

[0045] The multimodal motion control method for cleaning robots provided in this application can be applied to electronic devices. Those skilled in the art will understand that the electronic device structure involved in the embodiments of this invention does not constitute a limitation on the electronic device. An electronic device may include more or fewer components than illustrated, or combine certain components, or have different component arrangements. In the embodiments of this invention, the electronic device includes, but is not limited to, laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the embodiments of this application described and / or claimed herein.

[0046] Electronic devices may include processors, external memory interfaces, internal memory, universal serial bus (USB) interfaces, charging management modules, power management modules, batteries, wireless communication modules, audio modules, speakers, microphones, sensor modules, buttons, cameras, displays, and SIM card interfaces, etc.

[0047] A processor may include one or more processing units, such as: a central processing unit (CPU), an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural network processing unit (NPU). Different processing units may be independent devices or integrated into one or more processors.

[0048] The processor can serve as the nerve center and command center of an electronic device. The controller can generate operation control signals based on the instruction opcode and timing signals to control the fetching and execution of instructions.

[0049] The processor may also include memory for storing instructions and data. In some embodiments, the memory in the processor is a cache memory. This memory can store instructions or data that the processor has just used or that are used repeatedly. If the processor needs to use the instruction or data again, it can retrieve it directly from this memory. This avoids repeated accesses, reduces processor latency, and thus improves system efficiency.

[0050] An external storage interface (ESI) can be used to connect external memory cards, such as microSD cards, to expand the storage capacity of electronic devices. The external memory card communicates with the processor through the ESI to perform data storage functions, such as saving music and video files on the external memory card.

[0051] Internal memory can be used to store computer executable program code, which includes instructions. The processor executes various functional applications and data processing of electronic devices by running the instructions stored in internal memory. Internal memory can include a program storage area and a data storage area. Internal memory can include high-speed random access memory, and can also include non-volatile memory, such as at least one disk storage device, flash memory device, universal flash storage (UFS), etc.

[0052] Wireless communication functionality in electronic devices can be achieved through antennas, wireless communication modules, modem processors, and baseband processors.

[0053] Wireless communication modules can provide solutions for wireless communication applications in electronic devices, including wireless local area networks (WLANs) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), and infrared (IR) technologies.

[0054] Electronic devices can implement audio functions through audio modules, speakers, receivers, microphones, headphone jacks, and application processors.

[0055] Electronic devices can achieve shooting functions through ISPs, cameras, video codecs, GPUs, displays, and application processors.

[0056] Electronic devices can achieve display functions through GPUs, displays, and application processors.

[0057] A GPU is a microprocessor for image processing, connected to the display screen and application processor. GPUs are used to perform mathematical and geometric calculations for graphics rendering. A processor may include one or more GPUs, which execute program instructions to generate or modify display information.

[0058] A display screen is used to display images, videos, etc. A display screen includes a display panel.

[0059] The aforementioned electronic device realizes the multimodal motion control method for cleaning robots in this application. Through multimodal data acquisition and fusion processing, it achieves comprehensive environmental perception, intelligent path planning and dynamic motion decision-making to ensure efficient obstacle avoidance, and precise closed-loop control to ensure stable execution. At the same time, it combines voice interaction to improve the ease of operation, thus achieving the beneficial effects of comprehensive perception, efficient planning, precise control, and user-friendly interaction for cleaning robots, while taking into account both cleaning efficiency and effectiveness.

[0060] The storage medium provided in this application stores a program product capable of implementing a multimodal motion control method for a cleaning robot.

[0061] Multimodal motion control methods for cleaning robots include: Simultaneous acquisition of environmental images, distance information, voice signals, and collision signals from multiple sources, and output of raw sensor data; It receives raw sensor data, sequentially performs spatiotemporal alignment correction, multi-source feature fusion, and instruction semantic parsing, and outputs environmental perception results and structured control instructions; Based on environmental perception results and structured control commands, global path planning, dynamic obstacle trajectory prediction and motion parameter decision-making are performed, and motion control commands including motion modes, speed and angle are output. It receives motion control commands, converts them into actuator drive signals, and achieves closed-loop control through real-time status feedback, outputting motion execution results.

[0062] In some possible implementations, the multimodal motion control method for cleaning robots disclosed herein can be implemented as a program product comprising program code that, when the program product is run on a terminal device, causes the terminal device to perform the steps described in the "Exemplary Methods" section of this specification according to various exemplary embodiments of this disclosure.

[0063] The storage medium disclosed herein may be any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples (a non-exhaustive list) of readable storage media include: an electrical connection having one or more wires, a portable disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof.

[0064] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A multimodal motion control system for a cleaning robot, characterized in that, include: Multimodal data acquisition module, data fusion and processing module, motion planning and decision-making module, motion control execution module; The multimodal data acquisition module is used to synchronously acquire environmental images, distance information, voice signals and collision signals from multiple sources and output raw sensor data. The data fusion and processing module is used to receive raw sensor data, sequentially perform spatiotemporal alignment correction, multi-source feature fusion and instruction semantic parsing, and output environmental perception results and structured control instructions. The motion planning and decision-making module is used to perform global path planning, dynamic obstacle trajectory prediction and motion parameter decision-making based on environmental perception results and structured control commands, and output motion control commands including motion modes, speed and angle. The motion control execution module is used to receive motion control commands, convert them into actuator drive signals, realize closed-loop control through real-time status feedback, and output motion execution results.

2. The multimodal motion control system for the cleaning robot according to claim 1, characterized in that, The multimodal data acquisition module includes a visual sensor, a lidar, an ultrasonic sensor, a microphone, and a contact sensor. The vision sensor is mounted on the top center of the robot and uses a wide-angle camera to acquire environmental images at a first preset frequency. After Gaussian filtering and distortion correction, the environmental images are input into the YOLOv8 target detection model, which outputs the target category and 2D pixel coordinates, and determines the target's motion state variables. The target category includes obstacle targets and clean targets, and the motion state variables include target position and target velocity. The lidar is installed at the center of the robot and performs a 360° rotation scan at a second preset frequency to acquire distance data within a first preset range. After statistical filtering, the distance data within the first preset range is converted into a three-dimensional point cloud with the robot center as the origin and the forward direction as the X-axis. The ultrasonic sensors consist of six units, evenly distributed around the robot, and are used to collect distance data within a second preset range at a third preset frequency. After performing sliding window averaging on the distance data within the second preset range, the smoothed obstacle distance value is output. The microphone is a 4-microphone array, installed at the front of the robot, used to collect voice signals at a preset sampling rate, and convert the voice signals into a mono audio stream after noise suppression and echo cancellation. The contact sensors consist of eight piezoelectric contact sensors distributed along the edge of the robot's shell. They are used to output a 3.3V high-level signal when a collision force exceeding a preset threshold is detected.

3. The multimodal motion control system for the cleaning robot according to claim 2, characterized in that, The data fusion and processing module is specifically used for: Based on the robot system clock, timestamps are added to the original sensor data, and the 2D pixel coordinates, 3D point cloud and obstacle distance values ​​are uniformly converted to the robot body coordinate system through a coordinate transformation matrix. Multi-sensor data fusion: The Kalman filter algorithm is used to fuse visual and lidar data. The three-dimensional point cloud is used as the observation value and the motion state of the target is used as the state variable. A uniform motion model state transition matrix F is established to obtain the optimal estimated position of the target. A Bayesian network is used to fuse LiDAR and ultrasonic data. Distance data within a first preset range and distance data within a second preset range are used as evidence nodes. The posterior probability of the existence of an obstacle is calculated using a pre-trained conditional probability table. When the posterior probability is greater than the probability threshold, it is determined to be a valid obstacle. The mono audio stream is input into the GPT-4O speech recognition model and converted into text commands. The BERT semantic analysis model is then used to perform intent recognition on the text commands, and structured control commands are generated by combining them with a pre-set scene dictionary.

4. The multimodal motion control system for the cleaning robot according to claim 3, characterized in that, The motion planning and decision-making module is specifically used for: Using a LiDAR point cloud grid map constructed based on 3D point cloud as input, the target cleaning area's geometric center in the structured control command as the endpoint, and the robot's current pose as the starting point, the A* algorithm is used to plan the optimal path. The heuristic function of the algorithm is set to the Manhattan distance, and the cost function is set to F=0.6L+0.4D; where L is the comprehensive path length, D is the obstacle safety distance, and the safety distance D≥30cm; the A* algorithm outputs the coordinate sequence of the globally optimal path and the theoretical motion time. When the speed of the effective obstacle is greater than 0.5 m / s, a motion state equation including position, velocity and acceleration is established based on the Kalman filter algorithm to predict the trajectory of the effective obstacle in the next 2 seconds, which is then used as the predicted trajectory. The minimum distance between the predicted trajectory and the global optimal path is calculated using a collision detection algorithm. When the minimum distance is greater than or equal to 30 cm, the original path is maintained. When the minimum distance is less than 30 cm, local path replanning is triggered, and the Dijkstra algorithm is used to generate a detour path with a safe distance greater than or equal to 30 cm. The robot's motion modes are divided into straight lines, curves, and rotations in real time based on the robot's global path curvature and obstacle density. The speed parameters are dynamically adjusted in conjunction with the task priority, and corresponding speed and angle commands are output.

5. The multimodal motion control system for the cleaning robot according to claim 4, characterized in that, The real-time classification of the robot's motion modes into straight lines, curves, and rotations based on the robot's global path curvature and obstacle density, combined with the dynamic adjustment of the robot's speed parameters according to task priority, includes: If there are no obstacles in the path segment of 5 consecutive grids on the robot's current movement path and the distance between obstacles on both sides is greater than or equal to 1m, then control the robot to move in a straight line at a speed of 0.3m / s. If the path width on the robot's current movement path is 0.5-1m, then control the robot to move in a curved path with a turning radius of not less than 0.3m at a speed of 0.15m / s. If the robot is currently in a turning or fixed-point cleaning scenario, control the robot to rotate at an angular velocity of 0.5 rad / s; Combined with the task priority in the structured control instructions, the parameters are dynamically adjusted. If the robot is currently in the key cleaning mode, the robot's movement speed is set to 0.15m / s. If the robot is currently in the fast cleaning mode, and there are no obstacles in the path of 5 consecutive grids on the robot's current movement path and the distance between obstacles on both sides is greater than or equal to 1m, the robot is controlled to move in a straight line at a speed of 0.4m / s.

6. The multimodal motion control system for the cleaning robot according to claim 5, characterized in that, The motion control execution module is specifically used for: The motion control commands for speed and angle are converted into motor control signals, and the PWM duty cycle of the robot's built-in motor is proportional to the target speed. The robot's drive wheel encoders provide feedback on the actual rotation speed and steering angle at a frequency of 100Hz, and a PID controller is used in real time to adjust the robot's speed and angle to control the error between the actual rotation speed and steering angle and the corresponding target values. When the contact sensor triggers and outputs a 3.3V high-level signal, the motor output is cut off within 100ms, the PWM duty cycle is set to 0, and a replanning signal is sent to the motion planning module. The motion is restarted after the obstacle is removed.

7. The multimodal motion control system for the cleaning robot according to claim 3, characterized in that, The first preset frequency is 30fps, the second preset frequency is 30fps, and the third preset frequency is 16kHz.

8. A multimodal motion control method for a cleaning robot, characterized in that, The method employs the multimodal motion control system for the cleaning robot as described in any one of claims 1 to 7; The method includes: Simultaneous acquisition of environmental images, distance information, voice signals, and collision signals from multiple sources, and output of raw sensor data; It receives raw sensor data, sequentially performs spatiotemporal alignment correction, multi-source feature fusion, and instruction semantic parsing, and outputs environmental perception results and structured control instructions; Based on environmental perception results and structured control commands, global path planning, dynamic obstacle trajectory prediction and motion parameter decision-making are performed, and motion control commands including motion modes, speed and angle are output. It receives motion control commands, converts them into actuator drive signals, and achieves closed-loop control through real-time status feedback, outputting motion execution results.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the multimodal motion control method for cleaning robots as described in claim 8.

10. A storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the multimodal motion control method for the cleaning robot as described in claim 8.

Images