A multi-level multi-modal end-cloud collaborative robot control system and method based on a smart phone

Abstract

The application discloses a multi-level, multi-modal, end-cloud collaborative robot control system and method based on a smart phone. The system takes the smart phone as the core processing unit and sensing unit of the robot, is connected to a motion control module (MCU) through a USB-OTG to serial port wired connection, and supports Bluetooth / WiFi / 4G / 5G wireless interaction. The method comprises the following steps: system initialization and locking into Kiosk mode; supporting five control modes of emergency stop, voice, gesture, remote and automatic, and establishing a five-level priority arbitration mechanism (emergency stop > voice > gesture > remote > automatic); generating control instructions in each mode and packaging into a custom lightweight binary protocol packet (frame header 0xAA 0x55 + packet type + ID + instruction code + data + verification), and sending to the MCU for execution through a wired link; the MCU returns the state in real time, forming a closed loop. The automatic control mode innovatively combines the phone vision (YOLO-Nano identifies stairs / slopes / obstacles) with the intrinsic ultrasonic data, realizes autonomous decision-making such as stair obstacle avoidance, emergency parking, slope deceleration, etc. The application uses the smart phone to replace the expensive main control board, solves the real-time problem through the heterogeneous collaborative architecture of 'phone decision + MCU execution', significantly reduces the cost of the robot and improves the intelligent level, and is suitable for the fields of service, education, companionship, health management and the like.

Description

Technical Field

[0001] This invention relates to the field of intelligent robot control technology, specifically to a method and system for achieving autonomous robot control through a multi-layered architecture, multimodal interaction, and end-to-cloud collaboration, using a smartphone as the core processing unit. Background Technology

[0002] Existing robot control systems have the following drawbacks: High cost: The unit price of ARM / DSP / FPGA main control boards ranges from hundreds to thousands of yuan, which restricts their widespread adoption.

[0003] Poor scalability: Adding features requires hardware replacement, resulting in a long development cycle.

[0004] Weak interactivity: It is difficult to integrate AI capabilities such as voice and gestures.

[0005] The contradiction between real-time and intelligence: A single processor cannot simultaneously meet the needs of millisecond-level motion control and computationally intensive decision-making.

[0006] The control mode is limited: it lacks a priority arbitration mechanism for multi-source commands (voice, gesture, automatic, remote), which can easily lead to conflicts.

[0007] Existing technologies such as CN108287642A utilize mobile phones for interaction, but the mobile phone only serves as the terminal, and the robot still requires an independent main control board. This does not solve the problem of hardware cost substitution, and there is no multi-source fusion decision-making or priority arbitration. Summary of the Invention

[0008] I. Purpose of the Invention This addresses the problems of high cost, conflicting real-time intelligence, conflicting instructions, and weak autonomous decision-making in existing robots. Technical solution

[0009] (I) System Architecture include: Smartphone (Perception and Decision-Making Layer): Fixed to the robot, equipped with front and rear cameras, microphone, 4G / 5G, Bluetooth / WiFi, and running a Kiosk mode APP.

[0010] Motion control module (execution and feedback layer): MCU (such as STM32) drives servos, motors, ultrasonic sensors, and IMU.

[0011] Communication layer: USB-OTG to serial port (wired, simultaneously power supply and data transmission, latency <200ms) + Bluetooth / WiFi (near field) + 4G / 5G (cloud).

[0012] (II) Control Methods Step 1 Initialization: The phone automatically starts the APP and locks the Kiosk when it boots up, establishing a wired serial port link.

[0013] Step 2 Priority Arbitration: Supports five modes, with priority from highest to lowest: Emergency Stop (ultrasonic / physical button) > Voice > Gesture > Remote > Automatic. The highest priority command will be executed in case of conflict.

[0014] Step 3: Multimodal perception and decision-making: Voice: Wake word (Porcupine) → Cloud ASR+NLU (DeepSeek) when there is network access / Offline (Vosk) when there is no network access → Generate instruction sequence.

[0015] Gestures: Front camera ≥15fps → MediaPipe extracts 21 key points → LSTM classification (confidence >0.8) → Mapping instructions.

[0016] Automatic (Multi-source fusion): The rear camera YOLO identifies stairs / slopes / obstacles and fuses ultrasonic data → Stairs: Stop + fly over obstacles; Obstacles at close range: Emergency stop (highest priority); Slopes: Decelerate; Flat areas: Proceed.

[0017] Remote: The mobile phone connects to the MQTT server via 4G / 5G to receive and forward user commands; optional live video streaming is also available.

[0018] Step 4: Encapsulation and Transmission: A custom binary protocol packet (frame header 0xAA, 0x55 + packet type + device ID + command code + data length + data + XOR checksum, little-endian) is sent via USB-OTG.

[0019] Step 5 Execution and Closed Loop: The MCU drives the actuator and transmits battery, motor, mileage, ultrasonic, and IMU data back; the mobile phone displays / broadcasts information and adjusts adaptively according to the status (such as low power saving mode). Beneficial effects

[0020] Low cost: Smartphones replace expensive main control boards, with the total cost of the device being less than 500 yuan.

[0021] Heterogeneous collaboration: mobile phone decision-making + MCU real-time control, low latency and high reliability of <200ms.

[0022] Five-level priority arbitration: The first-ever complete arbitration from emergency stop to automatic mode ensures safe and natural interaction.

[0023] Multi-source fusion decision-making: The fusion of mobile vision and ultrasonic waves enables obstacle crossing on stairs, emergency stopping, and deceleration on slopes. It can still operate basically even if a single sensor fails.

[0024] Network-adaptive voice: Automatic switching between cloud and offline modes, usable in all scenarios.

[0025] End-to-end remote control: MQTT persistent connection + real-time video, suitable for remote companionship and security.

[0026] Easy to expand: Adding new features only requires a software update (such as tongue diagnosis health management), without the need to modify the hardware. Attached Figure Description Figure 1 This invention presents a flowchart illustrating the overall control process. It demonstrates the complete flow from hardware configuration, system initialization, control mode selection, instruction generation and encapsulation, transmission and execution, to status feedback. Figure 2 This invention provides a schematic diagram of the hardware connection between a smartphone and the robot body. It details the connection between the smartphone and the STM32 motion control module via a USB-OTG to serial port adapter, as well as the connection relationships between the motion control module and the servo motors, electric motors, ultrasonic sensors, and IMU. Figure 3 This invention provides a custom communication protocol data packet format diagram. It details the frame header, packet type, device ID, command / status code, data length, data fields, checksum byte offset, and related descriptions. Figure 4 This invention presents a flowchart of the voice control mode decision-making process. It illustrates the complete process from voice acquisition, local wake word detection, network judgment, cloud / offline recognition, intent parsing, to instruction encapsulation. Figure 5 This invention presents a flowchart of the gesture control mode. It illustrates the complete process of front-facing camera acquisition, MediaPipe Hands key point extraction, LSTM classifier recognition, confidence level determination, and command mapping. Figure 6 This invention presents a flowchart of the multi-source perception fusion decision-making process for automatic control mode. It illustrates the complete process of target detection by front / rear cameras, acquisition by ultrasonic sensors, and comprehensive judgment by the fusion decision module (stairs / obstacles / slopes / flat surfaces). Figure 7 This invention illustrates a control mode priority arbitration diagram. It shows five priority levels: emergency stop (highest), voice control (high), gesture control (medium), remote control (low), and automatic control (lowest). All control modes are processed via a mobile app, and the communication method between the mobile phone and the robot is exclusively via an OTG wired serial port. Detailed Implementation Example 1 (Narrow-legged body-bound intelligent robot) Hardware configuration: An OPPO A8 smartphone (costing 232 yuan) is fixed to the robot's chest slot, featuring a quad-core CPU, 3GB RAM, an 8MP front camera / 12MP rear camera, 4G / Bluetooth / WiFi. The motion control module uses an STM32F103C8T (15 yuan), driving 24 PWM servos (total cost 240 yuan), 2 DC motors, 2 HC-SR04 ultrasonic sensors, and an IMU MPU6050. Communication and power supply: USB-OTG to serial port (CP2102), with the phone providing 5V / 500mA power to the STM32, and a serial port baud rate of 115200. An optional retractable quadcopter flight module (<200g) is available at the bottom. The total cost is approximately 517 yuan. Software configuration: The phone is flashed with a custom Android ROM, pre-installed with the "Beiyang Robotics Control Center" APP and set to Kiosk mode (auto-start on boot, disable back / home buttons). It integrates the Porcupine wake word engine ("Hello [username]"), Vosk offline recognition (backup), DeepSeek cloud API, MediaPipe Hands (15fps), LSTM gesture classifier (5 classes), YOLO-Nano object detection (30fps), and MQTT client. Workflow example: Example 1 (Voice Control): The user says, "Hello [user's name], take me to the kitchen." After the phone wakes up, it calls the DeepSeekAPI to parse the intent and returns the path "Forward 2 meters → Turn left → Forward 3 meters → Stop". The APP encapsulates each protocol packet (0xAA0x55, instruction code 0x10 = forward, data 2 meters) and sends it to the STM32 via the OTG serial port for execution. The mileage data is then transmitted back, and the process is announced via voice. Example 2 (Gesture Control): The user waves to the left in front of the front-facing camera. The phone runs MediaPipe Hands at 15fps to extract 21 3D key points, and LSTM identifies it as "turn left" with a confidence score of 0.9. The app sends "turn left 90 degrees" to the STM32 via Bluetooth (or USB), which is completed within 1 second with a latency of <200ms. Example 3 (Automatic Controlled Staircase Obstacle Crossing): During automatic inspection, the rear-mounted YOLO-Nano camera detects a "staircase" with a confidence level of <0.7. The fusion decision module first issues a "stop" signal, then an "switch flight module" signal. The rotors deploy, cross three steps, and then retract. Simultaneously, the ultrasonic sensors detect a pedestrian 0.3 meters to the side, triggering an emergency stop (highest priority) to avoid the obstacle. The system resumes operation after the pedestrian has passed. Example 4 (Remote Control): A user logs into their MQTT account remotely via another mobile phone, clicks "Forward," and starts the video. The cloud pushes the command to the robot's mobile phone, which is then forwarded via OTG to the STM32 to execute the forward movement. Simultaneously, the robot's mobile phone streams H.264 video, which the user can view in real time. Upon detecting an obstacle, the user can remotely click "Turn Left" to intervene. Example 5 (Priority Arbitration): When the robot is performing automatic inspection (lowest priority), if the user says "Stop" (high voice priority), it immediately takes over the parking position. Subsequently, an ultrasonic sensor detects an obstacle 20cm ahead, triggering an emergency stop (highest priority) unconditionally, overriding the voice command to brake. The robot returns to idle after the obstacle is removed. Implement 6 (linking health management and services) This implementation is based on the hardware of Example 1 and is achieved solely through software expansion. The user says, "Give me a tongue diagnosis," and the phone's rear camera takes a picture, which is then uploaded to the cloud via 4G to a ResNet50 tongue coating / tongue body classifier. An analysis report is returned in approximately 5 seconds (e.g., "Thick, greasy tongue coating, reddish tongue body → damp-heat constitution"). The app provides voice-guided health advice ("Reduce oily foods, increase exercise, drink more water") and automatically adds a reminder to the phone's calendar. The robot adjusts its behavior based on the user's constitution: for a Qi-deficient constitution, it reduces movement speed and strenuous activity; for a damp-heat constitution, it recommends a light diet and reminds the user to drink water. No hardware modifications are required, demonstrating excellent scalability.

Claims

1. A multi-level, multi-modal, edge-cloud collaborative robot control method based on smartphones, characterized in that, Includes the following steps: a. Fix the smartphone to the robot body as the core processing unit and sensing unit, and connect it to the motion control module of the robot body through a wired communication interface; b. After the smartphone is powered on, the customized robot control application will automatically start and lock into the dedicated starter (Kiosk mode). c. The system supports multiple control modes, including emergency stop mode, voice control mode, gesture control mode, remote control mode, and automatic control mode, and arbitrates according to preset priorities, in descending order of priority: emergency stop > voice control > gesture control > remote control > automatic control; d. Based on the selected control mode, the smartphone generates corresponding control commands and encapsulates them into a custom communication protocol data packet; e. The smartphone sends data packets to the motion control module via the wired communication interface; f. The motion control module parses the data packets and drives the robot to perform corresponding actions; g. The motion control module periodically collects the robot's status information and transmits it back to the smartphone through the same wired communication interface, forming a closed-loop control.

2. The method according to claim 1, characterized in that, The wired communication interface is a USB-OTG to serial port, through which the smartphone powers the motion control module and transmits data.

3. The method according to claim 1, characterized in that, The custom communication protocol data packet format includes: a fixed frame header (0xAA, 0x55), packet type, device ID, command / status code, data length, data fields, and XOR checksum; wherein the packet type includes command, status feedback, heartbeat, and file transfer.

4. The method according to claim 1, characterized in that, The voice control mode includes: local wake word detection, and automatic selection of cloud-based ASR+NLU or local offline command word recognition based on network conditions.

5. The method according to claim 1, characterized in that, The gesture control mode includes: capturing gesture images in real time using the front-facing camera of a smartphone, extracting key hand points using MediaPipe Hands, and recognizing preset gestures using an LSTM classifier.

6. The method according to claim 1, characterized in that, The automatic control mode includes: using the smartphone's rear camera to run a lightweight target detection model to identify stairs, flat roads, slopes, and obstacles, and then fusing the data with ultrasonic sensor data returned by the motion control module to make decisions.

7. The method according to claim 6, characterized in that, The fusion decision includes: ● When identifying stairs, generate "Stop" and "Switch Flight Module" commands; ● An "emergency stop" command is generated when an obstacle is detected and the distance is less than a safety threshold. This command has the highest priority. ● Generates a "decelerate" command when a slope is detected; ● Generates a "forward" command when a flat road surface is identified.

8. The method according to claim 1, characterized in that, The remote control mode includes: the smartphone maintains a long connection with the cloud MQTT server via 4G / 5G, receives remote commands from the user, and can push real-time video streams.

9. A multi-level, multi-modal, edge-cloud collaborative robot control system based on a smartphone, characterized in that, include: ●A smartphone, fixed to the robot body, serves as the core processing and sensing unit; ●A motion control module, including a microcontroller and its driven servo motors, motors, ultrasonic sensors, and IMU; ●A wired communication interface, which is a USB-OTG to serial port, connects the smartphone and the motion control module; ● One wireless communication interface, including Bluetooth, WiFi and 4G / 5G; The system is controlled by the method described in any one of claims 1 to 8.

10. The system according to claim 9, characterized in that, The robot body also integrates a retractable rotor flight module. When the automatic control mode detects stairs, the smartphone triggers the flight module to deploy and start through the motion control module, enabling obstacle crossing.

11. The system according to claim 9, characterized in that, The robot control application also integrates health management functions, using a smartphone camera to collect images of the user's tongue or face, analyzing the health status through a cloud-based AI model, and adjusting the robot's service behavior accordingly.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor of a smartphone, it implements the steps of the method according to any one of claims 1 to 8.

Images