AMR autonomous navigation man-machine interaction system based on ROS2 and Qt and control method

Abstract

The invention provides an AMR autonomous navigation man-machine interaction system based on ROS2 and Qt and a control method, which are constructed based on an ROS2 robot operating system and a Qt application program framework and comprise an AMR initialization control module, an AMR state monitoring module, an AMR navigation task management module, an AMR message feedback module, an AMR security verification module and the like. All the modules work cooperatively through message communication and an event triggering mechanism to form a unified control closed loop, all the function modules are clear in responsibility, communication is conducted through a standard interface, and therefore the system is easy to expand, debug and maintain. Complex ROS2 node starting, state monitoring and task issuing functions are integrated in a unified graphical interface, one-key starting and one-stop monitoring are achieved, the operation threshold of an AMR system is greatly lowered, the transparency and observability of the system state are improved, and the reliability of the system in the industrial environment is enhanced.

Description

[0001] The application relates to the technical field of robot control and human-computer interaction, and in particular to an AMR autonomous mobile robot navigation human-computer interaction system and a control method based on a ROS2 and Qt application program framework. BACKGROUND

[0002] With the rapid development of intelligent manufacturing, smart logistics and flexible production modes, autonomous mobile robots (AMRs) have been widely used in warehouse, workshop, hospital and commercial service scenarios due to their high flexibility and autonomy.

[0003] At present, most AMR systems are developed based on a robot operating system. ROS1, as an early mainstream framework, has inherent defects in real-time performance, safety and distributed communication. Although ROS2 solves some of the above problems by introducing a data distribution service middleware, the underlying complexity still exists. For example, a patent with the patent name of a human-computer interaction system based on ROS2 and Qt and the patent announcement number CN115877949 A discloses a human-computer interaction system including a robot state monitoring unit, a robot motion state display unit, a robot position display unit, a real-time image display unit, a control instruction operation unit and the like. The communication between nodes is managed by a Qt program, and the communication data is transmitted to a message processing function in the form of a program message, so as to realize simple operation and smooth communication of human-computer interaction operation.

[0004] However, the control and monitoring of the existing AMR system are mostly dependent on command line terminals or simple interfaces with dispersed functions, and the following significant problems exist: 1) High operation threshold: The start of the system, node management and task publishing are seriously dependent on the familiarity of the operator with the ROS2 command, which is not conducive to the rapid start and deployment of non-professionals.

[0005] 2) System state is not intuitive: The robot body state, sensor data, navigation task progress and other information are scattered in different terminals and tools, and there is a lack of unified and integrated visual monitoring interface, which is low in operation and maintenance efficiency.

[0006] 3) Insufficient safety: Key operations lack effective safety verification mechanisms, and accidental pressing of the emergency stop button and task reset button may cause production interruption or safety accidents.

[0007] 4) Low module coupling degree: The initialization, state monitoring, task management and other functional modules lack a unified cooperative working mechanism, and the system integration degree and maintainability are poor.

[0008] Therefore, there is an urgent need in the field for an AMR human-computer interaction system and control method which can reduce the operation difficulty, provide full-process visual monitoring and have an operation safety protection mechanism. SUMMARY

[0009] Therefore, the application aims to provide a ROS2 and Qt-based AMR autonomous navigation human-computer interaction system and control method, which realizes one-key initialization, real-time state monitoring, safe task publishing and whole-process message feedback of the AMR system through a highly integrated and graphical way, thereby significantly improving the usability, safety and maintainability of the system.

[0010] In order to achieve the above technical purposes, the technical scheme adopted by the application is: The ROS2 and Qt-based AMR autonomous navigation human-computer interaction system is deployed in a computing device in communication connection with the AMR, and is constructed based on a ROS2 robot operating system and a Qt application framework, and the core is a highly integrated human-computer interaction module, which realizes bidirectional data transmission through loose coupling communication with the ROS2 system through an MQTT protocol; the module comprises: An AMR initialization control module: used for centralized control of starting and stopping of the AMR chassis, laser radar, camera and navigation system through button controls on the Qt interface, and management of connection and disconnection of Qt and the MQTT server. The AMR initialization control module starts and manages one or more ROS2 nodes through a QProcess object of Qt, and the ROS2 nodes comprise: an AMR chassis start node, a laser radar start node, a camera start node and an MQTT connection node.

[0011] An AMR state monitoring module: used for real-time monitoring and display of AMR running state, navigation task execution, battery power and MQTT connection state. The module receives state data topics from the ROS2 node through the MQTT protocol, and can trigger an alarm state when navigation failure or emergency stop and other abnormalities occur; the ROS2 node publishes a topic containing AMR state data to the MQTT server after successful communication between the AMR chassis and the MQTT; the MQTT server receives and forwards the topic to the Qt interface; the Qt interface updates and displays the AMR battery power and running state through a QLabel control after analyzing the message.

[0012] An AMR navigation task management module: used for publishing a navigation task signal to a ROS2 navigation node through the MQTT server to realize issuance of a target point and start of a navigation task.

[0013] An AMR message feedback module: used for real-time display of AMR system start messages, navigation process data and whole-process task logs through a QTextBrowser log output control, to realize visualization of ROS2 terminal messages, facilitating debugging and auditing.

[0014] AMR security verification module: for when the user triggers initialization, emergency stop, task reset and other preset key operations, a password input window is popped up for verification, and only after verification is passed, the corresponding operation is performed, effectively preventing misoperation.

[0015] And the above each module works cooperatively through message communication and event triggering mechanism to form a unified control closed loop.

[0016] And the application provides an AMR autonomous navigation control method based on the ROS2 and Qt-based AMR autonomous navigation human-computer interaction system; the method comprises the following steps: S1: system initialization: the AMR initialization control module starts the AMR chassis, laser radar, camera and navigation system in turn, and establishes a connection with the MQTT server. In this process, the Rviz2 visualization interface is started synchronously, and the real-time position of the AMR, laser radar point cloud information, global / local path planning information and camera picture are displayed in Rviz2; S2: state monitoring and feedback: the AMR state monitoring module acquires and displays the AMR state in real time, and the AMR message feedback module outputs system logs for the operator to monitor the system health status; S3: task publishing and execution: in response to user operation, the AMR navigation task management module publishes a navigation task signal, which is forwarded to the ROS2 navigation node after being parsed by the MQTT server, and finally drives the AMR to execute the navigation task.

[0017] Further, there is a logical dependency relationship between the modules in the method, for example, the navigation task management module is prohibited from publishing tasks before initialization is completed, ensuring the orderliness and safety of system operation.

[0018] Further, when it is detected that the AMR needs to be stopped or charged in the system initialization stage, all processes of the AMR need to be interrupted, and the interruption step is to close the navigation system first and then close the AMR. Advantages

[0019] Compared with the prior art, the application has the following remarkable advantages: 1), integration and ease of use: the complex ROS2 node startup, state monitoring and task publishing functions are integrated in a unified graphical interface, realizing "one-key startup" and "one-stop monitoring", greatly reducing the operation threshold of the AMR system.

[0020] 2) Real-time performance and transparency: By using MQTT to build an efficient data bridge between ROS2 and Qt, and combining QLabel status indicators and QTextBrowser log controls, operators are provided with real-time and intuitive system status feedback and complete operation logs, which improves the transparency and observability of system status.

[0021] 3) Security and Reliability: By introducing a password verification mechanism to protect critical operations, the system effectively avoids abnormal shutdowns or task interruptions caused by accidental human intervention, thus enhancing the system's reliability in industrial environments.

[0022] 4) Modularity and maintainability: The system adopts a highly cohesive and loosely coupled modular design, with clear responsibilities for each functional module. The modules communicate through standard interfaces, making the system easy to expand, debug, and maintain. Attached Figure Description

[0023] Figure 1 This is a schematic diagram of the overall architecture of the AMR autonomous navigation human-computer interaction system provided by the present invention.

[0024] Figure 2 This is a schematic diagram of the layout and functional partitioning of the Qt human-computer interaction interface provided by the present invention.

[0025] Figure 3 This is a flowchart of the AMR autonomous navigation control method provided by the present invention.

[0026] Figure 4 This is an interface diagram of the AMR autonomous navigation human-computer interaction system provided by the present invention. Detailed Implementation

[0027] The following embodiments provide a further detailed description of the present invention. It should be noted that the following embodiments are for illustrative purposes only and do not limit the scope of the invention. Similarly, the following embodiments are only some, not all, embodiments of the present invention, and all other embodiments obtained by those skilled in the art without inventive effort are within the scope of protection of the present invention.

[0028] See Figures 1 to 4 As shown, an autonomous navigation human-machine interaction system for AMRs based on ROS2 and Qt is presented, taking a material handling human-machine interaction system for warehouse AMRs as an example:

[0029] The operator first launches the application developed based on the Qt framework on the host computer (such as an industrial PC or touchscreen all-in-one machine). After the main interface loads, see [link to relevant documentation]. Figure 4As shown, the layout is divided into several areas: the top is the status monitoring display area (using multiple QLabel controls), the middle left is the initialization control button area, the bottom left is the navigation task publishing area, and the right side of the main interface is the display area for AMR message logs, navigation message logs, navigation task message logs, etc. (using QTextBrowser controls).

[0030] The Qt application starts its internal MQTT client, which establishes a TCP connection with the MQTT broker at a preset IP address and port (e.g., port 1883). Upon successful connection, the log area displays "MQTT connection successful".

[0031] The AMR initialization control module works as follows: The startup section creates a button to launch the ROS 2 launch file. This launch is triggered by a Qt button click event, using the Qt QProcess class to execute the ROS 2 launch command, which launches the corresponding launch file. The stop section creates a button to interrupt the previously launched ROS 2 launch file. This interruption is also triggered by a Qt button click event. The stop button's slot function calls `kill()` to send a specific signal (such as simulating Ctrl+C's SIGINT). Because the QProcess pointer was previously saved, this ensures accurate closure of the corresponding ROS 2 launch file.

[0032] Starting the AMR: Click the "Start AMR" button to launch the launch file. This file includes starting the AMR chassis, LiDAR, camera, and other sensors, and establishing communication between ROS2 and MQTT, as well as between Qt and MQTT. First, the ROS2 environment is initialized, and then the specified AMR chassis driver node is started. After the chassis node starts, the ROS2 terminal sends a connection request to MQTT. After communication with MQTT is completed, ROS2 publishes the robot's low-level status data (such as odometer readings, battery level, emergency stop status, etc.) to MQTT. MQTT receives the data and stores it in its database. Subsequently, the Qt status monitoring module can directly extract the status data from the MQTT database and update the AMR's operating status.

[0033] Then, the corresponding LiDAR driver nodes and camera driver nodes are started in sequence. After these nodes are started, the LiDAR and camera begin to publish LiDAR point cloud topics and image topics. The topics contain all their data content for subsequent navigation subscription.

[0034] During the startup process of each of the above nodes, the terminal information printed through standard output (stdout) and standard error (stderr) is captured in real time by the QProcess object and forwarded to the AMR message feedback module. Finally, it is scrolled and displayed in the QTextBrowser control on the right side of the interface, providing a detailed startup process and AMR control feedback log.

[0035] Starting the navigation system: Click the "Start Navigation" button. Qt starts the ROS2 Navigation2 navigation stack via QProcess. The navigation stack begins running, executing the command to start Rviz2, launching the Rviz2 visualization tool, loading a pre-configured Rviz2 configuration file (.rviz2), which has already set the content to be displayed and subscribed to the necessary sensor topics (such as LiDAR, camera, odometry) and pre-scanned map files. At this point, the Rviz2 window pops up, displaying: The static map of the warehouse previously scanned by LiDAR; The position and attitude of the AMR robot model are updated in real time by odometry information published by the chassis. The point cloud data from the lidar is overlaid on the map in the form of obstacles; The navigation algorithm generates a global cost map and a local cost map based on a static map, and generates a global planned path (green) and a local planned path (red) according to the navigation task. Displaying the camera view in a specific area in Rviz2; and so on.

[0036] At this time, the AMR status monitoring module receives the "Navigation system is ready" data forwarded from the ROS2 bridging node via MQTT, and updates the QLabel of "AMR navigation status" on the interface to "Ready".

[0037] When the AMR stops or charges, all AMR processes must be interrupted. The interruption procedure is to first shut down the navigation system and then shut down the AMR. First, click the "Shut Down Navigation" button. Qt sends a specific signal (such as SIGINT simulating Ctrl+C) to close the Navigation2 navigation stack and the Rviz window opened when navigation is started, and interrupts all navigation nodes. At this time, the AMR navigation status in the status monitoring module will display "Please start navigation...", which also means that the navigation system is completely shut down. Then, perform the AMR shutdown procedure. Similarly, click the "Shut Down AMR" button to shut down the previously started AMR chassis, LiDAR, camera, and other sensor processes and their corresponding nodes, and disconnect ROS2, Qt, and MQTT communication. The AMR running status in the status monitoring module will display "Please start AMR...", and the MQTT connection status will display "MQTT connecting...", then the AMR has been successfully shut down.

[0038] After the system is initialized and in normal condition, it enters the task execution phase.

[0039] Condition monitoring data stream: 1) ROS2 Data Publishing: The chassis node of AMR continuously publishes topics containing data such as battery level and operating status, which are subscribed to by a dedicated ROS2-MQTT bridge node.

[0040] 2) Message bridging and forwarding: The bridging node parses the received ROS2 messages (such as std_msgs / String or custom message types) and repackages them into a lightweight JSON format, and then publishes them to a preset topic through an MQTT client, such as publishing battery information to / amr / status / battery.

[0041] 3) Qt Data Parsing and Display: The Qt application's MQTT client subscribes to the ` / amr / status / #` topic. When it receives a message from ` / amr / status / battery`, it parses the JSON string, extracts the battery level value (assuming the current battery is 95%), and calls the UI update function to set the text of the corresponding AMR battery level QProgressBar control to "95%". Similarly, the robot's running status, MQTT connection status, etc., are updated in a similar way.

[0042] Safe navigation task publishing process: 1) In the 2D Pose Estimate or 2D Goal Estimate panel of Rviz2, the operator clicks and drags on the map to specify the target point pose (position and orientation) of the AMR.

[0043] 2) When the operator returns to the Qt interface, there are four buttons in the navigation task management module: "Navigation Task A", "Navigation Task B", "Navigation Task C" and "Navigation Task D". When we need to go to point A to pick up materials from the starting position, the staff clicks the "Navigation Task A" button on the interface. In this way, the navigation task is initially sent successfully and is waiting for MQTT to receive and process. Similarly, other navigation tasks are run in the same way.

[0044] 3) Security Verification: Clicking this button first triggers the AMR security verification module, which will immediately pop up a modal password input dialog box, blocking other interface operations.

[0045] 1. The operator enters the password and clicks "Confirm". The system then performs a secure comparison between the entered password and the pre-stored hash value.

[0046] Verification failed: A warning message pops up, and the navigation task publishing process is aborted.

[0047] Verification successful: The dialog box closes, and the process continues.

[0048] 2. Navigation Task Publication: After successful verification, the AMR navigation task management module begins operation. It encapsulates the target point's coordinates and orientation information into a structured JSON object, containing the target point's x and y coordinates and orientation quaternion in the map coordinate system. Then, through a connected MQTT client, it publishes this JSON string to a specified topic, such as / amr / navigation / goal.

[0049] 3. Navigation Task Execution: The navigation node on the ROS2 side subscribes to the ` / amr / navigation / goal` topic. After receiving an MQTT message, it parses the target point pose and converts it into the ROS2 `geometry_msgs / PoseStamped` message format. This is then handed over to the behavior tree of the Navigation2 stack for path planning and tracking control, and the AMR begins to move autonomously towards the target point.

[0050] End-to-end message feedback: Throughout the process, the AMR message feedback module continues to operate: it captures and displays the output of all ROS2 nodes started via QProcess. Simultaneously, by subscribing to MQTT feedback topics (such as / amr / navigation / feedback), it appends AMR messages, navigation messages, and navigation tasks (e.g., "Planning a route," "Approaching the target," "Reached the target," "Navigation failed: Unable to plan a route") to the corresponding log display area in real time. This creates a complete and traceable log record from system startup and status monitoring to task execution and result feedback.

[0051] III. Manifestation of Module Collaboration and Dependency Relationships: The modules of this system do not operate in isolation; their collaborative working mechanism is reflected in: Functional dependency: Before the AMR initialization control module is successfully completed (e.g., the chassis node is not started), the target buttons "Navigation Task A", "Navigation Task B", "Navigation Task C" and "Navigation Task D" in the AMR navigation task management module are disabled (grayed out) at the UI level, which eliminates the possibility of issuing tasks when the system is not ready from the interaction logic.

[0052] Data-driven: The updates of the AMR status monitoring module are entirely driven by real-time data received from ROS2 nodes via MQTT, ensuring the timeliness of status display.

[0053] Event Triggering: As a security gate, the AMR security verification module is triggered by the user's click event on a specific button, and the verification result determines whether subsequent control commands are executed.

Claims

1. An AMR autonomous navigation human-computer interaction system based on ROS2 and Qt, characterized in that, The system is built on the ROS2 robot operating system and the Qt application framework, and includes a human-machine interaction module, which includes: The AMR initialization control module is used to control the start and stop of the AMR chassis, LiDAR, camera, and navigation system through the Qt interface, as well as to control the connection and disconnection between Qt and the MQTT server. The AMR status monitoring module is used to monitor and display the AMR's operating status, navigation task execution status, battery level, and MQTT connection status in real time. The AMR navigation task management module is used to publish navigation task signals to the ROS2 navigation node through the MQTT server. The AMR message feedback module is used to display AMR system messages, navigation messages and task logs in real time through the log output control; The AMR security verification module is used to verify the password when the user triggers a preset key operation, and execute the corresponding operation only after the verification is successful. The modules mentioned above work together through message communication and event triggering mechanisms.

2. The AMR autonomous navigation human-computer interaction system based on ROS2 and Qt according to claim 1, characterized in that: The AMR initialization control module starts and manages one or more ROS2 nodes through Qt's QProcess object. The ROS2 nodes include: an AMR chassis startup node, a LiDAR startup node, a camera startup node, and an MQTT connection node.

3. The AMR autonomous navigation human-computer interaction system based on ROS2 and Qt according to claim 2, characterized in that: The monitoring data of the AMR status monitoring module comes from MQTT messages; after the AMR chassis successfully communicates with MQTT, the ROS2 node publishes a topic containing AMR status data to the MQTT server; the MQTT server receives and forwards the topic to the Qt interface; after parsing the message, the Qt interface updates and displays the AMR battery level and operating status through the QLabel control.

4. The AMR autonomous navigation human-computer interaction system based on ROS2 and Qt according to claim 3, characterized in that: When the AMR status monitoring module receives a navigation task failure or emergency stop switch trigger message via MQTT, the control status display control shows the robot alarm status.

5. The AMR autonomous navigation human-computer interaction system based on ROS2 and Qt according to claim 1, characterized in that: The AMR message feedback module receives and displays terminal messages output after the ROS2 node starts up through the QTextBrowser control, thereby visualizing the ROS2 node's operation process.

6. The AMR autonomous navigation human-computer interaction system based on ROS2 and Qt according to claim 1, characterized in that: When a user clicks a preset verification button in the Qt interface, the AMR security verification module pops up a password input window. The function associated with the button is only executed after the password verification is successful.

7. The control method for the AMR autonomous navigation human-computer interaction system based on ROS2 and Qt according to any one of claims 1-6, characterized in that: Includes the following steps: S1: System initialization: The AMR chassis, lidar, camera and navigation system are started sequentially through the AMR initialization control module, and a connection with the MQTT server is established. S2: Status monitoring and feedback: The AMR status is acquired and displayed in real time through the AMR status monitoring module, and the system log is output through the AMR message feedback module for monitoring purposes. S3: Task Issuance and Execution: In response to user operation, a navigation task signal is issued through the AMR navigation task management module. After being parsed by the MQTT server, the signal is forwarded to the ROS2 navigation node to drive the AMR to execute the navigation task.

8. The control method for the AMR autonomous navigation human-computer interaction system based on ROS2 and Qt according to claim 7, characterized in that: In step S1, after starting the AMR navigation system, the Rviz2 visualization interface is launched simultaneously, and the real-time position of the AMR, the lidar point cloud information, the global / local path planning information, and the camera image are displayed in Rviz2.

9. The control method for the AMR autonomous navigation human-computer interaction system based on ROS2 and Qt according to claim 7, characterized in that: The modules in the human-computer interaction module have logical dependencies on each other: before the AMR initialization control module completes initialization, the AMR navigation task management module is prohibited from issuing navigation task signals; the AMR status monitoring module continuously synchronizes and updates the AMR status throughout the entire system operation; and the AMR message feedback module records the entire process of operation message logs from initialization to task execution.

10. The control method for the AMR autonomous navigation human-computer interaction system based on ROS2 and Qt according to claim 7, characterized in that: If the system detects that the AMR needs to be shut down or charged during the system initialization phase, all AMR processes must be interrupted. The interruption procedure is to first shut down the navigation system and then shut down the AMR.

Images