A human-computer interaction system based on ROS2 and Qt

Abstract

A human-computer interaction system based on ROS2 and Qt includes a human-computer interaction unit based on the Qt and ROS2 operating systems, comprising: a robot state monitoring unit for monitoring the delay of a communication link, the robot battery voltage, and the device temperature; a robot motion state display unit for displaying the robot's motion state and detection information of the surrounding environment; a robot position display unit for displaying the robot's position; a real-time image display unit for displaying real-time images; and a control instruction operation unit for controlling communication nodes. The system inherits ROS2 nodes into the QThread class in Qt in the form of a class, manages node communication using the thread management function in the QThread class, and establishes a connection between ROS2 and Qt. Communication between nodes is managed by a Qt program, and communication data is transmitted to a message processing function in the form of program messages. The system has the advantages of simple operation, smooth communication, and a wide range of applications.

Description

Technical Field

[0001] The present invention mainly relates to the field of human-computer interaction technology, and in particular to a human-computer interaction system based on ROS2 and Qt. Background Art

[0002] Currently, the robotics field commonly uses the Robot Operating System (ROS) for robot design. It typically employs four communication methods, including topics and services, to achieve loosely coupled point-to-point connections between modules. This distributed architecture, with numerous tools and libraries, allows users and developers to develop and use the operating system directly, ignoring the underlying communication protocols and operating principles. However, ROS communication technology also has many shortcomings, such as its reliance on the ROS Master, which can lead to real-time and reliability issues.

[0003] ROS2 uses DDS communication to address these issues. DDS (Data Distribute Service) is a data-centric distributed real-time communication technology that provides QoS (Quality of Service) to ensure real-time, efficient, and flexible data distribution. DDS abandons the ROS Master, which is decentralized. This solves the problem of ROS communication relying on the master node and improves data transmission efficiency. In a distributed system, DDS sits between the operating system and the application, supporting multiple programming languages and multiple underlying protocols, eliminating the need to focus on the underlying communication protocol.

[0004] Since its emergence in 1980, human-computer interaction (HCI) has evolved through two major stages: command-line interfaces (CLIs) and graphical user interfaces (GUIs). With increasing emphasis on natural and convenient interactions, new forms of interaction, such as touch-based and three-dimensional (3D) interactions, have emerged and become future trends. Currently, HCI methods primarily include touch, voice, gesture, eye movement, virtual display input, multimodal, and intelligent interaction. While interfaces and methods continue to evolve at a system level, different interaction technologies have their own advantages and are not completely eliminated by the emergence of new technologies. Therefore, common HCI methods still combine multiple methods.

[0005] As an important window for long-range remote control, the human-computer interaction interface must ensure both practicality and friendliness. Currently, the most widely used interactive system is the multi-channel multimedia user interface. Tools suitable for developing interactive interfaces include Matlab, Labview, and Qt. From the perspective of interface development friendliness and the richness of functional modules, Qt is the preferred tool for interactive interface design. As a cross-platform C++ application development framework, it provides developers with the functions required to build graphical user interfaces and is widely used in the development of GUI programs. Its powerful interface development capabilities and user-friendly operation are its advantages in widespread application. Summary of the Invention

[0006] The technical problem to be solved by the present invention is: in response to the technical problems existing in the prior art, the present invention provides a human-computer interaction system based on ROS2 and Qt that is simple to operate, has smooth communication, and has a wide range of applications.

[0007] A human-computer interaction system based on ROS2 and Qt includes a human-computer interaction unit based on Qt and ROS2 operating systems, wherein the human-computer interaction unit includes: a robot state monitoring unit for monitoring the delay of a communication link, the robot battery voltage, and the device temperature; a robot motion state display unit for displaying the robot's motion state and detection information of the surrounding environment; a robot position display unit for displaying the robot's position; a real-time image display unit for displaying real-time images; and a control instruction operation unit for controlling communication nodes. The system inherits ROS2 nodes into the QThread class in Qt in the form of a class, manages node communication using the thread management function in the QThread class, and establishes a connection between ROS2 and Qt. The system manages communication between nodes through a Qt program, transmits communication data to a message processing function in the form of program messages, and realizes the association between the node communication data and a GUI window.

[0008] As a further improvement of the system of the present invention: the human-computer interaction unit is used to realize the interaction between the robot's sensor information and the control information of the controller, control the remote robot through the interface buttons and / or receive the information sent back by the robot in real time and display it to the controller.

[0009] As a further improvement of the system of the present invention: the sensor information and control information include control instruction information, robot motion information, video image information, robot state monitoring information, and robot posture information.

[0010] As a further improvement of the system of the present invention: each module in the human-computer interaction unit is encapsulated in a communication node Node, and signals are transmitted through the DDS communication mechanism. The interactive information is bound to various controls in Qt Designer to realize information processing and display.

[0011] As a further improvement of the system of the present invention: the human-computer interaction unit is based on Qt Designer, the interaction process is thread managed by QThread, the communication message of the node Node is converted into a program message by the Qt system kernel, and sent to the slot Slot in the form of a signal Signal, and the custom slot function Slot_handle() is used to process the signal and interact with the UI interface window or control.

[0012] As a further improvement of the system of the present invention: the human-computer interaction unit regards the ROS2 node as a specific event based on the signal and slot mechanism. When the node processes the program or communicates, it is regarded as an event trigger. The communication data or program processing results are regarded as signals. The transmitted information is mapped to the main window through the customized signal slot to realize information interaction.

[0013] As a further improvement of the system of the present invention: the robot status monitoring unit includes two parts: communication link monitoring and equipment voltage and temperature monitoring; for the communication link monitoring part, half of the information transmission and reception time is used as the observation value; for the voltage and temperature monitoring part, the measuring element transmits the data back to the control end through the serial port, and uses the register to store the data read by the serial port, and then reads the data from the register and determines whether the data is normal. For abnormal data, the frame header is retained and the remaining part of the frame is cropped for processing.

[0014] As a further improvement to the system of the present invention: in the robot position display unit, the coordinates of the selected scene map are first measured to produce a global map; then the actual coordinates of the robot are converted into the global map to display the relative position of the robot; and different colors are used to distinguish different terrains or objects.

[0015] As a further improvement of the system of the present invention: the robot motion state display unit is used to display the robot's motion state and detection information of the surrounding environment, including motion speed and angular velocity, speed gear and angular velocity gear, metal detection signal and digital recognition signal.

[0016] As a further improvement of the system of the present invention: the interactive interface of the real-time image display unit adopts three-way video information: forward view, rear view, and mobile view, wherein the forward view and rear view cameras are fixed cameras, and their optimal field of view angles are obtained by adjusting the camera installation position, which are responsible for observing the front and rear environment of the robot; the mobile camera is installed above the mechanical claw and is responsible for dynamically observing the surrounding environment.

[0017] Compared with the prior art, the advantages of the present invention are:

[0018] 1. The human-computer interaction system based on ROS2 and Qt of the present invention is simple to operate, has smooth communication, and has a wide range of applications. In response to user interaction needs and existing communication conditions, the present invention designs a graphical human-computer interaction system based on ROS2 and Qt suitable for long-range control of robot operations. It adopts the new DDS communication method to realize information transmission, and uses Qt's signal and slot mechanism to complete the conversion and display of interactive information.

[0019] 2. The present human-machine interaction system based on ROS2 and Qt utilizes ROS2's unique DDS communication method for human-machine communication and Qt's excellent cross-platform and user-friendly interface design. It enables interaction between the control end and the robot end through a node network and a signal and slot mechanism. Compared with the interaction method based on ROS1, the present invention uses the more efficient and reliable DDS communication method, while Qt's rich interface design modules enable diverse information processing, making it suitable for controlling robots at long distances. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 It is a schematic diagram of the present invention after being implemented in a specific application example.

[0021] Figure 2 This is a schematic diagram of a robot status monitoring unit in a specific application example of the present invention. Figure 1 .

[0022] Figure 3 This is a schematic diagram of a robot status monitoring unit in a specific application example of the present invention. Figure 2 .

[0023] Figure 4 It is a schematic diagram of a robot motion state unit in a specific application example of the present invention.

[0024] Figure 5 It is a schematic diagram of a robot position display unit in a specific application example of the present invention.

[0025] Figure 6 It is a schematic diagram of a real-time image display unit in a specific application example of the present invention.

[0026] Figure 7 It is a schematic diagram of a control instruction operation unit in a specific application example of the present invention.

[0027] Figure 8 It is a schematic diagram of the ROS2 (DDS) communication model in a specific application example of the present invention.

[0028] Figure 9 It is a schematic diagram of the relationship between communication nodes based on ROS2 in a specific application example of the present invention.

[0029] Figure 10 It is a schematic diagram of a communication delay measurement method in a specific application example of the present invention.

[0030] Figure 11 It is a schematic diagram of a global map display interface in a specific application example of the present invention. DETAILED DESCRIPTION

[0031] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0032] The present invention's human-computer interaction system, based on ROS2 and Qt, employs a graphical interaction method, primarily suitable for robot control. It primarily considers the interaction between the robot's sensor information and the operator's control information, including control instructions, robot motion information, video images, robot status monitoring information, robot position information, and other sensor information. Specifically, the system allows for remote robot control via interface buttons, as well as real-time reception of information transmitted by the robot and display of this information to the operator.

[0033] like Figures 1-11 The figure is a schematic diagram of the present invention after being implemented in an application example. The human-machine interaction system of the present invention includes a robot state monitoring unit, a real-time image display unit, a robot motion state display unit, a robot position display unit and a control instruction operation unit; wherein:

[0034] The robot status monitoring unit is responsible for monitoring communication link delay, robot battery voltage, and device temperature. This unit includes two components: communication link monitoring and device voltage and temperature monitoring. This unit primarily considers the robot's operating status during operation, allowing operators to promptly monitor the robot's operating status and prevent loss of control or damage.

[0035] The robot's motion status display unit is used to display the robot's motion status and detection information of the surrounding environment, including motion speed and angular velocity, speed and angular velocity gears, metal detection signals, and digital recognition signals. This takes into account the robot's execution of control commands and its perception of the surrounding motion environment.

[0036] The robot position display unit is used to display the robot's position. It mainly considers the robot's actual position under the global map. The control personnel predict the next action based on its position information and the target point position information. That is, the coordinates of the selected scene map are first measured to produce a global map; then the robot's actual coordinates are converted into the global map to display the robot's relative position; at the same time, different colors are used to distinguish different terrains or objects.

[0037] The real-time image display unit is used to display real-time images. This is mainly because the operator needs to understand the surrounding environment information when controlling the robot at a long distance, and visual information is the richest and most intuitive interactive signal.

[0038] The control command operation unit is used to control communication nodes. It primarily allows operators to remotely manage the robot and control communication nodes, such as starting and shutting them down. This functionality uses the system() function in C / C++, which executes specified command lines by calling a subprocess.

[0039] The present invention is designed based on Qt and ROS2 operating systems. The present invention inherits ROS2 nodes into the QThread class in Qt in the form of a class, thereby using the thread management function in the QThread class to manage node communication, thus establishing a connection between ROS2 and Qt; when the Qt program can manage communication between nodes, the present invention can transmit communication data to the message processing function in the form of program messages, thereby realizing the association between the node communication data and the GUI window.

[0040] In a specific application example, the present invention is based on the signal and slot mechanism, and regards the ROS2 node as a specific event. When the node processes the program or communicates, it is regarded as an event trigger. The communication data or program processing results are regarded as signals. The transmitted information is mapped to the main window through the custom signal slot to realize information interaction.

[0041] In the above scheme of the present invention, ROS2 adopts DDS (Date Distribution Service) communication mode, a data publish / subscribe standard specially designed for distributed real-time systems. Its technical key is the data-centric publish / subscribe model (Date-Centric Publish-Subscribe, DCPS). This model creates a "global data space" (global data space). Nodes use the "Discovery" discovery mechanism and topic communication mechanism to establish communication connections, and all independent nodes and applications can access the global data space. Analysis of the data flow of the communication link shows that the functions that the communication node should have include sending control signals, reading control instructions and converting them into motor drive instructions, reading and returning robot status information, reading and returning robot environment information, and robot working status information that the control personnel need to know.

[0042] The control node issues motion control instructions (motion_cmd) and robotic arm control instructions (kinova_angular). The motion control instructions (including the robot's motion speed, angular velocity, and front and rear fin positions) are subscribed to the robot's motion_exec node, compared with the current motion state, and generate speed instructions for the tracks on both sides, which are issued to the motor controller Elmo. The robotic arm control instructions are subscribed to the teleop_kinova node, which, after parsing, issues the robotic arm's motion angle to control its rotation. The fin and arm postures are fed back to the control station's rviz node for display. In addition, the robot side also includes the camera node usb_cam (responsible for collecting image information), the metal detection node metal_detect (responsible for collecting metal signals and voltage and temperature information), and the combined navigation node sbg_node (responsible for collecting posture information and mapping information). Most of the information on the robot side is transmitted back to the control side's ui_ros2 node, the human-computer interaction node. The various sensor information received is displayed on the UI interface for the operator to make decisions.

[0043] In a specific application example, the real-time image display unit uses three channels of video information (forward, rear, and mobile), taking into account communication bandwidth and task execution needs. The forward and rear cameras are fixed cameras, whose optimal field of view can be adjusted by adjusting the camera mounting position. They are primarily responsible for observing the robot's front and rear environments. The mobile camera is installed above the mechanical claw and is responsible for dynamic observation of the surrounding environment. Considering human eye perception and communication link bandwidth costs, image compression is used to reduce data size and communication pressure. Image_theora (video stream) is used as the node communication topic.

[0044] In a specific application example, each interactive module, including control command information, robot motion information, video image information, robot status monitoring information, robot posture information, and other sensor information, is encapsulated in a communication node. Signals are transmitted through the DDS communication mechanism. The interactive information is bound to various controls in Qt Designer (such as Pushbutton) to realize information processing and display.

[0045] In a specific application example, the human-computer interaction system of the present invention is designed based on Qt Designer, the interaction process is thread-managed by QThread, the communication message of the node (Node) is converted into a program message by the Qt system kernel, and sent to the slot (Slot) in the form of a signal (Signal), and the custom slot function (Slot_handle()) processes the signal and interacts with the UI interface window or other controls.

[0046] In an application example of an interactive system in which a robot is controlled in a competition field, the robot status monitoring unit consists of two parts: communication link monitoring and device voltage and temperature monitoring.

[0047] For the communication link monitoring part, half of the information sending and receiving time is used as the observation value. That is, when joy2motion issues a control command, the control station system time t1 is recorded. When motor_ctrl receives the control command and immediately executes the callback function feedback information ( / time), ui_ros2 records the system time t2 again after receiving the feedback information. Therefore, the communication delay is T = (t2-t1) / 2.

[0048] For the voltage and temperature monitoring part, the measuring element transmits data back to the control end through the serial port ( / stm32). Taking into account the possibility of byte loss or frame drop during serial communication, a communication error correction mechanism is designed. That is, the data read by the serial port is stored in the register, and then the data is read from the register to determine whether the data is normal. For abnormal data, the frame header is retained and the remaining part of the frame is cropped. That is, the last frame header in the erroneous data is retained to ensure that the next data is read from the frame header.

[0049] In this example, there are two ways to collect the robot's motion status information. One is to collect the control signals v and ω of the handle, and the other is to collect the speed and angular velocity information fed back by the robot's actuator to improve the timeliness and stability of the status information display. The present invention chooses the first method, that is, directly subscribing to the control signal.

[0050] In this example, the collection of metal detection signals requires serial communication. In the past, the ros_serial function package was used as the serial API on the ROS system to realize data transmission and reception. In the new operating system ROS2, since there is no function package similar to serial, the present invention uses the Boost::Asio library to implement the input / output programming of nodes and serial ports, and the metal detection function mainly uses the io_service class in the library. From the perspective of functional implementation, on the one hand, the operator should be able to control the switch of the metal detector, and on the other hand, the metal detection signal must ensure that the operator can notice it and have time to record it. The digital recognition signal is the same as the metal detection signal, and the operator's misrecognition rate should be reduced and sufficient recording time should be provided.

[0051] In the robot position display part, the present invention constructs two Cartesian coordinate systems, which represent the map coordinate system and the robot coordinate system respectively. Figure 4The rectangle enclosed by the four vertices is the actual range of the competition venue. The GPS coordinates of the four vertices are converted to map coordinates. The upper left corner is the coordinate origin (0, 0), and the lower right corner is the coordinate point (E, N) converted from the longitude and latitude differences Lon_diff and Lat_diff of the competition venue. Define the robot's center position as the origin of the robot coordinate system, and the robot's orientation is the y-axis direction. Now assume that the longitude and latitude of the robot position are (lon, lat) and the azimuth is heading (°). The conversion formula from the robot's actual coordinates (lon, lat) to the map coordinate system (x, y) and the orientation θ can be expressed as:

[0052]

[0053]

[0054] θ=heading (3)

[0055] Relying on Qt's QGraphicsItem class, the ui_ros2 node subscribes to the robot's GPS position and orientation information ( / odom) collected by the sbg_node node, calculates the robot's global map position and orientation through the coordinate conversion formula, and updates the global map through the drawing operations in the custom Map1 class.

[0056] Real-time image display unit, three-way video information is published through the usb_cam node, the message type is sensor_msgs::Image, the published image information is transmitted back from the robot end to the control end and subscribed to by the ui_ros2 node, and then the opencv tool is used to convert the Image type to Mat format (RGB8), and finally the Qt tool is used to convert the Mat format to QImage format that can be displayed on Qlabel and displayed in the interactive interface.

[0057] The control instruction operation unit uses the signal and slot mechanism. The Pushbutton control is an event. By clicking the control, the click() function is executed, the event is triggered and a signal is sent. The corresponding slot function calls the system() function, uses gnome-terminal to open the terminal and execute user commands.

[0058] The above are merely preferred embodiments of the present invention. The scope of protection of the present invention is not limited to the above embodiments. All technical solutions based on the principles of the present invention are within the scope of protection of the present invention. It should be noted that for those skilled in the art, various improvements and modifications that do not depart from the principles of the present invention should be considered within the scope of protection of the present invention.

Claims

1. A human-computer interaction system based on ROS2 and Qt, characterized in that: The system comprises a human-computer interaction unit based on Qt and ROS2 operating systems, the human-computer interaction unit comprising: a robot state monitoring unit for monitoring the delay of the communication link, the robot battery voltage and the device temperature; a robot motion state display unit for displaying the robot's motion state and detection information of the surrounding environment; a robot position display unit for displaying the robot's position; a real-time image display unit for displaying real-time images; a control instruction operation unit for controlling communication nodes; inheriting ROS2 nodes into the QThread class in Qt in the form of a class, using the thread management function in the QThread class to manage node communication, and establishing a connection between ROS2 and Qt; managing communication between nodes through the Qt program, transmitting communication data to the message processing function in the form of program messages, and realizing the association of node communication data with the GUI window; Each module in the human-computer interaction unit is encapsulated in a communication node Node, and signals are transmitted through the DDS communication mechanism. The interactive information is bound to various controls in Qt Designer to realize information processing and display; The human-computer interaction unit is based on Qt Designer, and the interaction process is managed by QThread. The communication message of the node Node is converted into a program message by the Qt system kernel and sent to the slot Slot in the form of a signal Signal. The custom slot function Slot_handle() is used to process the signal and interact with the UI interface window or control.

2. The human-computer interaction system based on ROS2 and Qt according to claim 1, characterized in that: The human-machine interaction unit is used to realize the interaction between the robot's sensor information and the control information of the controller, control the remote robot through the interface buttons and / or receive the information sent back by the robot in real time and display it to the controller.

3. The human-computer interaction system based on ROS2 and Qt according to claim 2, characterized in that: The sensor information and control information include control instruction information, robot motion information, video image information, robot status monitoring information, and robot posture information.

4. The human-computer interaction system based on ROS2 and Qt according to claim 1, characterized in that: The human-computer interaction unit is based on the signal and slot mechanism, which regards the ROS2 node as a specific event. When the node processes the program or communicates, it is regarded as an event trigger. The communication data or program processing results are regarded as signals. The transmitted information is mapped to the main window through the custom signal slot to realize information interaction.

5. The human-computer interaction system based on ROS2 and Qt according to claim 1, 2 or 3, characterized in that: The robot status monitoring unit includes two parts: communication link monitoring and equipment voltage and temperature monitoring. For the communication link monitoring part, half of the information transmission and reception time is used as the observation value; for the voltage and temperature monitoring part, the measuring element transmits the data back to the control end through the serial port, and uses the register to store the data read by the serial port, and then reads the data from the register to determine whether the data is normal. For abnormal data, the frame header is retained and the remaining part of the frame is cropped for processing.

6. The human-computer interaction system based on ROS2 and Qt according to claim 1, 2 or 3, characterized in that: In the robot position display unit, the coordinates of the selected scene map are first measured to produce a global map; then the actual coordinates of the robot are converted into the global map to display the relative position of the robot; and different colors are used to distinguish different terrains or objects.

7. The human-computer interaction system based on ROS2 and Qt according to claim 1, 2 or 3, characterized in that: The robot motion state display unit is used to display the robot's motion state and detection information of the surrounding environment, including motion speed and angular velocity, speed gear and angular velocity gear, metal detection signal and digital recognition signal.

8. The human-computer interaction system based on ROS2 and Qt according to claim 1, 2 or 3, characterized in that: The interactive interface of the real-time image display unit uses three-way video information: forward, rear, and mobile. The forward and rear cameras are fixed cameras, and their optimal field of view angles are obtained by adjusting the camera installation position. They are responsible for observing the front and rear environment of the robot. The mobile camera is installed above the mechanical claw and is responsible for dynamically observing the surrounding environment.

Images