Robot visual interaction control method and device, equipment and storage medium

Abstract

The invention belongs to the technical field of intelligent control, and discloses a robot visual interaction control method and device, electronic equipment and a storage medium, when the electronic equipment is started, an ROS2 communication environment is initialized, and a Qt graphical user interface is loaded; whenever any user interface subpage on the Qt graphical user interface is detected to be switched from the hidden state to the display state, adding one to the count value; when any user interface sub-page is detected to be switched from the display state to the hidden state, subtracting one from the count value; when the count value is changed from 0 to 1, starting ROS2 topic subscription; when the count value is changed from 1 to 0, the ROS2 topic subscription is canceled; by introducing a dynamic binding mechanism of the interface visibility and the ROS2 subscription state, the data subscription is automatically cancelled when the interface component is hidden, so that the continuous receiving and processing of background invalid data streams can be avoided, the CPU resource occupancy rate and communication resources are effectively reduced, the communication load and processing overhead are further reduced, and the operation fluency and response speed are improved.

Description

Technical Field

[0001] This invention relates to the field of intelligent control technology, specifically to a robot visual interactive control method and device, electronic device, and storage medium. Background Technology

[0002] With the continuous expansion of application scenarios such as intelligent robots and service robots, human-computer interaction systems have become a core component of robot systems. Traditional human-computer interaction methods often use custom proprietary communication protocols or are based on ROS1, employing a master-slave node communication model.

[0003] This architecture can meet basic interaction needs in static, centralized control environments, but in complex scenarios such as multi-robot coordination, cross-platform deployment, high real-time requirements, and frequent changes in dynamic environments, embedded device resources are limited. Even when the traditional host computer multi-task / process interface is in the background or minimized state, it is still refreshing the Topic / Service at full speed to receive unnecessary external data, which continuously occupies CPU resources and generates additional communication resources, easily causing network congestion. Summary of the Invention

[0004] To address the aforementioned shortcomings, the present invention aims to provide a robot visualization and interactive control method and device, electronic device, and storage medium, which can reduce CPU resource utilization and communication resources, thereby reducing the overall communication load and processing overhead, and improving operational smoothness and response speed.

[0005] The first aspect of this invention discloses a robot visualization and interactive control method, the method being applied to an electronic device, wherein the electronic device and the robot device communicate via a data distribution service protocol, the method comprising: When the electronic device is detected to be starting up, initialize the ROS2 communication environment of the electronic device and load the Qt graphical user interface of the electronic device; Whenever any user interface subpage on the Qt graphical user interface is detected to switch from a hidden state to a visible state, the counter is incremented by one; and whenever any user interface subpage on the Qt graphical user interface is detected to switch from a visible state to a hidden state, the counter is decremented by one. When the counter value changes from 0 to 1, the ROS2 topic subscription is initiated; When the counter value changes from 1 to 0, the ROS2 topic subscription is canceled.

[0006] In some embodiments, after initializing the ROS2 communication environment of the electronic device, the method further includes: The configuration file is determined based on the configuration parameters input by the user; Read the configuration file for syntax validation and data extraction, and generate a list of node description objects; Create a ROS2 communication node based on the list of node description objects.

[0007] In some embodiments, after creating a ROS2 communication node based on the list of node description objects, the method further includes: Register the node handle of the created ROS2 communication node to the communication management center; In response to the target unique identifier entered on the Qt graphical user interface, the target node handle is obtained from the communication management center; The target node handle is sent to the corresponding Relay object so that the Relay object can manage the data flow of the target ROS2 communication node corresponding to the target node handle.

[0008] In some embodiments, creating a ROS2 communication node based on the list of node description objects includes: Based on the mode field in the node description object list, identify the target communication type that needs to be created; By using a pre-configured registry, the msg_type string in the node description object list is mapped to obtain the target ROS2 message type; Construct the target QoS policy based on the QoS parameters in the node description object list; Invoke the constructor of the subclass that inherits from the unified abstract base class AbstractNodeHandle, which corresponds to the target communication type, and instantiate the corresponding target class object; The create() method of the target class object is called. Inside the create() method, the application programming interface of the ROS2 underlying library is used to create an underlying ROS2 communication node bound to the target ROS2 message type, and the target QoS policy is applied to the underlying ROS2 communication node.

[0009] In some embodiments, the robotic device includes a robotic arm; after loading the Qt graphical user interface of the electronic device, the method further includes: The URDF file of the robotic arm is loaded and parsed in real time, and the joint state topic data of the robotic arm is obtained. Update the joint poses of the robotic arm based on the URDF file and the joint state topic data; Based on the postures of each joint of the robotic arm, a corresponding 3D motion view of the robotic arm is drawn and displayed on the Qt graphical user interface.

[0010] A second aspect of this invention discloses a robot visualization and interactive control device, wherein the device communicates with a robot device via a data distribution service protocol, and the device comprises: An initialization unit is used to initialize the ROS2 communication environment of the electronic device when the electronic device is detected to be powered on. A loading unit is configured to load the Qt graphical user interface of the electronic device when the electronic device is detected to be powered on. The counting unit is configured to increment the counter value by one whenever any user interface subpage on the Qt graphical user interface is detected to switch from a hidden state to a displayed state; and decrement the counter value by one whenever any user interface subpage on the Qt graphical user interface is detected to switch from a displayed state to a hidden state. The subscription unit is used to initiate ROS2 topic subscription when the counter value changes from 0 to 1. The unsubscribe unit is used to unsubscribe from the ROS2 topic when the counter value changes from 1 to 0.

[0011] In some embodiments, the apparatus further includes: The acquisition unit is used to determine the configuration file based on the configuration parameters input by the user after the initialization unit initializes the ROS2 communication environment of the electronic device; The reading unit is used to read the configuration file for syntax verification and data extraction, and to generate a list of node description objects. A creation unit is used to create ROS2 communication nodes based on the list of node description objects.

[0012] In some embodiments, the apparatus further includes: The registration unit is used to register the node handle of the created ROS2 communication node to the communication management center after the creation unit creates the ROS2 communication node according to the node description object list. The retrieval unit is used to obtain the target node handle from the communication management center in response to the target unique identifier code input on the Qt graphical user interface; The data management unit is used to send the target node handle to the corresponding Relay object, so that the Relay object can perform data flow management on the target ROS2 communication node corresponding to the target node handle.

[0013] A third aspect of the present invention discloses an electronic device, including a memory storing executable program code and a processor coupled to the memory; the processor calls the executable program code stored in the memory to execute the robot visualization and interactive control method disclosed in the first aspect.

[0014] A fourth aspect of the present invention discloses a computer-readable storage medium storing a computer program, wherein the computer program causes a computer to execute the robot visualization and interactive control method disclosed in the first aspect.

[0015] Compared with the prior art, the beneficial effects of the present invention are as follows: when the electronic device is detected to be started, the ROS2 communication environment of the electronic device is initialized and the Qt graphical user interface of the electronic device is loaded; whenever any user interface subpage on the Qt graphical user interface is detected to switch from a hidden state to a visible state, the count value of the control counter is incremented by one; and whenever any user interface subpage on the Qt graphical user interface is detected to switch from a visible state to a hidden state, the count value of the control counter is decremented by one; when the count value of the counter changes from 0 to 1, ROS2 topic subscription is started; when the count value of the counter changes from 1 to 0, ROS2 topic subscription is canceled; by introducing a dynamic binding mechanism between interface visibility and ROS2 subscription status, data subscription is automatically canceled when the interface component is hidden, which can avoid the continuous reception and processing of invalid data streams in the background, effectively reducing CPU resource utilization, communication resource, memory consumption and network bandwidth pressure, thereby reducing the overall communication load and processing overhead, and improving the smoothness and response speed of operation. Attached Figure Description

[0016] Figure 1 This is a schematic diagram of communication between a human-computer interaction platform and a robot device disclosed in an embodiment of the present invention; Figure 2 This is a flowchart of a robot visualization and interactive control method disclosed in an embodiment of the present invention; Figure 3 This is a schematic diagram of the structure of a robot visual interactive control device disclosed in an embodiment of the present invention; Figure 4 This is a schematic diagram of the structure of an electronic device disclosed in an embodiment of the present invention; Figure 5 This is a schematic diagram of the structure of a computer device disclosed in an embodiment of the present invention.

[0017] Explanation of reference numerals in the attached figures: 301. Initialization unit; 302. Loading unit; 303. Counting unit; 304. Start subscription unit; 305. Unsubscribe unit; 401. Memory; 402. Processor. Detailed Implementation

[0018] To enable those skilled in the art to better understand the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings.

[0019] In some of the processes described in the specification, claims, and accompanying drawings of this invention, multiple operations appearing in a specific order are included. However, it should be clearly understood that these operations may not be executed in the order they appear herein, or may be executed in parallel. The operation numbers, such as 110, 120, etc., are merely used to distinguish different operations and do not themselves represent any execution order. Furthermore, these processes may include more or fewer operations, and these operations may be executed sequentially or in parallel. It should be noted that the descriptions such as "first," "second," etc., in this document are used to distinguish different messages, devices, modules, etc., and do not represent a sequential order, nor do they limit "first" and "second" to different types.

[0020] It will be understood by those skilled in the art that, unless specifically stated otherwise, the singular forms “a,” “an,” “the,” and “the” used herein may also include the plural forms. It should be further understood that the term “comprising” as used in this specification means the presence of the stated features, integers, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. It should be understood that when we say an element is “connected” or “coupled” to another element, it can be directly connected or coupled to the other element, or there may be intermediate elements. Furthermore, “connected” or “coupled” as used herein can include wireless connections or wireless coupling. The term “and / or” as used herein includes all or any units and all combinations of one or more associated listed items.

[0021] It will be understood by those skilled in the art that, unless otherwise defined, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by those skilled in the art to which this invention pertains. It should also be understood that terms such as those defined in general dictionaries should be understood to have the same meaning as in the context of the prior art and should not be interpreted in an idealized or overly formal sense unless specifically defined as herein.

[0022] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Throughout the description, the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0023] Please see Figure 1This invention discloses a human-computer interaction platform. The human-computer interaction platform and the robot device communicate with each other via the Data Distribution Service (DDS) protocol. The robot device includes a robot chassis system, a robot system, and a robot brain computing layer, which is responsible for robot logic operations, motion algorithms, and motion execution.

[0024] In terms of hardware, this human-machine interaction platform uses the Allwinner LT527 chip as its core processing device. This chip has multi-core heterogeneous computing capabilities, integrating a high-performance CPU, GPU, and rich peripheral interfaces, effectively supporting multiple task loads such as graphics rendering, network communication, and real-time data processing. The platform is also equipped with a high-resolution capacitive touchscreen that supports multi-touch operation, providing users with intuitive gesture interaction capabilities. Furthermore, the platform includes communication modules such as Wi-Fi and Bluetooth. The Wi-Fi module supports the IEEE 802.11ac protocol for transmitting high-bandwidth sensor data, such as sensor images and map information; the Bluetooth module supports the Bluetooth 5.0 standard for connecting external input devices, such as gamepads and keyboards, ensuring stable access to the Robot Operating System 2 (ROS2) network, enabling data interaction with robotic devices or other distributed nodes, and improving the system's scalability and flexibility.

[0025] In terms of software architecture, this human-computer interaction platform embeds a robot visual interaction control system, which includes an embedded main control module, a ROS2 communication module, a Relay object, a Binder object, a Qt graphical user interface, and a subscription control module. The embedded master control module, as the central hub of the entire system, coordinates the work of various components, including starting and managing the ROS2 communication environment and hardware driver-level software functions.

[0026] The ROS2 communication module provides a configuration interface to manage and abstract all ROS2 communication, ensuring that messages of different communication types (publish / subscribe, service / client, etc.) can be processed efficiently.

[0027] The Relay object is used to instantiate a Relay object for each ROS2 topic, cache the latest data packets for each ROS2 topic, and track which interfaces are using the data through a counting mechanism.

[0028] The Binder object is used to instantiate a Binder object for each Qt user interface subpage. It is responsible for managing the visibility state of each Qt user interface subpage and establishing or disconnecting data connections with the relevant Relay object as necessary.

[0029] The Qt graphical user interface provides an intuitive user interface that can display topical data from ROS2 in real time, such as a 3D robotic arm motion view and a 2D navigation map. The graphical user interface uses a multi-tab or window switching mechanism, with different tabs corresponding to the 3D robotic arm motion view, the 2D navigation map, or the parameter configuration panel, etc. Moreover, only the currently active page maintains the ROS2 data subscription status.

[0030] The subscription control module monitors the visibility status of each visual Qt component in the Qt graphical user interface. It automatically cancels the corresponding ROS2 topic subscription when a Qt component is hidden and re-establishes the subscription when the component becomes visible again. Specifically, it triggers the enabling or disabling of subscriptions through Qt's `visibilityChanged` signal or the `isHidden()` state detection mechanism.

[0031] Alternatively, the embedded main control module is developed based on the Linux+PreemptRt operating system, using the Qt framework and RViz2 functional components for deep customization. The Qt framework, as a cross-platform C++ graphical user interface development framework, is used to build the main control interface, including interactive elements such as menu systems, control buttons, status indicators, and parameter configuration panels. On this basis, the embedded main control module integrates an RViz2 rendering module, which is used to embed RViz2 rendering components into the Qt graphical user interface to achieve real-time visualization of the 3D robotic arm's motion view.

[0032] Through the plugin mechanism, the Unified Robot Description Format (URDF) file of the robotic arm can be loaded, and the posture of each joint of the robotic arm can be dynamically updated in combination with the joint state topic of the robotic arm, so as to achieve high-fidelity motion simulation and state feedback.

[0033] Meanwhile, the embedded main control module supports the synchronous display of 2D navigation maps. By subscribing to navigation-related topics in ROS2 (such as occupancy_grid, map, tf, etc.), the embedded main control module can render grid maps, robot poses, path planning trajectories, and obstacle information in independent views or overlays, helping operators to grasp the global environmental awareness status of the mobile platform.

[0034] When the embedded main control module controls the Qt graphical user interface to display a two-dimensional navigation map, it supports human-computer interaction operations such as zooming, panning, and coordinate positioning, and can achieve fast browsing through touch screen gestures.

[0035] Please see Figure 2 This invention discloses a robot visualization and interactive control method. The executing entity of this method can be the aforementioned human-computer interaction platform, which can be configured as an electronic device such as a computer, laptop, or tablet computer, or as a robot visualization and interactive control device embedded in an electronic device. This invention does not limit the specific implementation of this method. This invention will be described in detail using an electronic device as an example. Figure 2 As shown, the method includes the following steps 110-140: 110. When the electronic device is detected to be starting up, initialize the ROS2 communication environment of the electronic device and load the Qt graphical user interface of the electronic device.

[0036] As a preferred implementation, initializing the ROS2 communication environment of the electronic device includes: Configure the initialization of the ROS2 communication mode corresponding to the Relay object embedded in the electronic device, and manage and hold it for the Binder object embedded in the electronic device; among which, the ROS2 communication modes mainly include Topic, Service, Action, etc.

[0037] Preferably, to simplify development, this invention encapsulates various communication types (publish / subscribe, service / client, action, etc.) under the three main communication modes of ROS2—Topic, Service, and Action—into a unified communication node abstract interface, providing consistent operational semantics to the outside world. First, a unified abstract base class, AbstractNodeHandle, is defined, and three specific implementations are derived based on the three main communication modes of ROS2—Topic, Service, and Action—as shown in Table 1 below: Table 1. Unified Interface Design Correspondence Table

[0038] Table 1 defines the unified interface method, which specifies the unified protocol that all communication nodes must implement. `create()`, `send(data)`, and `onReceive(callback)` correspond to the creation, sending, and receiving methods, respectively. `create()` is called during the initialization phase and is responsible for creating the underlying ROS2 communication objects, such as Publisher and Server, based on the configuration. `send(data)` provides a unified interface for nodes that actively initiate communication, such as Publisher, Client, or ActionClient, to send data. Upper-layer business code only needs to call this interface without needing to know whether the underlying layer uses `publish`, `send_request`, or `send_goal`. `onReceive(callback)` provides a unified callback function registration interface for nodes that passively receive information, such as Subscriber, Server, or ActionServer. When a message, request, or feedback arrives at the underlying layer, the registered callback function is automatically triggered.

[0039] Table 1 shows how the abstract interface maps to the specific application programming interface (API) of ROS2: In the create() method, for a Topic, it internally creates a Publisher or Subscriber; for a Service, it internally creates a Server or Client; and for an Action, it internally creates an ActionServer or ActionClient. In the send(data) method, for Topic (publish), it internally calls publish(); for Service (client), it internally calls async_send_request(); for Action (client), it internally calls async_send_goal(). In the onReceive(callback) method, for Topic (subscription), register the function to receive message callbacks; for Service, register the function to handle requests; for Action, register the callback functions to handle the target, feedback, and result.

[0040] Based on this, the base class AbstractNodeHandle and its derived classes perfectly encapsulate all the differences between the three ROS2 communication modes, providing a simple and consistent operating interface for the upper layer. After initializing the ROS2 communication environment of the electronic device, developers or users can define the configuration parameters of the ROS2 communication nodes through configuration files, automatically completing the creation of ROS2 communication nodes. This reduces the threshold for ROS2 application development and improves the maintainability and scalability of the system.

[0041] In practical applications, after initializing the ROS2 communication environment of an electronic device, the creation of a ROS2 communication node may include the following steps 101-102 (not shown): 101. Determine the configuration file based on the configuration parameters input by the user, read the configuration file for syntax validation and data extraction, and generate a list of node description objects.

[0042] Users declare the ROS2 communication nodes they wish to create through a configuration file in YAML format, which is loaded upon system startup. The communication configuration parser (CommConfigParser) reads the configuration file and generates a list of node description objects. This list uses a specific data structure that the program can process internally. Each node description object corresponds to a communication requirement declared by the user in the configuration file. The communication requirement includes the ROS2 communication node's unique identifier (ID), communication mode (Topic, Service, or Action), communication type (publish / subscribe / service / client, etc.), topic name, ROS2 message type, and Quality of Service (QoS) policy. For example, the QoS policy might specifically require reliable transmission and non-persistent delivery.

[0043] 102. Create ROS2 communication nodes based on the list of node description objects.

[0044] Specifically, step 102 may include the following steps 1021-1025 (not shown): 1021. Identify the target communication type to be created based on the mode field in the node description object list.

[0045] 1022. By using a pre-configured registry, the msg_type string in the node description object list is mapped to obtain the target ROS2 message type.

[0046] 1023. Construct the target QoS policy based on the QoS parameters in the node description object list.

[0047] 1024. Call the constructor of the subclass of the target communication type that inherits from the unified abstract base class AbstractNodeHandle to instantiate the corresponding target class object.

[0048] 1025. Call the create() method of the target class object. Inside the create() method, use the application programming interface of the ROS2 underlying library to create an underlying ROS2 communication node that is bound to the target ROS2 message type, and at the same time apply the target QoS policy to this underlying ROS2 communication node.

[0049] The unified abstract base class `AbstractNodeHandle` provides a consistent interface for all communication nodes. By calling the constructor of a subclass inheriting from `AbstractNodeHandle`, a corresponding target class object is instantiated. Then, the `create()` method of the target class object is called. Inside the `create()` method, the underlying ROS2 communication node is created using the ROS2 library's API. Whether it's a publisher, subscriber, or service client, they are all uniformly operable objects from the perspective of the upper-level administrator, which greatly reduces the coupling between modules.

[0050] For example, based on the `type: "service_client"` field, the target communication type to be created is identified as a service client. Then, the corresponding `ServiceClientNode` class object is instantiated. This class inherits from the unified abstract base class `AbstractNodeHandle`. Next, the `create()` method of the `ServiceClientNode` class object is called. Inside this `create()` method, the underlying ROS2 library API is used to create a real service client bound to the specified ROS2 message type; simultaneously, the QoS policy declared in the configuration is applied to this client to ensure that the reliability requirements of communication are met.

[0051] Steps 101-102 simplify the node configuration process, lower the development and debugging threshold, and improve the modularity and scalability of the system by abstracting, encapsulating and centrally managing various communication types of ROS2 (publish / subscribe, service / client, action, etc.), making it easier to quickly adapt to different types of robot platforms.

[0052] After completing all the above initializations, developers do not need to worry about whether the node is a service, a topic, or another communication mechanism in their business code. They only need to use the obtained node handle (node_handle) to call the unified interface to send requests. The underlying ServiceClientNode implementation will be responsible for translating this call into a correct ROS2 service call, which achieves isolation and decoupling of communication details from business logic.

[0053] Further, optionally, after performing steps 101-102, the following steps 103-105 may also be performed: 103. Register the node handle of the created ROS2 communication node to the communication management center.

[0054] Among them, the Communication Management Center (CommManager) is a core manager that maintains an index of all dynamically created nodes, usually indexed by the node's ID, for easy retrieval on demand.

[0055] 104. In response to the target unique identifier entered on the Qt graphical user interface, obtain the target node handle from the communication management center.

[0056] 105. Send the target node handle to the corresponding Relay object so that the Relay object can manage the data flow of the target ROS2 communication node corresponding to the target node handle.

[0057] Since each node handle corresponds to an ID, users can input the ID through the Qt graphical user interface. After the electronic device detects the ID input by the user in the Qt graphical user interface, it retrieves the corresponding node handle from the communication management center and then forwards it to the corresponding Relay object for data flow management.

[0058] In this embodiment of the invention, after determining the target node handle, if data needs to be sent—whether it's issuing a speed command, sending a service request, or initiating a navigation goal—the Relay object uniformly calls `node_handle->send(data)`. Whether it's issuing a topic or sending a request is handled internally by the subclass. If data needs to be received, the Relay object uniformly calls `node_handle->onReceive(callback)` to register its own processing function. When data arrives, whether it's a topic message, a service request, or action feedback, it will be notified through this unified callback interface.

[0059] In some embodiments, if the created ROS2 communication node is a subscriber, its callback function is pre-set during initialization. When a ROS2 topic message arrives, this callback function is triggered, and the bound Relay::onDataReceived() method is called internally, thereby automatically and seamlessly passing data from the ROS2 communication layer to the system's business logic or UI layer for processing and display.

[0060] Alternatively, after loading the Qt graphical user interface of the electronic device, a three-dimensional view of the robotic arm's motion can be displayed in real time, specifically by executing the following steps S1~S3 (not shown): S1. Load and parse the URDF file of the robot arm in real time, and obtain the joint state topic data of the robot arm.

[0061] S2. Update the poses of each joint of the robot arm based on the URDF file and joint state topic data.

[0062] S3. Based on the posture of each joint of the robot arm, draw the corresponding 3D motion view of the robot arm and display it on the Qt graphical user interface.

[0063] Alternatively, after loading the Qt graphical user interface of the electronic device, a two-dimensional navigation map can also be displayed on the touchscreen in real time. The specific steps are as follows: Subscribe to occupancy_grid, tf, and map topic data for rendering to obtain navigation information in real time; draw the corresponding two-dimensional navigation map based on the navigation information and display it on the Qt graphical user interface.

[0064] Furthermore, when a user's input gesture command is detected, zooming, panning, or coordinate positioning operations can be performed on the two-dimensional navigation map based on the gesture command.

[0065] By enabling the synchronous visualization of a 3D robotic arm model and a 2D navigation map on an embedded terminal, users can monitor the robot's spatial posture and environmental awareness in real time. This significantly enhances the robot's situational awareness during operation, reduces the risk of misoperation, and improves control precision and task execution efficiency.

[0066] 120. Whenever any user interface subpage on the Qt graphical user interface is detected to switch from a hidden state to a visible state, the counter is incremented by one; and whenever any user interface subpage on the Qt graphical user interface is detected to switch from a visible state to a hidden state, the counter is decremented by one.

[0067] In this embodiment of the invention, the Qt interface thread corresponding to the Qt graphical user interface includes multiple embedded RViz2 rendering components, such as QWidget, QMainWindow, or custom view class components. Each embedded RViz2 rendering component corresponds to a user interface (UI) subpage.

[0068] UI subpages include, but are not limited to, interface windows such as 3D robotic arm motion views or 2D navigation maps. Each UI subpage is associated with a Binder object, which listens for changes in the interface state of each UI subpage via Qt's built-in signals. The interface state includes a hidden state (not visible to the user, such as when the subpage is minimized or closed) and a visible state (visible to the user). The visibility of the UI subpage holding the Binder object is configured through the Qt graphical user interface; if displayed, it is set to True, and the RefCnt counter is incremented.

[0069] 130. When the counter value changes from 0 to 1, start the ROS2 topic subscription.

[0070] Starting a ROS2 topic subscription involves calling the underlying ROS2 client library (such as rclcpp) to create an rclcpp::Subscription instance and starting the subscription.

[0071] Simultaneously, when the counter value changes from 0 to 1, it indicates that this is the first time the UI subpage is displayed, and ROS2 topic subscription needs to be started. Furthermore, after starting ROS2 topic subscription, a connection is established with the ROS thread, controlling the ROS thread to actively listen for ROS2 topic communication data in real time. When the ROS thread receives the latest message packet from the ROS2 topic, it sends the latest message packet to the Qt UI thread and simultaneously emits an update signal to notify the Qt UI thread to update the UI subpage.

[0072] In other words, if at least one UI subpage is currently displayed, once a communication data message packet for the ROS2 topic arrives, it will be forwarded to the Binder object and the corresponding registration callback function will be responded to. The UI subpage controls will then display the corresponding elements in the registration callback function.

[0073] 140. When the counter value changes from 1 to 0, cancel the ROS2 topic subscription.

[0074] When the counter changes from 1 to 0, indicating that the last UI subpage has been hidden or closed (i.e., the entire Qt graphical user interface has exited the display), the listening for ROS2 topic communication data should be stopped. Specifically, canceling the ROS2 topic subscription includes calling `subscription.reset()` to destroy the subscription object and stopping receiving ROS2 topic messages. In other words, after canceling the ROS2 topic subscription, the ROS thread will no longer receive the latest message packets from the ROS2 topic.

[0075] In this embodiment of the invention, updates are performed on demand. Time-consuming cross-thread signal emission and UI control data refresh operations are only performed when a visible UI subpage requires data. When no UI subpage is visible, receiving relevant topic data ceases. When a UI subpage becomes visible again, the subscription is automatically resumed and the latest state is resynchronized, avoiding unnecessary computation and communication overhead when the UI subpage is invisible.

[0076] In summary, by implementing the embodiments of the present invention, a dynamic binding mechanism between interface visibility and ROS2 subscription state is introduced, which automatically cancels data subscription when interface components are hidden. This avoids the continuous reception and processing of invalid background data streams, effectively reduces CPU utilization, memory consumption, and network bandwidth pressure, extends the stable operating time of embedded devices, significantly reduces the overall communication load and processing overhead of the system, and improves the smoothness and response speed of operation.

[0077] Furthermore, through the highly integrated design of the hardware platform and the deep customization and optimization of the software system, this invention constructs a fully functional, responsive, and resource-efficient local control terminal for robots. It is particularly suitable for scenarios requiring localized human-computer interaction, such as mobile robots and robotic arm workstations, and has good practicality and promotional value.

[0078] like Figure 3 As shown in the figure, this embodiment of the invention also discloses a robot visual interactive control device. This device communicates with the robot device via a data distribution service protocol. The device includes an initialization unit 301, a loading unit 302, a counting unit 303, a subscription initiation unit 304, and a subscription cancellation unit 305. Initialization unit 301 is used to initialize the ROS2 communication environment of the electronic device when the electronic device is detected to be starting up; The loading unit 302 is used to load the Qt graphical user interface of the electronic device when the electronic device is detected to be starting up; The counting unit 303 is configured to increment the counter value by one whenever any user interface subpage on the Qt graphical user interface is detected to switch from a hidden state to a displayed state; and decrement the counter value by one whenever any user interface subpage on the Qt graphical user interface is detected to switch from a displayed state to a hidden state. Start subscription unit 304 is used to start ROS2 topic subscription when the counter value changes from 0 to 1; The unsubscribe unit 305 is used to unsubscribe from the ROS2 topic when the counter value changes from 1 to 0.

[0079] As an optional implementation, the device further includes the following units (not shown): The acquisition unit is used to determine the configuration file based on the configuration parameters input by the user after the initialization unit 301 initializes the ROS2 communication environment of the electronic device. The reading unit is used to read the configuration file for syntax validation and data extraction, and to generate a list of node description objects. Create a cell to create a ROS2 communication node based on a list of node description objects.

[0080] As an optional implementation, the device further includes the following units (not shown): The registration unit is used to register the node handle of the created ROS2 communication node to the communication management center after the creation unit creates the ROS2 communication node based on the list of node description objects. The retrieval unit is used to obtain the target node handle from the communication management center in response to the target unique identifier entered on the Qt graphical user interface; The data management unit is used to send the target node handle to the corresponding Relay object, so that the Relay object can manage the data flow of the target ROS2 communication node corresponding to the target node handle.

[0081] As an optional implementation, the creation unit may include the following sub-units (not shown): The identification sub-unit is used to identify the target communication type to be created based on the mode field in the node description object list; The mapping subunit is used to map the msg_type string in the node description object list to obtain the target ROS2 message type through a preset registry; The construction sub-unit is used to construct the target QoS policy based on the QoS parameters in the node description object list; The calling sub-unit is used to call the constructor of the subclass corresponding to the target communication type, which inherits from the unified abstract base class AbstractNodeHandle, and instantiate the corresponding target class object; A sub-unit is created to call the create() method of the target class object. Inside the create() method, the application programming interface of the underlying ROS2 library is used to create an underlying ROS2 communication node bound to the target ROS2 message type, and the target QoS policy is applied to the underlying ROS2 communication node.

[0082] As an optional implementation, the robot device includes a robotic arm; the control device also includes the following units not shown: The parsing unit is used to load and parse the URDF file of the robot arm in real time after the loading unit 302 loads the Qt graphical user interface of the electronic device, and obtain the joint state topic data of the robot arm. The pose update unit is used to update the pose of each joint of the robot arm based on the URDF file and joint state topic data. The drawing unit is used to draw the corresponding 3D motion view of the robot arm based on the posture of each joint and display it on the Qt graphical user interface.

[0083] Further optionally, the drawing unit is also used to subscribe to occupancy_grid, tf, and map topic data for rendering after the loading unit 302 loads the Qt graphical user interface of the electronic device, so as to obtain navigation information in real time; and to draw the corresponding two-dimensional navigation map according to the navigation information and display it on the Qt graphical user interface.

[0084] like Figure 4 As shown, this embodiment of the invention also discloses an electronic device, including a memory 401 storing executable program code and a processor 402 coupled to the memory 401; The processor 402 calls the executable program code stored in the memory 401 to execute the robot visualization and interactive control method described in the above embodiments.

[0085] like Figure 5As shown in the illustration, this invention also discloses a computer device. This computer device includes a processor, memory, network interface, display screen, and input device connected via a system bus. The processor in this computer design provides computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The database of the computer device is used to store data related to the robot visualization and interactive control method. The network interface of the computer device is used for communication with external terminals via a network connection. When the computer program is executed by the processor, it implements the robot visualization and interactive control method described in the above embodiments.

[0086] This invention also discloses a computer-readable storage medium storing a computer program that causes a computer to execute the robot visualization and interactive control method described in the above embodiments. The storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, or optical data storage device, etc.

[0087] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. This computer program can be stored in a storage medium, and when executed, it can include the processes of the embodiments of the methods described above. The aforementioned storage medium can be a non-volatile storage medium such as a magnetic disk, optical disk, or read-only memory (ROM), or random access memory (RAM).

[0088] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0089] The embodiments described above are merely examples of several implementations of the present invention, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these modifications and improvements all fall within the scope of protection of the present invention.

Claims

1. A robot visualization and interactive control method, characterized in that, The method is applied to an electronic device, wherein the electronic device communicates with a robotic device via a data distribution service protocol, and the method includes: When the electronic device is detected to be starting up, initialize the ROS2 communication environment of the electronic device and load the Qt graphical user interface of the electronic device; Whenever any user interface subpage on the Qt graphical user interface is detected to switch from a hidden state to a visible state, the counter is incremented by one; and whenever any user interface subpage on the Qt graphical user interface is detected to switch from a visible state to a hidden state, the counter is decremented by one. When the counter value changes from 0 to 1, the ROS2 topic subscription is initiated; When the counter value changes from 1 to 0, the ROS2 topic subscription is canceled.

2. The robot visualization and interactive control method according to claim 1, characterized in that, After initializing the ROS2 communication environment of the electronic device, the method further includes: The configuration file is determined based on the configuration parameters input by the user; The configuration file is read for syntax validation and data extraction to generate a list of node description objects. Create a ROS2 communication node based on the list of node description objects.

3. The robot visualization and interactive control method according to claim 2, characterized in that, After creating the ROS2 communication node based on the list of node description objects, the method further includes: Register the node handle of the created ROS2 communication node to the communication management center; In response to the target unique identifier entered on the Qt graphical user interface, the target node handle is obtained from the communication management center; The target node handle is sent to the corresponding Relay object so that the Relay object can manage the data flow of the target ROS2 communication node corresponding to the target node handle.

4. The robot visualization and interactive control method according to claim 2, characterized in that, Based on the list of node description objects, create ROS2 communication nodes, including: Based on the mode field in the node description object list, identify the target communication type that needs to be created; By using a pre-configured registry, the msg_type string in the node description object list is mapped to obtain the target ROS2 message type; Construct the target QoS policy based on the QoS parameters in the node description object list; Invoke the constructor of the subclass that inherits from the unified abstract base class AbstractNodeHandle, which corresponds to the target communication type, and instantiate the corresponding target class object; The create() method of the target class object is called. Inside the create() method, the application programming interface of the ROS2 underlying library is used to create an underlying ROS2 communication node bound to the target ROS2 message type, and the target QoS policy is applied to the underlying ROS2 communication node.

5. The robot visualization and interactive control method according to any one of claims 1 to 4, characterized in that, The robotic device includes a robotic arm; after loading the Qt graphical user interface of the electronic device, the method further includes: The URDF file of the robotic arm is loaded and parsed in real time, and the joint state topic data of the robotic arm is obtained. Update the joint poses of the robotic arm based on the URDF file and the joint state topic data; Based on the postures of each joint of the robotic arm, a corresponding 3D motion view of the robotic arm is drawn and displayed on the Qt graphical user interface.

6. A robot visual interactive control device, characterized in that, The device communicates with the robotic equipment via a data distribution service protocol. The device includes: An initialization unit is used to initialize the ROS2 communication environment of the electronic device when the electronic device is detected to be powered on. A loading unit is used to load the Qt graphical user interface of the electronic device when the electronic device is detected to be powered on. The counting unit is configured to increment the counter value by one whenever any user interface subpage on the Qt graphical user interface is detected to switch from a hidden state to a displayed state; and decrement the counter value by one whenever any user interface subpage on the Qt graphical user interface is detected to switch from a displayed state to a hidden state. The subscription unit is used to initiate ROS2 topic subscription when the counter value changes from 0 to 1. The unsubscribe unit is used to unsubscribe from the ROS2 topic when the counter value changes from 1 to 0.

7. The robot visualization and interactive control device according to claim 6, characterized in that, The device further includes: The acquisition unit is used to determine the configuration file based on the configuration parameters input by the user after the initialization unit initializes the ROS2 communication environment of the electronic device; The reading unit is used to read the configuration file for syntax verification and data extraction, and to generate a list of node description objects. A creation unit is used to create ROS2 communication nodes based on the list of node description objects.

8. The robot visualization and interactive control device according to claim 7, characterized in that, The device further includes: The registration unit is used to register the node handle of the created ROS2 communication node to the communication management center after the creation unit creates the ROS2 communication node according to the node description object list. The retrieval unit is used to obtain the target node handle from the communication management center in response to the target unique identifier code input on the Qt graphical user interface; The data management unit is used to send the target node handle to the corresponding Relay object, so that the Relay object can perform data flow management on the target ROS2 communication node corresponding to the target node handle.

9. An electronic device, characterized in that, It includes a memory storing executable program code and a processor coupled to the memory; the processor calls the executable program code stored in the memory to execute the robot visualization and interactive control method according to any one of claims 1 to 5.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, wherein the computer program causes a computer to perform the robot visualization and interactive control method according to any one of claims 1 to 5.

Images