Deep reinforcement learning and heterogeneous real-time bus robot control system and method

Abstract

The application discloses a kind of deep reinforcement learning and robot control system and method of heterogeneous real-time bus, by constructing the heterogeneous real-time bus architecture including shared memory exchange layer, utilize mmap mapping technology realizes the zero-copy data interaction between EtherCAT real-time control layer and ROS2 application communication layer, significantly reduce the communication delay and CPU resource occupation caused by multiple memory copy and protocol conversion in traditional gateway scheme;Combined with the fault accurate positioning mechanism based on domain work counter (WC), can compare in real time and quickly lock the broken wire slave station index, solve the problem that fault positioning is fuzzy in the prior art, long maintenance cycle;In addition, introduce the CNN long-time encoder of 66 frames history observation of stacking and gradient penalty loss strategy, enhance the feature extraction capability to motion mode and inhibit joint action buffeting, so as to guarantee high real-time communication, significantly improve the control stability, response speed and fault fast positioning and safety protection ability of robot system.

Description

Technical Field

[0001] This invention relates to the field of robotics, specifically to a robot control system and method based on deep reinforcement learning and heterogeneous real-time bus. Background Technology

[0002] With the rapid development of robotics technology, higher demands are being placed on the real-time performance, flexibility, and intelligence of robot control systems. Existing robot control systems typically adopt a layered architecture. The bottom layer connects joint drivers and actuators via industrial fieldbuses (such as EtherCAT), responsible for high-frequency real-time data acquisition and control command issuance. The upper layer runs on a general-purpose operating system (such as Linux), utilizing the ROS (Robot Operating System) framework for logic planning, perception processing, and intelligent decision-making.

[0003] In traditional heterogeneous system communication schemes, data interaction between the real-time control layer (based on an EtherCAT master station) and the non-real-time computing layer (based on ROS nodes) typically relies on gateway translation or frequent inter-process communication (IPC). This traditional approach has the following significant problems: Large communication latency and jitter: Data needs to be copied and converted multiple times between different memory spaces, different protocol stacks, and even different processes, which increases end-to-end latency and makes it difficult to guarantee microsecond-level determinism, thus failing to meet the control requirements of highly dynamic robots (such as bipedal walking and high-speed grasping).

[0004] High resource consumption: Frequent memory copying and context switching consume a large amount of CPU resources, limiting the system's ability to process complex algorithms with limited computing power.

[0005] Difficulty in fault location: In complex EtherCAT networks, when a slave station disconnects or communication fails, traditional methods often only report a "domain error" and cannot accurately locate the specific faulty slave station. This results in a long time to troubleshoot after the system is shut down, affecting the availability of the robot.

[0006] Lack of long-term motion pattern memory: Existing reinforcement learning controllers mostly make decisions based on single-frame or short-sequence observations, making it difficult to capture long-term dependencies and inertial characteristics in robot motion. This results in insufficient motion stability under complex terrain or disturbances, and joint chattering is prone to occur. Summary of the Invention

[0007] This invention aims to provide a robot control system and method based on deep reinforcement learning and heterogeneous real-time bus, in order to solve the technical problems existing in the prior art, such as high data exchange latency between heterogeneous buses, lack of zero-copy mechanism, and inaccurate fault location.

[0008] To achieve the above objectives, the technical solution adopted by this invention is: a robot control system based on deep reinforcement learning and heterogeneous real-time bus, comprising: The heterogeneous real-time bus layer includes an EtherCAT bus and a DDS data distribution service bus. The EtherCAT bus is used to connect robot joint actuators and dexterous hand slaves to exchange data in real time at a 1kHz cycle. A shared memory exchange layer is defined, which defines a unified data structure SharedData_t to realize zero-copy data exchange between the EtherCAT real-time control layer and the ROS2 node. The shared memory exchange layer is created by the EtherCAT master process, which creates named memory objects and maps them to the address space through mmap. The ROS2 application communication layer includes a joint command publishing node, a joint status subscription node, and an IMU data publishing node. The ROS2 application communication layer interacts with the shared memory exchange layer through the DDS protocol. A deep reinforcement learning decision layer, comprising a CNN long-time encoder, a state estimator MLP, and a policy network Actor, wherein the CNN long-time encoder extracts motion patterns from stacked 66 frames of observations; The EtherCAT real-time control layer includes an IgHEtherCAT master station, a domain work counter monitoring module, and a fault precise location module. The EtherCAT real-time control layer reads control commands from the shared memory exchange layer and writes them to the TxPDO at a 1kHz cycle, and reads feedback from the RxPDO and writes it to the shared memory exchange layer.

[0009] Preferably, the total training loss of the deep reinforcement learning decision layer includes gradient penalty loss. The gradient penalty loss is used to suppress robot joint chattering, wherein For the gradient operator with respect to the observed input o, Let N be the log probability of the policy's output action given the observations, and let N be the batch size sampled during training.

[0010] Preferably, the fault precise location module maintains the expected working counter accumulation value wc_steps[] for each slave station. When the actual working counter actual_wc does not match the expectation, it determines the first index i that satisfies actual_wc≤wc_steps[i] by traversing the wc_steps[] array. The index i corresponds to the first faulty slave station and triggers an emergency shutdown.

[0011] Preferably, the inference model of the deep reinforcement learning decision layer is trained in PyTorch and exported in TorchScriptJIT or ONNX format. The CNN long-term encoder, state estimator and Actor network are encapsulated in the ExportedDH class for use by C++ real-time nodes.

[0012] Preferably, the SharedData_t structure defined in the shared memory exchange layer includes joint control data tx_data, joint feedback data rx_data, dexterous hand data ryhand, master status master_status, and synchronization primitives pthread_mutex_t+pthread_cond_t. The synchronization primitives are used to protect access to critical fields. The lock-free counters tx_data_cnt and rx_data_cnt embedded in the shared memory header are used to notify control instruction updates and status feedback updates, respectively.

[0013] Preferably, the domain working counter monitoring module in the EtherCAT real-time control layer obtains the actual working counter of the current domain through ecrt_domain_state() and compares it with the expected working counter expected_wc. If they are inconsistent, the fault precise location process is triggered. The fault precise location process traverses the pre-calculated wc_steps[] array to determine the specific disconnected slave station.

[0014] Preferably, the CNN long-time encoder of the deep reinforcement learning decision layer has 66 input channels, corresponding to 66 stacked historical observation data frames. The sampling interval of each observation frame is consistent with the real-time control cycle of the robot system, and the time window covered by the 66 frames is 66 milliseconds.

[0015] Preferably, the EtherCAT real-time control layer includes a domain1, and the PDO entries of the joint actuators and dexterous hand slaves are registered to this domain through ecrt_domain_reg_pdo_entry_list(); All PDOs within domain1 are exchanged within the same bus cycle to ensure data consistency.

[0016] Preferably, the data exchange process of the shared memory exchange layer includes: after receiving a message, the ROS2 node writes it to the shared memory through a callback function; the EtherCAT master station periodic task reads control instructions from the shared memory at a 1kHz cycle and writes them to the corresponding slave station's TxPDO buffer; and the EtherCAT periodic task writes the feedback data read from the RxPDO to the shared memory.

[0017] On the other hand, this invention proposes a robot control method based on deep reinforcement learning and heterogeneous real-time bus, comprising the following steps: S1: Start the EtherCAT master process, create a shared memory object, initialize the mutex and condition variable, set data_valid=0, scan the bus and configure the PDO (process data object) mapping of the joint driver and the dexterous hand slave, register the domain entry offset, enable the distributed clock with a period of 1ms; activate the master and start the 1kHz real-time loop. The following steps are executed sequentially in each EtherCAT cycle (1ms): Clock synchronization: Call ecrt_master_application_time() (the function to set the application time of the master station) and ecrt_master_sync_slave_clocks() to synchronize the clocks of all slave stations.

[0018] Receive EtherCAT frames: ecrt_master_receive(master).

[0019] Processing domain data: ecrt_domain_process(domain1) unpacks the received process data object into the domain memory.

[0020] S2: Start the ROS2 node, start the hipnuc_imu node via ros2launch to publish IMU data, start the jointcmd_pub node to subscribe to expected joint commands, start the ry_joint_state node to open an existing shared memory object, wait for the data_valid flag, read rx_data and publish the joint state; S3: Start the deep reinforcement learning inference node, load the ExportedDH model, subscribe to joint state and IMU data, perform inference through ONNX runtime, output joint commands and publish them to the desired joint command Topic. S4: In each EtherCAT cycle, call ecrt_master_receive() to receive EtherCAT frames, call ecrt_domain_process() to unpack the received RxPDO data into the domain memory area, and read control instructions from the shared memory, convert them to units, and write them into the TxPDO location in the domain memory. S5: After converting the RxPDO feedback data of each joint to a unit, store it in the rx_data field of the shared memory, and increment rx_data_cnt. If a change in rx_data_cnt is detected, lock and copy the rx_data field and assemble and publish the JointState message. S6: Call ecrt_domain_state() to get the domain status. If the actual working counter is not equal to the expected working counter, iterate through the wc_steps[] array to determine the first fault slave index, output the fault information, and call emergency_stop_all() to set all joint control words to 0x0002 for emergency shutdown.

[0021] Compared with the prior art, the beneficial effects of the present invention are as follows: This invention constructs a heterogeneous real-time bus architecture including a shared memory exchange layer and utilizes mmap mapping technology to achieve zero-copy data interaction between the EtherCAT real-time control layer and the ROS2 application communication layer. This significantly reduces communication latency and CPU resource consumption caused by multiple memory copies and protocol conversions in traditional gateway solutions. Simultaneously, by combining a fault-accurate location mechanism based on domain working counters (WC), it can compare and quickly locate the index of disconnected slave stations in real time, solving the problems of ambiguous fault location and long repair cycles in existing technologies. Furthermore, by introducing a CNN long-term encoder with stacked 66 frames of historical observations and a gradient penalty loss strategy, it enhances the feature extraction capability of motion patterns and suppresses joint jitter. Thus, while ensuring high real-time communication, it significantly improves the control stability, response speed, and rapid fault location and safety protection capabilities of the robot system. Attached Figure Description

[0022] Figure 1 This is a block diagram of the robot control system based on deep reinforcement learning and heterogeneous real-time bus of the present invention. Figure 2 This is a flowchart of the robot control method based on deep reinforcement learning and heterogeneous real-time bus of the present invention. Detailed Implementation

[0023] The following description is intended to disclose the invention and enable those skilled in the art to implement it. The preferred embodiments described below are merely examples, and other obvious variations will occur to those skilled in the art.

[0024] like Figure 1 As shown, this invention proposes a robot control system based on deep reinforcement learning and a heterogeneous real-time bus, comprising: The heterogeneous real-time bus layer comprises an EtherCAT (Ethernet Control Automation Technology) bus and a DDS (Data Distribution Service) bus. The EtherCAT bus connects robot joint actuators and dexterous hand slave stations, exchanging data in real-time at a 1kHz cycle. The DDS bus connects the host computer decision-making system and non-real-time task nodes. Data exchange between EtherCAT and DDS is achieved through shared memory (SharedData_t) as a bridge, eliminating the need for a separate protocol converter.

[0025] The conversion mechanism between EtherCAT and DDS is as follows: They are not directly converted through a separate "protocol converter / gateway" device. Instead, shared memory (SharedData_t) serves as the data exchange center. ROS2 (Robot Operating System 2) nodes and the EtherCAT master station's periodic tasks read from and write to the shared memory from both ends, achieving indirect data conversion and synchronization. The shared memory itself does not perform protocol conversion; it only acts as a bridge for data exchange.

[0026] DDS Side: ROS2 Topic → Shared Memory: In the ROS2 / DDS communication layer, data is transmitted via the DDS publish / subscribe mechanism according to a preset message format (such as JointCmd.msg (joint command message file) and RyHandCmd.msg (dexterous hand command message file)). After receiving the message, the ROS2 node writes it to shared memory through the callback function in ry_joint_state.cpp (ROS2 joint state node source file).

[0027] Shared Memory: Data Intermediary: The SharedData_t structure defines uniform data fields that act as a "translation" intermediary between DDS message fields and EtherCAT PDO fields.

[0028] EtherCAT side: Shared memory → TxPDO (Transmission Process Data Object): The EtherCAT master station periodic task (ryhand6_b3.c) reads control commands from shared memory at a 1kHz cycle and writes them to the corresponding slave station's TxPDO buffer.

[0029] Reverse data flow: RxPDO (Receive Process Data Object) → Shared Memory → DDS: The reverse data flow also uses shared memory as a transit point. EtherCAT periodic tasks write feedback data read from RxPDO into shared memory; after a ROS2 node (ry_joint_state.cpp) detects a change in rx_data_cnt (Receive Data Counter), it reads data from shared memory and publishes it as a ROS2 Topic.

[0030] A shared memory exchange layer defines a unified data structure SharedData_t to implement zero-copy data exchange between the EtherCAT real-time control layer and the ROS2 node. The shared memory exchange layer is created by the EtherCAT master process, which creates named memory objects and maps them to the address space through mmap (memory mapping). Furthermore, the SharedData_t structure defined in the shared memory exchange layer includes joint control data tx_data, joint feedback data rx_data, dexterous hand data ryhand, master status master_status, and synchronization primitives pthread_mutex_t+pthread_cond_t. The synchronization primitives are used to protect access to critical fields. The lock-free counters tx_data_cnt and rx_data_cnt embedded in the shared memory header are used to notify control instruction updates and status feedback updates, respectively.

[0031] Shared memory can be created in the following ways: Using the EtherCAT master process as the creator, a named memory object (named / ethercat_data) is generated in the / dev / shm directory by calling shm_open (the shared memory open function). Its size is then set to the number of bytes in a SharedData_t structure using ftruncate (the file truncation function). Subsequently, mmap is called to map this object to the process's own address space, and the MAP_SHARED (shared mapping flag) is set to support multi-process sharing. To ensure cross-process synchronization, a mutex (pthread_mutex_t) and a condition variable (pthread_cond_t) with the PTHREAD_PROCESS_SHARED (process sharing attribute) are embedded in the shared memory header. Two additional lock-free counters, tx_data_cnt and rx_data_cnt, are designed to notify control command updates and status feedback updates, respectively. After a ROS2 node (such as ry_joint_state) starts, it opens the same shared memory object in read-only / read-write mode using shm_open + mmap. A mutex lock protects access to critical fields, and data readiness is detected by polling a counter or waiting for a condition variable. When all processes exit, the master process calls munmap (to unmap the memory) to unmap it, and then executes shm_unlink (to delete the shared memory object) to delete the shared memory object, completing resource reclamation. This solution achieves zero-copy, low-latency data exchange between EtherCAT real-time tasks and ROS2 non-real-time tasks without requiring a separate gateway or protocol converter.

[0032] The ROS2 application communication layer includes a joint command publishing node, a joint state subscription node, and an IMU (Inertial Measurement Unit) data publishing node. The ROS2 application communication layer interacts with the shared memory exchange layer through the DDS protocol. The deep reinforcement learning decision layer includes a CNN (Convolutional Neural Network) long-term encoder, a state estimator MLP (Multilayer Perceptron), and a policy network Actor. The CNN long-term encoder extracts motion patterns from a stack of 66 frames of observations, the state estimator estimates missing states (linear velocities) from a 5-frame short-term window, and the policy network outputs joint control commands. The specific time window for the 66 frames is explained as follows: The input channels of the CNN long-term encoder are in_channels=66, corresponding to 66 stacked historical observation data frames. The sampling interval of each observation frame is consistent with the real-time control cycle of the robot system; in this embodiment, the EtherCAT real-time control layer runs at a frequency of 1kHz (i.e., 1 millisecond interval per frame), so the time window covered by the 66 frames is 66 milliseconds (ms). This time window can be adjusted according to the robot's dynamic characteristics or application scenario (e.g., it can be appropriately extended for a slower-moving robotic arm), but in the default embodiment, 66ms is used as the effective time window for extracting long-term motion patterns. If a different control cycle (e.g., 10ms) is used during the simulation training phase, the corresponding time window is 0.66 seconds; however, in actual deployment, 66 frames will still be sampled at 1kHz and stacked to ensure consistency between real-time performance and training settings.

[0033] Furthermore, the total training loss of a deep reinforcement learning decision layer includes gradient penalty loss. The gradient penalty loss is used to suppress robot joint chattering, wherein: Gradient penalty loss is added as a regularization term to the total training loss of the deep reinforcement learning decision layer to suppress jitter in the policy output action. Summing all samples within the batch; Here, is the gradient operator with respect to the observation input o; o is the observation tensor (containing long and short time history information) input to the policy network. Let be the log probability of the policy's output action given observations; for a Gaussian policy, Obey the mean variance is The normal distribution; N is the batch size during training, i.e. the number of samples used in one parameter update; The L2 norm of a vector is the square of its components, i.e., the sum of the squares of the vector's components. Here, the gradient vector is calculated. The sum of squares of each component.

[0034] This loss term penalizes the policy output for its sensitivity to the observed input: when the observation changes slightly, if the logarithmic probability of the action fluctuates drastically, the gradient norm will increase, resulting in a larger penalty. This is achieved by minimizing... This loss term forces the policy network to learn action mappings that are insensitive to observation noise, thereby significantly reducing joint chattering caused by sensor noise or latency during real robot deployment.

[0035] In addition, the inference model of the deep reinforcement learning decision layer is trained by PyTorch (a deep learning framework) and exported in TorchScriptJIT (just-in-time compilation) or ONNX (open neural network exchange) format. The CNN long-term encoder, state estimator and Actor network are encapsulated by the ExportedDH (exported deep reinforcement learning inference model class) class for C++ real-time nodes to call.

[0036] Specifically, the CNN long-time encoder of the deep reinforcement learning decision layer has 66 input channels, corresponding to 66 stacked historical observation data. The sampling interval of each observation frame is consistent with the real-time control cycle of the robot system, and the time window covered by the 66 frames is 66 milliseconds.

[0037] The EtherCAT real-time control layer includes an IgHEtherCAT master station, a domain work counter monitoring module, and a fault precise location module. The EtherCAT real-time control layer reads control commands from the shared memory exchange layer and writes them to the TxPDO at a 1kHz cycle, and reads feedback from the RxPDO and writes it to the shared memory exchange layer.

[0038] The fault precise location module maintains the expected cumulative working counter value wc_steps[] (working counter step size array) for each slave station. When the actual working counter actual_wc does not match the expectation, it iterates through the wc_steps[] array to determine the first index i that satisfies actual_wc ≤ wc_steps[i]. This index i corresponds to the first faulty slave station, and an emergency shutdown is triggered. Here, wc_steps[i] represents the expected sum of working counters when all the first i+1 slave stations are working normally; actual_wc is the actual working counter.

[0039] Furthermore, the domain working counter monitoring module in the EtherCAT real-time control layer obtains the actual working counter of the current domain through ecrt_domain_state() (EtherCAT domain state acquisition function) and compares it with the expected working counter expected_wc. If they are inconsistent, the fault precise location process is triggered. The fault precise location process traverses the pre-calculated wc_steps[] array to determine the specific disconnected slave station.

[0040] The EtherCAT real-time control layer includes a domain1, and the PDO entries of the joint drivers and dexterous hand slaves are registered to this domain through the ecrt_domain_reg_pdo_entry_list() (domain PDO entry registration function); all PDOs in the domain1 ((EtherCAT domain instance)) are exchanged within the same bus cycle to ensure data consistency.

[0041] In the EtherCAT master protocol stack (such as IgH EtherCAT Master), a domain is a logical data collection used to organize process data objects (PDOs) from multiple slave stations, enabling the master station to complete data exchange uniformly within a single bus cycle. Specifically: Data aggregation: A domain can contain multiple PDO entries (such as the position, speed, torque, etc. of multiple joints) from different slave stations. The master station can synchronously update the transmitted data (TxPDO) and received data (RxPDO) of all PDOs in the domain through a single domain processing operation.

[0042] Memory mapping: Each domain corresponds to a contiguous memory region, called the domain process data area. The master station maps the PDOs of each slave station to this region sequentially and records the memory offset of each PDO. Applications (such as the EtherCAT periodic task in this invention) can directly read and write domain memory through the offset, without needing to concern themselves with the independent message assembly of each slave station.

[0043] Synchronization and Consistency: All PDOs within the domain are exchanged within the same bus cycle to ensure data consistency. The master station processes the received frame data by calling `ecrt_domain_process()` (domain data processing function), unpacking the RxPDOs fed back by each slave station into the domain memory; at the same time, it packs the TxPDOs in the domain memory into the transmit frame queue using `ecrt_domain_queue()` (domain transmit queue function).

[0044] Working Counter (WC): Each domain maintains a working counter during communication to detect whether all PDOs within the domain have been correctly responded to by slaves. The master station expects the WC value to equal the sum of the number of slave operations involved in all PDO entries within the domain (e.g., each slave read operation counts as 1, write operation as 2, and both read and write operations as 3). If the actual WC does not match the expected value, it indicates a slave communication anomaly or a PDO mapping error.

[0045] Specifically, the data exchange process of the shared memory exchange layer includes: after receiving a message, the ROS2 node writes it to the shared memory through a callback function; the EtherCAT master station periodic task reads control commands from the shared memory at a 1kHz cycle and writes them to the corresponding slave station's TxPDO buffer; and the EtherCAT periodic task writes the feedback data read from the RxPDO to the shared memory.

[0046] On the other hand, this invention proposes a robot control method based on deep reinforcement learning and a heterogeneous real-time bus, such as... Figure 2 As shown, it includes the following steps: S1: Start the EtherCAT master process, create a shared memory object, initialize mutexes and condition variables, set data_valid=0, scan the bus and configure the PDO mapping of the joint driver and the dexterous hand slave, register the domain entry offset, enable the distributed clock with a period of 1ms; activate the master and start the 1kHz real-time loop. The following steps are executed sequentially in each EtherCAT cycle (1ms): Clock synchronization: Call ecrt_master_application_time() and ecrt_master_sync_slave_clocks() (slave clock synchronization functions) to synchronize all slave clocks.

[0047] Receive EtherCAT frames: ecrt_master_receive(master).

[0048] Processing domain data: ecrt_domain_process(domain1) unpacks the received process data object into the domain memory.

[0049] S2: Start the ROS2 node, start the hipnuc_imu node via ros2launch to publish IMU data, start the jointcmd_pub node to subscribe to the expected joint commands, start the ry_joint_state node to open the existing shared memory object, wait for the data_valid (data valid flag) flag, then read rx_data and publish the joint state; S3: Start the deep reinforcement learning inference node, load the ExportedDH model, subscribe to joint states and IMU data, perform inference through the ONNX runtime, output joint commands and publish them to the desired joint command Topic; S4: In each EtherCAT cycle, call ecrt_master_receive() (master station receive function) to receive EtherCAT frames, call ecrt_domain_process() to unpack the received RxPDO data into the domain memory area, and read control instructions from the shared memory, convert them to units, and write them into the TxPDO location in the domain memory. S5: After converting the RxPDO feedback data of each joint to a unit, store it in the rx_data field of the shared memory, and increment rx_data_cnt. If a change in rx_data_cnt is detected, lock and copy the rx_data field and assemble and publish the JointState message. S6: Call ecrt_domain_state() to get the domain status. If the actual working counter is not equal to the expected working counter, iterate through the wc_steps[] array to determine the first fault slave index, output the fault information, and call emergency_stop_all() to set all joint control words to 0x0002 for emergency shutdown.

[0050] The process of reading shared memory control instructions is as follows: Read shared_data->tx_data_cnt. If the value is different from the last read value, it indicates that the ROS2 node has written new joint instructions; Read the target position, velocity, and control word from shared_data->joints[0..29].tx_data, convert them to units (radians → encoder value), and write them to the corresponding joint's transmission process data object buffer; Read the dexter instructions from shared_data->ryhand_hcmd (dexter advanced motion instruction structure), convert them to process data object data through BusWrite (bus write callback function), and write them to pTxPdo1 / pTxPdo2 (dexter transmit PDO pointers); Finally, update last_tx_cnt (last transmission count).

[0051] The process of writing joint feedback into shared memory is as follows: Read the data objects of each joint receiving process (actual position, speed, torque, status word), convert them to units (encoder value → radians, 0.01rpm → rad / s) and store them in shared_data->joints[i].rx_data; the dexterous hand feedback is extracted from pRxPdo1 / pRxPdo2, parsed into a FingerServoInfo_t (finger servo information structure) structure, filled into shared_data->ryhand[0 / 1].rx_data, and shared_data->rx_data_cnt is incremented.

[0052] The fault detection and precise location process is as follows: Call `ecrt_domain_state(domain1, &ds)` to obtain the domain status; if `ds.working_counter` is not equal to `expected_wc`, traverse the pre-calculated `wc_steps[]` array and find the first index `i` that satisfies `actual_wc ≤ wc_steps[i]`; output "[ERROR] Slave i communication interrupted!", call `emergency_stop_all()` (emergency stop function) to set all joint control words to 0x0002 (fast stop control word), and simultaneously set `shared_data->shutdown_flag=1`. `shutdown_flag` is the shutdown flag.

[0053] The EtherCAT frame sending process is as follows: the sending process data object is packaged through ecrt_domain_queue(domain1), and then sent through ecrt_master_send() (the master station sending function).

[0054] Meanwhile, the ROS2 node ry_joint_state performs the following operations in a 500μs timer loop: read rx_data_cnt, if its value changes, lock and copy the rx_data field, unlock and assemble the JointState message and publish it to the / state / joint topic; the dexterous hand state processing flow is the same.

[0055] Furthermore, the offline training and online deployment process for deep reinforcement learning decision-making is as follows: Training Phase: A URDF (Unified Robot Description Format) model with a structure consistent with the real robot was built in the NVIDIA Isaac Gym (NVIDIA Reinforcement Learning Simulation Environment) simulation environment. The `train.py` script was run to create the ActorCriticDH network: the input layer receives 66 stacked observations (47 dimensions per frame), extracting long-term motion features through one-dimensional convolutional layers (kernel sizes of 6 and 4, and output channels of 32 and 16 respectively); simultaneously, short-term information is extracted from the most recent 5 observations, and the linear velocity (v_x, v_y, ω_z) is predicted by a state estimation multilayer perceptron; the long-term features, short-term observations, and estimated velocities are concatenated and input into the policy network multilayer perceptron (three-layer structure, dimensions 256-256-128), outputting position commands for 12 joints. The loss function includes policy loss, value loss, state estimation MSE loss, and gradient penalty loss (coefficient 1e-5). The model was exported after training until the average step size reward stabilized.

[0056] Model export: Run export_policy_dh.py (model export script) to convert the trained network into the ExportedDH class, retain only the forward path required for inference (state estimator + long-term encoder + policy network), and export it as an just-in-time compiled file or an ONNX format file.

[0057] Simulation verification: Run sim2sim.py (simulation transfer script) to load the exported model, and verify it using the same robot model in the Mujoco (multi-joint contact physics simulation engine). It is confirmed that the strategy can be directly transferred without retraining, and the tracking error and stability performance are similar to those in the Isaac Gym environment.

[0058] Real robot deployment: The C++ inference node loads the model through the ONNX runtime and subscribes to the / state / joint and / IMU_data topics; it performs inference once per inference cycle (2ms), outputs the target positions of the 12 joints, and publishes them to the / desired / joint topic through the jointcmd_pub node to form a closed-loop control.

[0059] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely principles of the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the claimed invention. The scope of protection claimed by the appended claims and their equivalents is defined.

Claims

1. A robot control system based on deep reinforcement learning and heterogeneous real-time bus, characterized in that, include: The heterogeneous real-time bus layer includes an EtherCAT bus and a DDS data distribution service bus. The EtherCAT bus is used to connect robot joint actuators and dexterous hand slave stations to exchange data in real time at a 1kHz cycle. The DDS bus is used to connect the host computer decision system and non-real-time task nodes. A shared memory exchange layer is defined, which defines a unified data structure SharedData_t to realize zero-copy data exchange between the EtherCAT real-time control layer and the ROS2 node. The shared memory exchange layer is created by the EtherCAT master process, which creates named memory objects and maps them to the address space through mmap. The ROS2 application communication layer includes a joint command publishing node, a joint status subscription node, and an IMU data publishing node. The ROS2 application communication layer interacts with the shared memory exchange layer through the DDS protocol. A deep reinforcement learning decision layer, comprising a CNN long-time encoder, a state estimator MLP, and a policy network Actor, wherein the CNN long-time encoder extracts motion patterns from stacked 66 frames of observations; The EtherCAT real-time control layer includes an IgHEtherCAT master station, a domain work counter monitoring module, and a fault precise location module. The EtherCAT real-time control layer reads control commands from the shared memory exchange layer and writes them to the TxPDO at a 1kHz cycle, and reads feedback from the RxPDO and writes it to the shared memory exchange layer.

2. The robot control system based on deep reinforcement learning and heterogeneous real-time bus according to claim 1, characterized in that, The total training loss of the deep reinforcement learning decision layer includes gradient penalty loss. The gradient penalty loss is used to suppress robot joint chattering, wherein; This represents the gradient penalty loss value. Here is the gradient operator with respect to the observed input o; For strategy In a given observation Down Output Action The logarithmic probability; N is the batch size sampled during training; It is the square of the L2 norm of the vector.

3. The robot control system based on deep reinforcement learning and heterogeneous real-time bus according to claim 1, characterized in that, The fault precise location module maintains the expected cumulative working counter value wc_steps[] for each slave station. When the actual working counter actual_wc does not match the expectation, it traverses the wc_steps[] array to determine the first index i that satisfies actual_wc≤wc_steps[i]. The index i corresponds to the first faulty slave station and triggers an emergency shutdown. Here, wc_steps[i] represents the expected sum of working counters when all the first i+1 slave stations are working normally.

4. The robot control system based on deep reinforcement learning and heterogeneous real-time bus according to claim 1, characterized in that, The inference model of the deep reinforcement learning decision layer is trained in PyTorch and exported in TorchScriptJIT or ONNX format. The CNN long-term encoder, state estimator and Actor network are encapsulated in the ExportedDH class for C++ real-time nodes to call.

5. A robot control system based on deep reinforcement learning and heterogeneous real-time bus according to claim 1, characterized in that, The SharedData_t structure defined in the shared memory exchange layer includes joint control data tx_data, joint feedback data rx_data, dexterous hand data ryhand, master status master_status, and synchronization primitives pthread_mutex_t+pthread_cond_t. The synchronization primitives are used to protect access to critical fields. The lock-free counters tx_data_cnt and rx_data_cnt embedded in the shared memory header are used to notify control instruction updates and status feedback updates, respectively.

6. A robot control system based on deep reinforcement learning and heterogeneous real-time bus according to claim 1, characterized in that, The domain working counter monitoring module in the EtherCAT real-time control layer obtains the actual working counter of the current domain through ecrt_domain_state() and compares it with the expected working counter expected_wc. If they are inconsistent, the fault precise location process is triggered. The fault precise location process traverses the pre-calculated wc_steps[] array to determine the specific disconnected slave station.

7. A robot control system based on deep reinforcement learning and heterogeneous real-time bus according to claim 1, characterized in that, The CNN long-time encoder of the deep reinforcement learning decision layer has 66 input channels, corresponding to 66 stacked historical observation data frames. The sampling interval of each observation frame is consistent with the real-time control cycle of the robot system, and the time window covered by the 66 frames is 66 milliseconds.

8. A robot control system based on deep reinforcement learning and heterogeneous real-time bus according to claim 1, characterized in that, The EtherCAT real-time control layer includes a domain1, and the PDO entries of the joint actuators and dexterous hand slaves are registered to this domain through ecrt_domain_reg_pdo_entry_list(). All PDOs within domain1 are exchanged within the same bus cycle to ensure data consistency.

9. A robot control system based on deep reinforcement learning and heterogeneous real-time bus according to claim 1, characterized in that, The data exchange process of the shared memory exchange layer includes: after receiving a message, the ROS2 node writes it to the shared memory through a callback function; the EtherCAT master station periodic task reads control instructions from the shared memory at a 1kHz cycle and writes them to the corresponding slave station's TxPDO buffer; and the EtherCAT periodic task writes the feedback data read from the RxPDO to the shared memory.

10. A robot control method based on deep reinforcement learning and heterogeneous real-time bus according to any one of claims 1-9, characterized in that, Includes the following steps: S1: Start the EtherCAT master process, create a shared memory object, initialize mutexes and condition variables, scan the bus and configure the PDO mapping of the joint driver and the dexterous hand slave, register the domain entry offset, enable the distributed clock and activate the master to start a 1kHz real-time loop. S2: Start the ROS2 node, start the hipnuc_imu node via ros2launch to publish IMU data, start the jointcmd_pub node to subscribe to expected joint commands, start the ry_joint_state node to open an existing shared memory object, wait for the data_valid flag, read rx_data and publish the joint state; S3: Start the deep reinforcement learning inference node, load the ExportedDH model, subscribe to joint states and IMU data, perform inference through the ONNX runtime, output joint commands and publish them to the desired joint command Topic; S4: In each EtherCAT cycle, call ecrt_master_receive() to receive EtherCAT frames, call ecrt_domain_process() to unpack the received RxPDO data into the domain memory area, and read control instructions from the shared memory, convert them to units, and write them into the TxPDO location in the domain memory. S5: After converting the RxPDO feedback data of each joint to a unit, store it in the rx_data field of the shared memory, and increment rx_data_cnt. If a change in rx_data_cnt is detected, lock and copy the rx_data field and assemble and publish the JointState message. S6: Call ecrt_domain_state() to get the domain status. If the actual working counter is not equal to the expected working counter, iterate through the wc_steps[] array to determine the first fault slave index, output the fault information, and call emergency_stop_all() to set all joint control words to 0x0002 for emergency shutdown.

Images