Control system and method

Abstract

The embodiment of the present specification provides a control system and method, the control system comprises a control end and a simulation end, the simulation end and the control end are connected through a virtual hardware interface in a simulation mode, the virtual hardware interface comprises a control input channel and a sensor output channel, and the method comprises the following steps: the simulation end determines motion data according to a simulation task corresponding to a control program; the motion data is converted into a virtual sensor frame according to a preset mapping relationship and is sent to the control end through the sensor output channel; the control end constructs an actuator command according to a control command and the virtual sensor frame associated with the simulation task through the control program and sends the actuator command to the simulation end through the control input channel; the simulation end updates the state of a dynamic model based on the actuator command; and the control end switches the virtual hardware interface to an entity hardware interface between the control end and the target robot when the control program meets the simulation condition of the simulation task according to the updated state, so as to drive the target robot through the control program.

Description

Technical Field

[0001] The embodiments in this specification relate to the field of robot control technology, and particularly to control systems and methods. Background Technology

[0002] The development of robot control programs heavily relies on physical components such as joints, inertial measurement units, force sensors, motor drivers, and robot middleware to achieve precise motion control and environmental interaction. During the algorithm verification phase, simulation systems are widely used to simulate robot behavior, but traditional methods have significant limitations. Some simulation schemes build simplified controller logic within the simulator, resulting in fundamental differences in architecture from the actual control program. Other schemes design dedicated input / output interfaces for the simulation environment, leading to inconsistencies between the simulation controller and the actual program in areas such as data structure organization, joint arrangement order, time reference source management, state machine operation logic, runtime library dependencies, and upper-level interface specifications. This inconsistency necessitates additional adaptation work after the control algorithm passes simulation verification, including reconnecting to the physical robot's software development kit, configuring robot operating system topics, adjusting device naming rules, and setting up underlying network communication endpoints. This repetitive adaptation process not only prolongs the development cycle and increases manpower investment but may also cause differences in the operating environment not covered in the simulation phase to manifest during physical robot deployment, leading to potential operational failure risks. Meanwhile, some existing simulation schemes only provide pseudo-sensor data by simulating the return results of the upper-level application interface, without actually running the control program that is completely identical to the real machine; while other schemes can start the control program, but there is no unified mapping mechanism between sensor data and actuator commands, and it is impossible to form a complete dynamic closed loop with the participation of the real control state machine. As a result, the simulation results cannot accurately reflect the dynamic response characteristics of the physical robot. Therefore, an effective solution is urgently needed to solve the above problems. Summary of the Invention

[0003] In view of this, embodiments of this specification provide a control system. One or more embodiments of this specification also relate to control methods, control devices, a computing device, a computer-readable storage medium, and a computer program product, to address technical deficiencies in the prior art.

[0004] According to a first aspect of the embodiments of this specification, a control system is provided, including a control terminal and a simulation terminal. The control terminal is configured with a control program, and the simulation terminal is configured with a dynamic model of a target robot. The simulation terminal and the control terminal are connected in simulation mode through a virtual hardware interface, and the virtual hardware interface includes a control input channel and a sensor output channel, comprising: The simulation terminal is used to determine motion data according to the simulation task corresponding to the control program; convert the motion data into virtual sensor frames according to a preset mapping relationship, and send them to the control terminal through the sensor output channel; The control terminal is used to construct actuator commands based on the control commands associated with the simulation task and the virtual sensor frame through the control program, and send them to the simulation terminal through the control input channel. The simulation terminal is used to update the state of the dynamic model based on the executor commands; The control terminal is used to switch the virtual hardware interface to the physical hardware interface between the control terminal and the target robot when the control program meets the simulation conditions of the simulation task based on the updated state, so as to drive the target robot through the control program.

[0005] According to a second aspect of the embodiments of this specification, a control method is provided, applied to a control terminal, wherein the control terminal is configured with a control program, the control terminal connects to a simulation terminal in simulation mode through a virtual hardware interface, the simulation terminal is configured with a dynamic model of an associated target robot, and the virtual hardware interface includes a control input channel and a sensor output channel, comprising: The simulation terminal receives a virtual sensor frame sent through the sensor output channel, wherein the virtual sensor frame is obtained by converting motion data according to a preset mapping relationship, and the motion data is determined according to the simulation task corresponding to the control program. The control program constructs actuator commands based on the control commands associated with the simulation task and the virtual sensor frame, and sends them to the simulation terminal through the control input channel. Based on the state of the dynamic model updated by the simulation terminal according to the actuator command, and after determining that the control program meets the simulation conditions of the simulation task, the virtual hardware interface is switched to the physical hardware interface between the control terminal and the target robot, for driving the target robot through the control program.

[0006] According to a third aspect of the embodiments of this specification, a control device is provided, applied to a control terminal, the control terminal being configured with a control program, the control terminal being connected to a simulation terminal in a simulation mode via a virtual hardware interface, the simulation terminal being configured with a dynamic model of an associated target robot, and the virtual hardware interface including a control input channel and a sensor output channel, comprising: The receiving module is configured to receive virtual sensor frames sent by the simulation terminal through the sensor output channel, wherein the virtual sensor frames are obtained by converting motion data according to a preset mapping relationship, and the motion data is determined according to the simulation task corresponding to the control program. The construction module is configured to construct actuator commands through the control program based on the control commands associated with the simulation task and the virtual sensor frame, and send them to the simulation terminal through the control input channel. The determination module is configured to, based on the state of the dynamic model updated by the simulation terminal according to the actuator command, and if the control program satisfies the simulation conditions of the simulation task, switch the virtual hardware interface to the physical hardware interface between the control terminal and the target robot, for driving the target robot through the control program.

[0007] According to a fourth aspect of the embodiments of this specification, a computing device is provided, comprising: Memory and processor; The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions, which, when executed by the processor, implement the steps of the above-described control method.

[0008] According to a fifth aspect of the embodiments of this specification, a computer-readable storage medium is provided that stores computer-executable instructions that, when executed by a processor, implement the steps of the control method described above.

[0009] According to a sixth aspect of the embodiments of this specification, a computer program product is provided, including a computer program or instructions that, when executed by a processor, implement the steps of the control method described above.

[0010] The control system provided in this embodiment includes a control terminal and a simulation terminal. The control terminal is configured with a control program, and the simulation terminal is configured with a dynamic model associated with the target robot. The simulation terminal and the control terminal are connected in simulation mode through a virtual hardware interface, which includes a control input channel and a sensor output channel. Based on this, the simulation terminal determines motion data according to the simulation task corresponding to the control program; converts the motion data into virtual sensor frames according to a preset mapping relationship, and sends them to the control terminal through the sensor output channel; the control terminal constructs actuator commands based on the control commands associated with the simulation task and the virtual sensor frames through the control program, and sends them to the simulation terminal through the control input channel; the simulation terminal updates the state of the dynamic model based on the actuator commands; if the control terminal determines that the control program meets the simulation conditions of the simulation task based on the updated state, it switches the virtual hardware interface to a physical hardware interface between the control terminal and the target robot, for driving the target robot through the control program. By establishing a closed-loop communication mechanism between the control end and the simulation end through a virtual hardware interface, lossless conversion of motion data and virtual sensor frames is achieved using a preset mapping relationship. When the simulation conditions are met, the system seamlessly switches to the physical hardware interface, allowing the control program to drive the target robot without modification. This solves the problem of inconsistency between the simulation environment and the physical robot control program, avoids repeated adaptation work, shortens the development cycle, and reduces the risk of operational failures. Attached Figure Description

[0011] Figure 1 This is a schematic diagram of the structure of a control system provided in one embodiment of this specification; Figure 2 This is a schematic diagram of the simulation and actual machine switching process in a control system provided in one embodiment of this specification; Figure 3 This is a flowchart illustrating the processing procedure of a control system according to one embodiment of this specification; Figure 4 This is a flowchart illustrating a control method provided in one embodiment of this specification; Figure 5 This is a schematic diagram of the structure of a control device provided in one embodiment of this specification; Figure 6 This is a structural block diagram of a computing device provided in one embodiment of this specification. Detailed Implementation

[0012] Many specific details are set forth in the following description to provide a full understanding of this specification. However, this specification can be implemented in many other ways than those described herein, and those skilled in the art can make similar extensions without departing from the spirit of this specification. Therefore, this specification is not limited to the specific implementations disclosed below.

[0013] The terminology used in one or more embodiments of this specification is for the purpose of describing particular embodiments only and is not intended to be limiting of the one or more embodiments of this specification. The singular forms “a,” “described,” and “the” as used in one or more embodiments of this specification and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in one or more embodiments of this specification refers to and includes any or all possible combinations of one or more associated listed items.

[0014] It should be understood that although the terms first, second, etc., may be used to describe various information in one or more embodiments of this specification, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, first may also be referred to as second without departing from the scope of one or more embodiments of this specification, and similarly, second may also be referred to as first. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to a determination."

[0015] Furthermore, it should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in one or more embodiments of this specification are all information and data authorized by the user or fully authorized by all parties. Moreover, the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0016] This specification provides a control system. One or more embodiments of this specification also relate to control methods, control devices, a computing device, a computer-readable storage medium, and a computer program product, which will be described in detail in the following embodiments.

[0017] See Figure 1 , Figure 1This is a schematic diagram of a control system provided in one embodiment of this specification. The control system 100 includes a simulation terminal 110 and a control terminal 120. The control terminal 120 is configured with a control program, and the simulation terminal 110 is configured with a dynamic model of the associated target robot. The simulation terminal 110 and the control terminal 120 are connected in simulation mode through a virtual hardware interface, and the virtual hardware interface includes a control input channel and a sensor output channel, including: The simulation terminal 110 is used to determine motion data according to the simulation task corresponding to the control program; convert the motion data into virtual sensor frames according to a preset mapping relationship, and send them to the control terminal through the sensor output channel; The control terminal 120 is used to construct actuator commands based on the control commands associated with the simulation task and the virtual sensor frame through the control program, and send them to the simulation terminal through the control input channel. The simulation terminal 110 is used to update the state of the dynamic model based on the executor command; The control terminal 120 is used to switch the virtual hardware interface to the physical hardware interface between the control terminal and the target robot when the control program meets the simulation conditions of the simulation task based on the updated state, so as to drive the target robot through the control program.

[0018] Specifically, the control unit refers to a computing unit or device configured with a control program. Its main function is to run control algorithms and generate actuator instructions based on sensor inputs and control commands. The simulation unit refers to a computing unit or device configured with a dynamic model of the target robot and the dynamic model itself. Its main function is to simulate the physical behavior of the target robot, receive actuator instructions and update the model state, and generate virtual sensor data based on the model state. The control program is the software code used to drive the target robot to perform specific tasks, and it includes the robot's control logic, state machine, and algorithms. The dynamic model is a mathematical model describing the geometric relationship between the robot's joint positions, postures, and the end effector positions and postures, without considering forces or mass. The virtual hardware interface is a software layer that simulates the hardware interface of a real robot in the simulation environment, allowing the control program to interact with the simulation environment without modifying the code.

[0019] Accordingly, the control input channel refers to the data path in the virtual hardware interface used to send actuator commands from the control end to the simulation end. The sensor output channel refers to the data path in the virtual hardware interface used to send virtual sensor frames from the simulation end to the control end. The simulation task refers to the specific operational goal or behavioral sequence set for the robot in the simulation environment. Motion data refers to the raw data describing the robot's motion state, determined at the simulation end according to the simulation task, such as joint angles, velocity, acceleration, and end effector position. The virtual sensor frame refers to the data packet that converts motion data into a sensor data format recognizable by the control program according to a preset mapping relationship and sends it through the sensor output channel. The actuator command refers to the instruction generated by the control program based on the virtual sensor frame and control commands, used to drive the robot's actuators (such as motors). The physical hardware interface refers to the actual physical interface for data communication and control command transmission between the control end and the target robot. The target robot refers to the physical robot device that this control system ultimately needs to drive.

[0020] This embodiment provides a control system including a control terminal and a simulation terminal. The control terminal is configured with a control program responsible for generating control commands. The simulation terminal is configured with a dynamic model of the associated target robot, which is used to simulate the motion characteristics of the target robot. The simulation terminal and the control terminal are connected in simulation mode through a virtual hardware interface. This virtual hardware interface is designed to include control input channels and sensor output channels to achieve bidirectional data communication.

[0021] Specifically, the control unit can be configured with various types of control programs. For example, the control program could be a joint position controller based on a PID algorithm, used to precisely control the movement of the robot's joints; or, it could be a force control algorithm based on force / torque sensors, used to achieve compliant interaction between the robot and its environment. The dynamic model configured on the simulation unit could be a simplified geometric model, used only to calculate the correspondence between joint angles and end effector positions; or it could be a complex model containing an inverse kinematics solver, used to calculate the required joint angles based on the target position.

[0022] The simulation terminal is used to determine motion data based on the simulation task corresponding to the control program. For example, the simulation task can be set to make the robot's end effector move along a specific trajectory. The simulation terminal can calculate the theoretical position, velocity, and other information of each joint of the robot at different time points based on the trajectory and the dynamic model; this information constitutes the motion data. Alternatively, the simulation task can simulate the robot performing a grasping action. The simulation terminal then generates the motion sequence of each joint of the robot as motion data based on the definition of the grasping action. Subsequently, the simulation terminal converts the motion data into virtual sensor frames according to a preset mapping relationship and sends them to the control terminal through the sensor output channel. For example, the preset mapping relationship can map the joint angle data generated by the simulation terminal to the encoder reading format desired by the control program, and map the end effector position data generated by the simulation terminal to the vision sensor output desired by the control program.

[0023] The control unit is used to construct actuator commands based on control commands and virtual sensor frames related to the simulation task, and then send them to the simulation unit through the control input channel. For example, after receiving a virtual sensor frame (such as a virtual encoder reading), the control program combines it with preset control commands (such as target joint angles) to calculate the torque or speed commands required for each joint motor; these commands are the actuator commands. Alternatively, the control program can also generate actuator commands for adjusting the robot's posture based on virtual sensor frames (such as virtual force sensor readings) and control commands (such as target contact force).

[0024] The simulation terminal is used to update the state of the dynamic model based on actuator commands. For example, after receiving actuator commands (such as joint torque commands) from the control terminal, the simulation terminal applies them to the corresponding joints in the dynamic model and calculates the new state of the model in the next time step, such as joint position, velocity, and acceleration, according to the dynamic equations. As another implementation, the actuator commands can also be joint velocity commands, and the simulation terminal updates the motion state of the model based on these velocity commands and the dynamic model.

[0025] The control unit, based on the updated state, determines that the control program meets the simulation conditions for the simulation task. It then switches the virtual hardware interface to the physical hardware interface between the control unit and the target robot, allowing the control program to drive the target robot. For example, simulation conditions can be set such that the robot's end effector successfully reaches the target position in the simulation environment with errors within acceptable limits. When the control unit detects that these conditions are met, it triggers the interface switch. Alternatively, simulation conditions can also be that the robot successfully completes a series of complex actions without collisions or anomalies throughout the simulation. Once these conditions are met, the control unit executes the interface switch operation.

[0026] For example, suppose there is a robot control system whose goal is to enable a multi-jointed robotic arm (the target robot) to accurately grasp an object placed in a specific location. Traditional methods may require developing a simplified controller in a simulation environment and then redeveloping or adapting a real controller onto the physical robotic arm, leading to significant repetitive work and potential compatibility issues. To address this, this application proposes a control system. First, in simulation mode, the control unit is configured with a control program for the grasping task, which includes motion planning, trajectory tracking, and grasping strategies for the robotic arm. The simulation unit is configured with the robotic arm's dynamics model and a dynamic model that accurately simulates the robotic arm's physical behavior. The control unit and the simulation unit are connected through a virtual hardware interface, which includes control input channels and sensor output channels.

[0027] When a simulation task is initiated, for example, a robotic arm is programmed to move from an initial position over an object and perform a grasping action. The simulation terminal determines the theoretical motion data for each joint of the robotic arm based on this task, such as joint angles, angular velocities, and end effector positions. Subsequently, the simulation terminal converts this motion data into virtual sensor frames according to a preset mapping relationship. For example, joint angles are converted into virtual encoder readings, and end effector positions are converted into virtual vision sensor data, which are then sent to the control terminal via sensor output channels.

[0028] After receiving these virtual sensor frames, the control unit's internal control program constructs actuator commands to drive the robotic arm's joints, such as torque commands for each joint motor, based on preset grasping control commands (e.g., target grasping point coordinates, grasping force magnitude) and the received virtual sensor frames. These actuator commands are then sent back to the simulation unit via the control input channel.

[0029] After receiving actuator commands, the simulation end applies them to its internal dynamics model and updates the model's state based on these commands. For example, based on torque commands and the robot arm's inertial parameters, it calculates the robot arm's new joint positions, velocities, and attitudes for the next time step. This process forms a closed loop, enabling the control program to operate in the simulation environment as if controlling a real robot arm. Its output actuator commands directly affect the behavior of the simulation model, while the feedback from the simulation model serves as the input to the control program.

[0030] During simulation, the control unit continuously monitors the updated status to determine if the control program meets the simulation conditions for the task. For example, if the robotic arm's end effector successfully reaches the target object in the simulation, and the trajectory error of the grasping action is within an acceptable range, the simulation conditions are considered met. Once the simulation conditions are met, the control unit seamlessly switches the virtual hardware interface to the physical hardware interface with the real robotic arm. At this point, the same control program that ran in the simulation environment can directly drive the real robotic arm through the physical hardware interface without any modification, enabling it to perform the actual grasping task.

[0031] Based on the above examples, the technical concept of this application achieves seamless migration of the control program between the simulation environment and the physical environment by constructing a unified virtual hardware interface. Traditional solutions often require the control algorithm to reconnect to the physical SDK, ROS / DDS topics, device naming, and low-level communication endpoints after simulation verification. This not only increases development workload but may also expose differences in the operating environment not covered in the simulation on the physical robot. This application ensures that the data format of the simulation output is completely consistent with the sensor input expected by the control program by determining the motion data according to the simulation task at the simulation end and converting it into virtual sensor frames using a preset mapping relationship. This allows the control program to receive input signals isomorphic to the physical environment during the simulation phase, thereby guaranteeing the high realism of the verification results of the control algorithm in the simulation.

[0032] Furthermore, the control end processes virtual sensor frames and constructs actuator commands using a control program, feeding these commands back to the dynamic model on the simulation end, thus achieving closed-loop simulation involving the real control state machine. This differs from some simulation schemes that only simulate the results returned by upper-level APIs; this application ensures the integrity and realism of the control logic. By switching the virtual hardware interface to a physical hardware interface after the simulation conditions are met, and directly driving the target robot using the same control program, this application eliminates the environmental differences between simulation and the actual machine, achieving a smooth migration of the control logic. This design not only reduces the adaptation workload for developers during the simulation-to-actual-machine switching process but also effectively avoids potential operational risks caused by environmental differences by maintaining the consistency of the control program, ensuring the integrity and reliability of the control strategy from virtual to reality.

[0033] In summary, by establishing a closed-loop communication mechanism between the control end and the simulation end through a virtual hardware interface, lossless conversion of motion data and virtual sensor frames is achieved using a preset mapping relationship, and seamless switching to the physical hardware interface is possible when simulation conditions are met. This allows the control program to drive the target robot without modification, solving the problem of inconsistency between the simulation environment and the physical robot control program, avoiding repeated adaptation work, shortening the development cycle, and reducing the risk of operational failures.

[0034] In practical applications, inconsistencies often exist between the dynamic model in the simulation environment and the actual robot control program in terms of data structure, joint sequence, and sensor definitions. This can prevent the control program from running directly in the simulation environment or cause the simulation data to be incorrectly parsed by the control program, thus limiting the reliability of seamless switching between simulation and actual operation. To address this, in this embodiment, the simulation terminal is further configured to load a dynamic model containing robot joints, actuators, and sensors; obtain the joint identifiers and joint sequences associated with the robot joints, the actuator index information and inertial measurement unit associated with the actuators, and the sensor information associated with the sensors; obtain the hardware interface description information corresponding to the control program; determine the control program joints, control program sensors, and control program output channels based on the hardware interface description information; establish a first mapping relationship between the control program joints and the joint identifiers and joint sequences, a second mapping relationship between the control program sensors and the sensor information, and a third mapping relationship between the control program output channels and the actuator index information and the inertial measurement unit; and use the first, second, and third mapping relationships as preset mapping relationships.

[0035] Specifically, loading a dynamic model containing robot joints, actuators, and sensors refers to importing a digital model describing the robot's physical structure, kinematics, dynamic characteristics, and sensor configuration into the simulation environment. This dynamic model can be in various standard formats, such as the Unified Robot Description Format (URDF), Simulation Description Format (SDF), or the MJCF format of the MuJoCo multibody physics engine, or a custom XML or JSON format. The loading process can involve reading the model file from locally stored files using a file parser, or retrieving it from a model library via a network interface.

[0036] Accordingly, the joint identifiers and joint sequences associated with the robot joints in the dynamic model, the actuator index information and inertial measurement units (IMUs) associated with the actuators, and the sensor information associated with the sensors are obtained. This aims to extract key physical component information defined internally from the loaded dynamic model. Joint identifiers can be unique names or IDs of the joints, while the joint sequence indicates the topological relationship or data arrangement order of the joints in the robot's kinematic chain. Actuator index information is used to uniquely identify each actuator in the model, such as a motor or hydraulic cylinder, while the inertial measurement unit (IMU) indicates the sensor in the model used to measure attitude, angular velocity, and linear acceleration. Sensor information covers other sensors besides the IMU, such as force sensors, contact sensors, and vision sensors, including their type, location, measurement range, and data format. This information can be obtained by parsing the model file structure or by calling the API interface provided by the simulation physics engine.

[0037] Accordingly, obtaining the hardware interface description information corresponding to the control program refers to obtaining the standardized information such as the data format, naming conventions, and communication protocols expected by the control program when interacting with external hardware. This description information can be a separate configuration file, such as a YAML, JSON, or XML file, which defines in detail the joint names, sensor data types, actuator command formats, etc., expected by the control program. Furthermore, this information can also be queried through the API interface provided by the control program, or its hardware interface requirements can be inferred through static analysis or dynamic debugging of the control program.

[0038] Accordingly, determining the control program joints, control program sensors, and control program output channels based on the hardware interface description information means identifying the logical entities defined within the control program to represent robot joints, sensor inputs, and actuator outputs, based on the acquired hardware interface description information. A control program joint is an abstract concept used by the control program internally to represent each joint of the robot; it may have specific naming rules or indexes. Control program sensors refer to the feedback data from sensors that the control program expects to receive, such as joint position, velocity, torque, and IMU data. Control program output channels refer to the logical paths or data structures through which the control program sends commands to the actuators, such as motor current, position commands, or velocity commands.

[0039] Accordingly, a first mapping relationship is established between the control program joints and the joint identifiers and joint sequences; a second mapping relationship is established between the control program sensors and the sensor information; and a third mapping relationship is established between the control program output channels and the actuator index information and the inertial measurement unit (IMU). This aims to solve the incompatibility problems between the simulation model and the control program caused by differences in naming, order, data format, etc. The first mapping relationship associates the joints defined internally by the control program with the actual joint identifiers in the simulation dynamics model and their order in the kinematic chain, ensuring that the joint commands issued by the control program can accurately act on the corresponding joints in the simulation model. The second mapping relationship associates the sensor inputs expected by the control program with the sensor data generated by the simulation dynamics model, enabling the sensor feedback generated by the simulation environment to be correctly parsed and utilized by the control program. The third mapping relationship associates the actuator commands issued by the control program with the actuator indexes and inertial measurement units (IMUs) in the simulation dynamics model, ensuring that the control program can effectively drive the actuators in the simulation model and receive key feedback such as IMUs. These mapping relationships may include name mapping, index mapping, data type conversion, unit conversion, coordinate system conversion, and data order adjustment, etc.

[0040] Correspondingly, Using the first, second, and third mapping relationships as preset mapping relationships means solidifying and storing the various mapping rules established above for reuse during simulation. These preset mapping relationships can be stored in configuration files, such as JSON, YAML, or XML files, or they can be loaded into memory as data structures when the simulation system starts. By presetting these mapping relationships, reconfiguration can be avoided at the start of each simulation task, improving system efficiency and consistency.

[0041] Based on this, a multi-dimensional mapping mechanism was constructed to solve the problem of mismatch between the underlying data definitions of the simulation model and the control program, achieving deep alignment between the simulation environment and the actual machine control logic. By loading the dynamic model and extracting joint identifiers, joint sequences, actuator indices, and sensor information, the physical property characteristics of the simulation side can be accurately obtained, providing basic data support for subsequent data conversion. By obtaining the hardware interface description information corresponding to the control program, the specific format requirements of the input and output data of the control program can be clarified, thereby determining the joint, sensor, and output channel definitions on the control program side. On this basis, a first mapping relationship is established to associate the joints of the control program with the joint identifiers and sequences of the simulation model, ensuring that motion control commands can be accurately applied to the corresponding physical joints and avoiding control failures caused by misaligned joint sequences. At the same time, a second mapping relationship is established to associate the sensors of the control program with the sensor information of the simulation model, enabling the sensor data generated by the simulation to be correctly identified and parsed by the control program, ensuring that the state feedback obtained by the control algorithm is real and effective. Furthermore, a third mapping relationship is established, associating the control program output channel with the actuator index and inertial measurement unit. This achieves precise mapping from control commands to the dynamic model actuators, ensuring the accuracy of the dynamic closed-loop control. This mapping relationship, as a preset configuration, allows the simulation end to determine motion data based on the simulation task corresponding to the control program, convert the motion data into virtual sensor frames according to these preset mapping relationships, and send them to the control end through the sensor output channel. The control end then constructs actuator commands based on the control commands of the associated simulation task and the virtual sensor frames, and sends them to the simulation end through the control input channel. The simulation end then updates the state of the dynamic model based on the actuator commands. This mechanism allows the control program to operate in simulation mode without needing to perceive whether the underlying hardware interface is virtual or physical. Its interaction with the simulation end is consistent with its interaction with the real robot, thus achieving seamless operation of the control program in the simulation environment and laying the foundation for subsequent switching to physical hardware interfaces.

[0042] For example, the simulation client can load a robot dynamics model in the form of a URDF (Unified Robot Description Format) file. This URDF file details the robot's joints (e.g., joint_shoulder_pan, joint_elbow_flex), actuators (e.g., motor_shoulder, motor_elbow), and sensors (e.g., imu_base, force_sensor_wrist). The simulation client parses this URDF file to obtain the unique identifier of each joint and its order in the robot's kinematic chain, as well as the index of each actuator and detailed parameters of the IMU sensors. Simultaneously, the control program may include a YAML configuration file defining the hardware interfaces it expects. For example, this configuration file might specify that the control program sends joint position commands through a channel named robot_joints_cmd and expects to receive a data packet named robot_state_feedback, containing fields such as joint_angles, joint_velocities, and imu_data. Based on this YAML file, the simulation end determines the joints (e.g., control_joint_0, control_joint_1), sensors (e.g., control_imu, control_force), and output channels (e.g., control_motor_output) defined within the control program. Subsequently, the simulation end establishes mapping relationships: for example, mapping control_joint_0 to joint_shoulder_pan in the URDF model, mapping control_joint_1 to joint_elbow_flex, and specifying their data order; mapping control_imu to imu_base in the URDF model, and defining data format conversion rules; mapping control_motor_output to the actuator indices of motor_shoulder and motor_elbow in the URDF model. These mapping relationships are stored in a JSON file as a preset configuration. During simulation runtime, the simulation end converts the motion data generated by the simulation model (such as joint angles and IMU readings) into the virtual sensor frame format expected by the control program according to these preset mapping relationships, and sends it to the control end through the virtual hardware interface. Conversely, the actuator commands issued by the control terminal will also be correctly parsed and applied to the simulation model through these mapping relationships.

[0043] In summary, the above-described approach effectively resolves the inconsistencies in data structure, joint sequence, and sensor definitions between the dynamics model in the simulation environment and the actual robot control program. By establishing a multi-dimensional mapping relationship, the control program can run directly in the simulation environment without modification, significantly reducing the workload required for developers to switch between simulation and real-world operation. This not only improves the realism and reliability of simulation verification but also accelerates the development and iteration process of robot control algorithms, avoiding potential errors caused by data mismatches, thereby enhancing the development efficiency and robustness of the entire robot system.

[0044] Existing robot control programs typically rely on physical joints, inertial measurement units (IMUs), force sensors, motor drives, and robot middleware. Traditional simulation systems often implement simplified controllers internally within the simulator or design dedicated input / output interfaces for the simulation environment. The simulation controller and the real robot control program may differ in data structures, joint sequences, time sources, state machines, runtime libraries, and upper-level interfaces. Therefore, after the control algorithm is verified in simulation, it still needs to be re-integrated with the actual robot SDK, ROS / DDS topics, device naming, and low-level communication endpoints. This re-adaptation not only increases development work but may also expose differences in the operating environment not covered in the simulation on the actual robot. Some simulation schemes only simulate the return results of the upper-level API without actually running the same control program as the actual robot; other schemes can start the control program, but the sensor data and actuator commands lack a unified mapping, failing to form a complete dynamic closed loop involving the real control state machine. In some of the solutions described above in this application, a virtual hardware interface is used to connect the control terminal and the simulation terminal to achieve simulation control. However, in practical applications, if the control terminal directly runs the control program, it often faces the problem of how to ensure that the control program can seamlessly integrate with the simulation environment and maintain consistency with the physical robot's operating environment. If the control program cannot correctly load the runtime library, configuration information, and middleware of the same origin as the target robot, or if the communication configuration parameters are set improperly, the control program will be unable to recognize the virtual hardware interface, thus failing to achieve an effective closed loop between the simulation environment and the real control logic. To address this, in this embodiment, the control terminal is also used to start a control program of the same origin as the target robot and load the runtime library, configuration information, and robot middleware associated with the target robot; and to connect the control program with the virtual hardware interface through preset communication configuration parameters.

[0045] Specifically, "starting a control program of the same origin as the target robot" means running a software instance on the control end that is exactly the same as the control program used on the actual target robot. This ensures that the control logic and algorithm implementation used in the simulation are consistent with the physical robot, avoiding verification failures caused by differences between the simulation environment and the actual machine environment. This startup process can be achieved by directly executing the binary file or script of the control program through the operating system, or by starting the control program instance through containerization technology (such as Docker) or virtual machines (VMs) to provide an isolated and consistent runtime environment. "Loading the runtime library associated with the target robot" means introducing the low-level functions, APIs, and dependencies required for the operation of the control program into the execution environment of the control program. These runtime libraries typically contain encapsulations of functions such as robot hardware interfaces, kinematic / dynamic calculations, and signal processing. This can be achieved by automatically loading the library when the control program starts through the operating system's dynamic linking mechanism (such as .so files under Linux and .dll files under Windows), or by configuring environment variables (such as LD_LIBRARY_PATH) or explicitly specifying the library path in the control program code. "Loading the configuration information associated with the target robot" means providing the control program with the parameters, settings, and initialization data required for its operation. This configuration information may include robot joint limits, PID controller parameters, safety zone definitions, communication protocol parameters, etc. This can be achieved by reading configuration files in a specific format (such as XML, JSON, YAML), retrieving configuration parameters from a database or key-value store, or passing configuration information at startup via command-line parameters or environment variables. "Loading the robot middleware associated with the target robot" refers to integrating and starting the software layer that provides advanced services to the robot system (such as inter-module communication, data management, task scheduling, state machine management, etc.). Common robot middleware includes Robot Operating System (ROS) and Data Distribution Service (DDS). This can be achieved by starting specific services of the robot middleware (such as ROS Master, DDS domain), or by loading SDKs or APIs provided by specific robot manufacturers, enabling them to work collaboratively with the control program. "Connecting the control program to the virtual hardware interface through preset communication configuration parameters" refers to modifying the communication settings of the control program, redirecting its original communication link to the physical hardware to the virtual hardware interface. This allows the control program to transparently interact with the simulation environment without changing its core logic. This can be achieved by modifying the configuration file inside the control program to point the IP address, port number, or device path of the hardware interface to the address of the virtual hardware interface, or by overriding the default hardware interface configuration through command-line parameters or environment variables when the control program starts, so that it points to the virtual hardware interface.

[0046] Based on this, when the simulation mode is started, the control terminal first launches a control program that originates from the target robot. This startup process is accompanied by loading the runtime libraries, configuration information, and robot middleware required for the actual operation of the target robot. This ensures that the control program has a software environment and context completely consistent with the physical robot's runtime environment in the simulation environment. Subsequently, through preset communication configuration parameters, the control program is guided to establish a connection with the virtual hardware interface. This means that the interface calls originally used by the control program to communicate with the physical robot hardware are now redirected to the virtual hardware interface. The virtual hardware interface, acting as a bridge between the control program and the simulation terminal, receives virtual sensor frames sent by the simulation terminal through the sensor output channel and passes them to the control program. Based on these virtual sensor frames and its internal control logic, the control program generates actuator commands and sends these commands back to the simulation terminal through the control input channel. After receiving the actuator commands, the simulation terminal updates the state of its dynamic model, thus forming a complete, high-fidelity control closed loop. This mechanism allows the control program to run in the same way as the physical robot during the simulation phase, thereby achieving seamless integration and verification between the simulation environment and the real control logic without modifying the core logic of the control program.

[0047] For example, the control terminal could be an industrial PC running a Linux operating system. When simulation is required, this industrial PC executes a startup script to launch a control program derived from the target robot. For instance, this script might contain the commands `roslaunch robot_control_pkg robot_main.launch` to launch the ROS-based robot control program. During startup, the control program automatically loads its required dynamic link libraries, such as `librobot_kinematics.so` (for kinematics calculations) and `libmotor_driver_interface.so` (for the motor drive interface). Simultaneously, the control program reads configuration files located at ` / opt / robot / config / robot_params.yaml`, which contain the robot's physical parameters, controller gains, etc. Furthermore, ROS middleware (such as the ROS Master and associated ROS nodes) is also started to provide inter-node communication services. To enable the control program to connect to the virtual hardware interface, environment variables can be set in the startup script, such as `export ROBOT_HW_INTERFACE_TYPE=VIRTUAL` and `export VIRTUAL_HW_PATH= / dev / shm / robot_virtual_interface`. The hardware abstraction layer inside the control program uses these environment variables to redirect communication originally directed to the actual hardware device (such as ` / dev / ttyUSB0`) to the specified shared memory region ` / dev / shm / robot_virtual_interface`, which is the implementation of the virtual hardware interface. The emulation end is responsible for writing virtual sensor frames to this shared memory region and reading actuator commands from it.

[0048] In summary, by initiating a control program of the same origin as the target robot on the control end through the above processing, and loading its associated runtime library, configuration information, and robot middleware, the control logic in the simulation environment is ensured to be completely consistent with that of the physical robot, avoiding verification failures caused by differences between the simulation and physical environments. Simultaneously, by using preset communication configuration parameters to connect the control program with the virtual hardware interface, the control program can transparently interact with the simulation end without modification, solving the problem of the control program's inability to recognize the virtual hardware interface. Combining the mechanism of connecting the control end and the simulation end through the virtual hardware interface to achieve simulation mode connection, this solution enables the control program to run in the same way as the physical robot during the simulation phase, thereby achieving an effective closed loop between the simulation environment and the real control logic. This greatly improves the accuracy and reliability of simulation verification and lays the foundation for a smooth transition to the physical robot. In practical applications, the raw motion data generated by the simulation terminal is often in diverse formats and lacks standardization, making it difficult for the control terminal to directly identify or parse this data. This hinders the construction of virtual sensor frames that meet the requirements of the control program protocol, limiting the effective data loop between the simulation environment and the real control program. To address this, in this embodiment, the simulation terminal is further configured to determine the simulation task corresponding to the control program, and based on the simulation task, acquire joint position information, joint velocity information, body posture information, angular velocity information, linear acceleration information, contact state information, and / or force information as the motion data; map the motion data according to a preset mapping relationship to obtain an initial virtual sensor frame; encapsulate the initial virtual sensor frame according to the protocol structure corresponding to the control program to obtain a virtual sensor frame, and send it to the control terminal through the sensor output channel.

[0049] Specifically, the simulation end first needs to determine the simulation task corresponding to the control program. This task can be determined in several ways. For example, the simulation end can parse the control program's configuration file or startup parameters to automatically identify its expected simulation target, such as whether the control program is used to perform a specific trajectory tracking task or force control operations. Alternatively, the operator can manually input or select the simulation task type and parameters corresponding to the current control program through a user interface or API interface, for example, specifying the robot to perform tasks such as grasping, transporting, or inspection.

[0050] Accordingly, after determining the simulation task, the simulation client acquires joint position information, joint velocity information, body posture information, angular velocity information, linear acceleration information, contact state information, and / or force information based on the simulation task, as motion data. This motion data is the original representation of the robot's current physical state. Specifically, the simulation client can use its built-in physics engine or dynamics model to calculate and extract in real time the current position and velocity of each joint of the simulated robot, as well as the robot's posture (such as Euler angles or quaternions), angular velocity, linear acceleration, and other information. Simultaneously, it can detect the robot's contact with the environment and calculate the contact force. Furthermore, the simulation client can also subscribe to data streams published by other modules (such as the environment perception module and collision detection module) in the simulation environment to obtain the kinematic and dynamic states of various parts of the robot, including but not limited to joint encoder readings, IMU (Inertial Measurement Unit) data, and force sensor readings.

[0051] Subsequently, the simulation end performs mapping processing on the above motion data according to the preset mapping relationship to obtain the initial virtual sensor frame. The preset mapping relationship can be a configuration file or a data table, which defines the correspondence between simulation data fields (such as sim_joint_pos_x) and sensor frame fields expected by the control program (such as robot_sensor_frame.joint_angle[0]), and includes rules such as unit conversion, coordinate system conversion, and data type conversion. The mapping processing can involve data reordering, unit conversion (such as radians to angles, meters to millimeters), direction conversion (such as coordinate axis reversal), timestamp generation, and data format encapsulation (such as combining multiple independent data points into a structure or array). For example, the joint angle (radians) output by the simulator is converted into the joint angle (degrees) expected by the control program, and the current simulation timestamp is added.

[0052] Based on this, the simulation end encapsulates the initial virtual sensor frame according to the protocol structure corresponding to the control program, thereby obtaining the final virtual sensor frame. This protocol structure can be a specific message format, such as the sensor_msgs / JointState or sensor_msgs / Imu message types in ROS (Robot Operating System), or a specific topic data structure in DDS (Data Distribution Service). The encapsulation process includes adding message headers, checksums, and serializing data to ensure that the data fully conforms to the communication specifications of the control program during transmission.

[0053] Finally, the simulation end sends the acquired virtual sensor frames to the control end through the sensor output channel. The sensor output channel can be a shared memory area, where the simulation end writes the virtual sensor frames and the control end reads them. Alternatively, the sensor output channel can be a network socket (such as TCP / IP or UDP), through which the simulation end sends data packets and the control end listens for data. Furthermore, the sensor output channel can also be an inter-process communication (IPC) mechanism, such as a message queue, pipes, or named pipes, used to transfer data between different processes on the same machine.

[0054] Based on this, through the aforementioned series of refined data processing steps, seamless data integration between the simulation environment and the control program was achieved. First, the simulation end identified the simulation task of the control program, clarifying the robot behavior to be simulated and the required data types, providing clear guidance for subsequent data acquisition. Next, the simulation end comprehensively acquired the physical state parameters required for robot operation from its dynamic model, including joint position information, joint velocity information, body posture information, angular velocity information, linear acceleration information, contact state information, and / or force information, among other key motion data. This comprehensive data coverage of the physical state parameters required for robot operation provides a detailed input foundation for subsequent control decisions.

[0055] Subsequently, the simulation end uses a preset mapping relationship to process these raw motion data, transforming the simulated physical quantities into initial virtual sensor frames that the control program can understand. This step effectively bridges the semantic gap between the simulation physical model and the control program's data definitions, ensuring the correctness of the data content. Furthermore, by encapsulating the initial virtual sensor frames according to the control program's specific protocol structure, it ensures that the data fully conforms to the control program's communication specifications during transmission, enabling the control end to directly read and process this data.

[0056] The entire process forms a complete data processing chain, from raw simulation data to protocol-compatible virtual sensor frames, and finally sent to the control end through the sensor output channel. This mechanism allows the control program to receive data consistent with the feedback from the physical robot's sensors during the simulation phase, enabling it to run a control program derived from the real robot in the virtual environment and form a complete dynamic closed loop involving the real control state machine. This is closely integrated with the step in the basic scheme where the simulation end converts motion data into virtual sensor frames and sends them to the control end. Through refined acquisition, mapping processing, and protocol encapsulation of motion data, the realism and usability of the virtual sensor frames are greatly improved, allowing the control program to be verified in the simulation environment without modification.

[0057] For example, suppose the target robot is a six-axis robotic arm whose control program expects to receive a sensor data packet containing six joint angles (in degrees), six joint angular velocities (in degrees / second), and three-axis IMU data (containing linear acceleration and angular velocity, in g and degrees / second), which uses a specific binary protocol structure.

[0058] As a specific implementation method, the simulation terminal first determines the current simulation task as trajectory tracking of the robotic arm based on the configuration file of the control program. Based on this task, the simulation terminal obtains the real-time position (radians) and velocity (radians / second) of the six joints of the robotic arm, as well as the attitude, angular velocity, and linear acceleration of the robotic arm's end effector, from its internal physics engine. This raw data constitutes the motion data.

[0059] Next, the simulation terminal will consult a pre-defined mapping table. This mapping table defines: 1. The joint positions (in radians) output by the physics engine need to be multiplied by 180 / π to convert them to degrees and arranged in a specific order.

[0060] 2. The joint velocities (radians / second) output by the physics engine need to be multiplied by 180 / π to convert them to degrees / second and arranged in a specific order.

[0061] 3. The linear acceleration and angular velocity output by the physics engine need to undergo coordinate system transformation (e.g., from the global coordinate system of the simulation world to the robot's body coordinate system) and unit transformation (e.g., from m / s² to g, from rad / s to deg / s).

[0062] 4. All data needs to be timestamped with the current simulation timestamp.

[0063] Through these mapping processes, the simulation end converts the raw motion data into an initial virtual sensor frame, which contains joint angles, joint angular velocities in degrees, and converted IMU data.

[0064] Subsequently, the simulation end encapsulates this initial virtual sensor frame according to the binary protocol structure expected by the control program. This may include adding a synchronization word, a data length field, and a timestamp field to the packet header, then filling the packet with joint data and IMU data sequentially according to a predefined byte order and data type (e.g., floating-point numbers converted to IEEE 754 standard format), and adding a CRC checksum to the packet tail.

[0065] Finally, this encapsulated virtual sensor frame is sent to the control unit via a shared memory area. The control unit reads data from this shared memory area and parses it into sensor inputs that the control program can directly use, thereby driving the control program to perform subsequent trajectory calculations and actuator command generation.

[0066] In summary, the above processing effectively solves the problem that the diverse formats of raw motion data generated by the simulation end, lacking standardization, prevent the control end from directly recognizing or parsing it. By performing refined acquisition, mapping, and protocol encapsulation of motion data, this application ensures that the simulation end can generate virtual sensor frames that are fully compatible with the real hardware interface. This allows the control program to receive data consistent with the feedback from the physical robot's sensors in the simulation environment, enabling verification in the simulation environment without modification, greatly improving the realism and reliability of simulation verification. In particular, combined with the pre-defined mapping relationship in the above scheme, this application can effectively bridge the semantic gap between the simulation physical model and the control program data definition, and further ensures that the data fully complies with the communication specifications of the control program during transmission through protocol encapsulation. This achieves an efficient and accurate data closed loop between the simulation environment and the real control program, significantly reducing the workload and potential risks of adapting the control algorithm from simulation to actual deployment.

[0067] During simulation, effectively fusing the virtual sensor data received by the control terminal with the control commands corresponding to the simulation task, and transforming them into actuator commands capable of driving the dynamic model update, thereby achieving closed-loop interaction between the real control logic and the simulation dynamic environment, remains a technical problem to be solved. In this embodiment, the control terminal is further configured to receive the virtual sensor frame and determine the control command corresponding to the simulation task; process the virtual sensor frame and the control command through the control program to obtain actuator commands, and send them to the simulation terminal through the control input channel; the simulation terminal is further configured to convert the actuator commands into model control information, write the model control information into the actuator corresponding to the dynamic model, and update the state of the dynamic model through the actuator based on the model control information.

[0068] Specifically, the control terminal receives the virtual sensor frames, which are data packets generated by the simulation terminal based on the dynamic model state, simulating the output of real robot sensors. These virtual sensor frames contain measurement values ​​from virtual sensors such as robot joints, inertial measurement units, and force sensors in the simulation environment. The control terminal can receive virtual sensor frames by reading a shared memory area written by the simulation terminal; alternatively, the control terminal can receive virtual sensor frame data streams sent from the simulation terminal via network sockets (such as TCP / IP or UDP). The control terminal is also used to determine the control commands corresponding to the simulation task. These control commands are instructions issued by the upper-layer application or user to complete a specific simulation task (such as moving to a certain location or executing a sequence of actions). The control terminal can read the control commands for the current simulation task from a preset configuration file or task planning module, or dynamically determine the control commands by receiving input from the user interface or the upper-layer task scheduling system.

[0069] Accordingly, the control program is software logic designed to drive the target robot, encompassing functions such as robot kinematics and dynamics control algorithms, state machine management, and path planning. In simulation mode, this program interacts with the virtual hardware interface as if interacting with real hardware. The control program processes the virtual sensor frames and the control commands; this processing is the core function of the control program. It combines the environmental perception information provided by the virtual sensors with the control intent issued from the upper layer, and calculates the appropriate actions of the robot actuators through internal control algorithms (such as PID control, torque control, path tracking algorithms, etc.). For example, the feedback controller inside the control program compares the current state (such as joint position and velocity) provided by the virtual sensor frames with the target state set by the control commands, calculates the error, and generates the corresponding control output; or, the state estimation module in the control program uses the virtual sensor frames to perform state fusion to obtain a more accurate robot state estimate, and then inputs this estimated state into the motion planning or behavior decision module, combining it with the control commands to generate the next action instruction. After processing, actuator commands are obtained. These actuator commands are instructions calculated by the control program to drive robot actuators (such as motors and hydraulic cylinders), typically expressed as torque, speed, position, or current. The control terminal sends the actuator commands to the simulation terminal through the control input channel, which is a virtual communication path for transmitting actuator commands from the control terminal to the simulation terminal. The control terminal can write the actuator commands into a pre-allocated shared memory area, from which the simulation terminal reads them; alternatively, the control terminal can encapsulate the actuator commands into data packets and send them to the simulation terminal via network sockets (such as UDP or TCP).

[0070] Accordingly, upon receiving the actuator command, the simulation terminal converts it into model control information. The actuator command is an abstract instruction at the control program level, while the dynamics model requires lower-level control quantities compatible with the physics engine. This conversion process maps the abstract command to parameters that the physics engine can understand and apply. For example, the simulation terminal can directly convert the actuator command (such as joint torque) into the control input (such as the applied torque value) of the corresponding actuator (such as a joint motor) in the dynamics model according to a preset mapping table or conversion function; or, if the actuator command is a higher-level instruction (such as a target position), the simulation terminal may need an internal lower-level controller (such as a virtual PID controller) to convert it into the torque or speed command required by the model. Subsequently, the simulation terminal writes the model control information into the actuator corresponding to the dynamics model. This step actually applies the converted physical control quantity to the virtual actuator in the simulation environment, thereby affecting the behavior of the dynamics model. For example, the simulation engine can directly set model control information (such as torque values) to the actuators of specific joints in the dynamic model by calling the API interface provided by the dynamics simulation engine; alternatively, the simulation engine can maintain a virtual actuator state table, update the model control information in this table, and the dynamics engine reads this table to drive the model in each simulation step. Finally, the actuators update the state of the dynamic model according to the model control information. This is the core function of the dynamics simulation engine, which calculates the state of the robot model in the next time step (such as joint position, velocity, acceleration, attitude, etc.) based on the control information applied to the actuators and other physical laws (such as gravity, collision). For example, within each simulation time step, the dynamics simulation engine calculates the new position, velocity, and attitude of each joint and the robot body through numerical integration based on the torque or velocity commands applied to the virtual actuators, combined with the robot's inertial parameters, friction, etc.

[0071] Based on this, a closed-loop data processing mechanism between the control end and the simulation end is constructed, achieving deep coupling between the control logic and the physical simulation environment. In simulation mode, the simulation end first determines the motion data according to the simulation task corresponding to the control program, and converts it into virtual sensor frames according to a preset mapping relationship, sending them to the control end through the sensor output channel. After receiving these virtual sensor frames, the control end combines them with the control commands corresponding to the simulation task issued by the upper-layer application or the user, and performs comprehensive processing using its internal control program. This process ensures that the control commands are generated based on real-time feedback from the current simulation environment, thereby guaranteeing the effectiveness and adaptability of the control logic. Specifically, the control program compares the current robot state provided by the virtual sensor frames with the target state set by the control commands, uses its built-in control algorithm to calculate the instructions required to drive the robot actuators, i.e., actuator commands, and sends them back to the simulation end through the control input channel. After receiving the actuator commands, the simulation end converts them into model control information compatible with the dynamic model and writes them into the corresponding virtual actuator in the dynamic model. The dynamics simulation engine updates the overall state of the dynamics model by performing numerical integration through a physics solver based on the model control information applied to the actuators, combined with the physical parameters of the robot model and environmental factors. The updated dynamics model state then becomes the basis for generating virtual sensor frames in the next simulation cycle, thus forming a complete closed-loop control system with the participation of the real control program. This mechanism allows the control program to operate the simulation model as if it were operating a real robot, completing a full closed loop from sensor data input to actuator action output through actuator state updates. Building upon this, this scheme, combined with the virtual hardware interface mechanism proposed in the aforementioned control system, further enhances the consistency between simulation and actual operation. Through the virtual hardware interface, the control program does not need to distinguish between interacting with virtual and real sensors, nor does it need to distinguish between sending actuator commands to virtual and real actuators. This seamless switching capability benefits from the precise processing of virtual sensor frames and control commands by the control end in this scheme, as well as the accurate conversion of actuator commands to model control information and the real-time updating of the dynamics model by the simulation end. It ensures that the control logic running in the simulation environment can fully participate in the dynamic closed loop, thereby fully verifying the performance and robustness of the control program during the simulation phase.

[0072] For example, the control end can be an embedded controller or an industrial PC running a control program based on a real-time operating system (such as RT-Linux or VxWorks). This control program can use a Robot Operating System (ROS) or a Data Distribution Service (DDS) as its communication framework. When the simulation end (e.g., a server running a physics simulation engine such as Gazebo or MuJoCo) generates virtual sensor frames, these frames can contain the current angles and angular velocities of each robot joint, linear acceleration and angular velocities provided by the inertial measurement unit (IMU), and contact force information provided by the force sensors. This data is encapsulated into specific message formats (e.g., ROS's sensor_msgs / JointState or sensor_msgs / Imu messages) and sent to the control end through a virtual communication channel (e.g., a shared-memory-based ROS topic publishing mechanism or a DDS data stream). Upon receiving these virtual sensor frames, the control end's internal control program parses this data. Simultaneously, the control program receives control commands corresponding to the current simulation task from the upper-level task manager, such as a target joint position sequence or an end effector trajectory. The motion controller in the control program (e.g., an algorithm based on model predictive control (MPC) or inverse dynamics control) compares the current state provided by the received virtual sensor frames with the target state set by the control commands to calculate the torque or velocity command required for each joint. These calculated torque or velocity commands are the actuator commands. These actuator commands are then encapsulated into another message format (e.g., ROS's std_msgs / Float64MultiArray or control_msgs / JointCommand messages) and sent back to the simulation end via another virtual communication channel (e.g., ROS topic subscription mechanism with shared memory or DDS data stream). Upon receiving the actuator commands, the simulation end's internal virtual hardware interface module parses these commands. For example, if the actuator command is a joint torque, the simulation end directly uses these torque values ​​as model control information. If the actuator command is a joint target position or velocity, the simulation end may internally run a simplified virtual joint controller (such as a PID controller) to convert these high-level commands into the joint torque required by the physics engine. This model control information is then written into the corresponding virtual actuator in the dynamics model. For example, the simulation engine's API is invoked to apply the calculated torque values ​​to the motor models of each joint in the robot model. In the next simulation step, the physics simulation engine updates the robot model's joint positions, velocities, accelerations, and overall posture in the simulation environment based on these applied torques and the robot model's physical properties such as inertia and friction. This updated state is then used to generate the virtual sensor frames for the next cycle, thus forming a continuous closed-loop simulation process.

[0073] In summary, the above processing effectively solves the technical problem of how to effectively fuse the virtual sensor data received by the control terminal with the control commands corresponding to the simulation task during the simulation process, and transform it into actuator commands that can drive the update of the dynamic model, thereby realizing the closed-loop interaction between the real control logic and the simulation dynamic environment. Specifically, the control terminal can accurately receive and process the virtual sensor frames from the simulation terminal, combine them with the upper-level control commands, and generate actuator commands that conform to the actual control logic through its internal control program. This processing mechanism ensures that the control program can make "decision-making" based on real-time "perception" information in the simulation environment, just like operating a real robot. At the same time, the simulation terminal can accurately convert these actuator commands into model control information that the dynamic model can recognize, and update the state of the dynamic model in real time. This tight closed-loop interaction allows the control program to fully participate in the dynamic calculation of the simulation environment, thereby fully verifying the performance, stability, and robustness of the control algorithm during the simulation stage. Compared with the loose coupling or simplified controller of the controller and the simulation environment in traditional simulation schemes, this scheme greatly improves the realism and reliability of the simulation results by allowing the control program of the same origin to directly participate in the dynamic closed loop. This not only effectively identifies and resolves potential problems in the control program during actual operation, but also significantly reduces the debugging and adaptation work required from simulation to actual deployment, thereby accelerating the development and verification cycle of the robot system and reducing development costs and risks.

[0074] Furthermore, during the actual switching process, due to the differences between the simulation environment and the physical environment in terms of underlying configurations such as running location and network address, it is often necessary to reconfigure the control program, modify the calling method, or adjust the data structure. This results in the control program after simulation verification not being able to be seamlessly migrated to the physical robot, increasing the complexity of deployment and the risk of errors. Therefore, in this embodiment, when the virtual hardware interface is switched to the physical hardware interface, the target robot maintains the same calling method and control data structure, while the running location, running mode, hardware interface, or network address of the control program is switched.

[0075] Specifically, this switching mechanism aims to achieve a seamless transition of the underlying channel for interaction between the control program and the robot from a virtual environment to a physical environment. In essence, it replaces the simulation interface originally connected to the control program with an interface that communicates directly with the real robot hardware. Implementation methods can include: dynamically unbinding the control program from the virtual interface and re-establishing the binding with the physical interface through system-level configuration or software API calls; or, within the operating system or runtime environment, utilizing the loading and unloading mechanisms of dynamic link libraries or shared objects to replace the implementation module of the virtual interface at runtime and load and enable the implementation module of the physical interface. "Calling method" refers to the interface specifications and communication protocols followed when the control program interacts with the target robot, such as function signatures, API interface definitions, and message passing patterns. Maintaining the calling method means that the control program uses the same instruction set and interaction paradigm to control the robot or obtain robot status in both simulation and physical modes. This can be achieved by: designing a unified abstraction layer or adaptation layer that provides a standardized calling interface to the upper-level control program and internally encapsulates the differences between virtual and physical interfaces; or, predefining a set of virtual interface APIs that are fully compatible with the real robot SDK (Software Development Kit), enabling the control program to call the same API functions or methods in both modes. "Control data structure" refers to the data format, data type, field order, and data encoding method used by the control program when sending control commands (such as joint angle and speed commands) or receiving sensor feedback (such as joint position and inertial data). Maintaining the control data structure ensures the consistency of data transmission and parsing between simulation and physical environments. This can be achieved by: defining a standardized data message format, such as using the message types of the Robot Operating System (ROS), the data types of the Data Distribution Service (DDS), or a custom protocol buffer, and strictly adhering to this standard in both simulation and physical environments; or, through a unified serialization and deserialization mechanism, ensuring that the data transmission and parsing format remains consistent across different operating environments. "Running position" refers to the actual computing node or physical device on which the control program executes. Switching the execution location means that the control program can be migrated from one computing resource (such as a development workstation or server used for simulation) to another computing resource (such as an embedded controller integrated on the robot body or an industrial PC). Implementation methods can include: utilizing containerization technologies (such as Docker and Kubernetes) to package the control program and its dependencies into portable images, enabling deployment and execution on different computing nodes; or, through distributed system frameworks or remote procedure call mechanisms, achieving flexible execution and scheduling of the control program logic at different physical locations."Operating mode" refers to the control program's adjustment of its internal behavior, parameter configuration, or algorithm logic based on the current environment (simulation or physical machine). Switching operating modes allows the control program to adapt to the characteristics of different environments without modifying its core code. Implementation methods can include: loading different preset sets of operating parameters (e.g., gain parameters and safety thresholds for a PID controller) for the simulation or physical machine environment at startup via external configuration files, environment variables, or command-line parameters; or implementing a mode-switching logic within the control program to dynamically adjust its internal state machine or algorithm branches based on received mode-switching instructions or detected environment flags. "Hardware interface" refers to the specific channel or driver program through which the control program communicates with the underlying hardware (whether virtual or physical). Switching the hardware interface means that the interface the control program connects to changes from a virtual interface (e.g., an API provided by the emulator) to an actual physical interface (e.g., a CAN bus driver, an EtherCAT driver, or GPIO). Implementation methods can include: modifying the interface address, device handle, or driver path pointed to internally by the control program through software configuration or registry settings; or dynamically binding to different driver modules or device files at the underlying level using the device abstraction layer or hardware abstraction layer provided by the operating system. "Network address" refers to network endpoint information such as IP address and port number used by the control program to communicate with the robot or its related components. Switching network addresses is necessary to adapt to the differences in network configuration between the simulation environment (usually a local loopback or LAN) and the physical robot environment (which may involve different IP segments or remote communication). Implementation methods can include: dynamically obtaining the target robot's current network address through service discovery mechanisms (such as DNS service, mDNS, or a custom service registry); or configuring multiple network configuration files in the control program and loading the corresponding network parameters, such as the target IP address, port number, or communication protocol parameters, based on the current operating environment (simulation or physical machine).

[0076] Based on this, by decoupling the logic layer and the underlying communication layer of the control program, seamless integration between the simulation and the actual machine environment is achieved. When the control system performs a simulation task, the control end constructs actuator commands based on control commands and virtual sensor frames through the control program, and sends them to the simulation end through the control input channel. The simulation end updates the state of the dynamic model based on the actuator commands. When the control program meets the simulation conditions of the simulation task, it needs to switch the virtual hardware interface to the physical hardware interface to drive the target robot. The key to this switching process is that the target robot maintains its calling method and control data structure unchanged. This means that regardless of whether the control program communicates with the virtual hardware interface or the physical hardware interface, the API interface, message format, and data type used to send instructions (such as joint positions and velocities) to the robot or receive robot states (such as sensor data) remain consistent. This consistency ensures that the control logic and algorithms verified in the simulation phase can be directly applied to the physical robot without code-level modifications or re-adaptation. Furthermore, to adapt to the transition from the simulation environment to the physical environment, the corresponding running position, running mode, hardware interface, or network address of the control program will be dynamically switched. For example, the control program might be migrated from the development workstation to the embedded controller of the robot (operational position switching), its internal parameters might be adjusted from simulation mode to live mode (operational mode switching), its communication channel might be switched from a virtual interface to an actual CAN bus or EtherCAT interface (hardware interface switching), or its communication target IP address might be switched from a local loopback address to the robot's actual network address (network address switching). In this way, the core logic of the control program remains stable, while differences in the underlying environment are addressed through adjustments to configuration or runtime parameters. This design cleverly solves the deployment complexity and error risks caused by differences between simulation and live environments in traditional solutions, greatly simplifying the process from simulation verification to live deployment and ensuring the consistency and stability of the control program under different operating environments.

[0077] For example, consider a robot control system based on a Robot Operating System (ROS). During simulation, the control program (e.g., a motion planning node or a torque control node) publishes joint commands (e.g., ` / robot / joint_group_controller / command`) and subscribes to sensor feedback (e.g., ` / joint_states`, ` / imu / data`) through ROS topics. The message types of these ROS topics (e.g., `std_msgs / Float64MultiArray`, `sensor_msgs / JointState`, `sensor_msgs / Imu`) remain completely consistent between simulation and physical implementation, ensuring that the target robot maintains its calling methods and control data structures unchanged. In the simulation environment, a robot simulator (e.g., Gazebo) acts as a virtual hardware interface, subscribing to these command topics and publishing sensor feedback topics. When switching to the physical robot is necessary, the control program can remain unchanged. At this point, the operating position of the control program can be switched from the development computer to the embedded controller on the robot body. The operating mode of the control program can be implemented by loading different configuration files. For example, in simulation mode, certain safety mechanisms may be disabled or a simplified physical model may be used, while in live mode, full safety monitoring and precise control parameters are enabled. The hardware interface switching for the control program is reflected in the fact that ROS topics, which originally communicated with the simulator, are now subscribed to and published by a hardware interface node that interacts directly with the real robot hardware (such as motor drivers, encoders, and IMU sensors). This hardware interface node is responsible for converting ROS commands into hardware instructions and hardware feedback into ROS messages. The network address switching for the control program can be achieved by modifying the URI (Uniform Resource Identifier) ​​environment variable of the ROS Master node to point to the IP address of the physical robot, thereby enabling cross-network communication between the control program and the physical robot. In this way, after simulation verification, the control program can be seamlessly migrated from the simulation environment to run on the physical robot without modifying the code, simply by adjusting the configuration files and switching the operating environment.

[0078] In summary, the above approach effectively solves the problem that traditional solutions require re-adapting the control algorithm to the physical SDK, ROS / DDS topics, device naming, and underlying communication endpoints after simulation verification, leading to increased development workload and potential exposure of uncovered operational environment differences. Specifically, by forcing the target robot to maintain its calling method and control data structure unchanged, the necessity of code-level refactoring or adaptation during the switch between simulation and physical deployment is eliminated, greatly simplifying the process from simulation verification to physical deployment. Simultaneously, by dynamically switching the control program's corresponding running position, running mode, hardware interface, or network address, the control program can automatically adapt to the underlying communication endpoints and running parameters according to the current simulation or physical environment, ensuring the consistency and stability of the control program under different operating environments. This not only reduces deployment complexity and error risk but also improves development efficiency and system reliability, achieving a smooth and seamless migration of the control program from simulation to physical deployment.

[0079] During high-frequency data interaction between the simulation and control ends, conventional communication methods may lead to timing discrepancies between the dynamic model updates at the simulation end and the command generation at the control end due to data transmission delays, imperfect synchronization mechanisms, or excessive communication overhead. This can affect the real-time performance and accuracy of the simulation closed loop. To address this, in this embodiment, the simulation end is further configured to encode the joint information and inertial state information associated with the motion data into virtual sensor frames corresponding to the control program according to a preset mapping relationship, and write them into a first shared memory through the sensor output channel. The control end is further configured to read the virtual sensor frames from the first shared memory, construct actuator commands based on the virtual sensor frames and the control commands corresponding to the simulation task, write the actuator commands into a second shared memory through the control input channel, and read the actuator commands from the second shared memory to update the state of the dynamic model based on the actuator commands.

[0080] Specifically, the pre-defined mapping relationship defines how the raw motion data (such as joint information and inertial state information) generated by the simulation end is converted into the format and content of virtual sensor frames that the control program can understand and process. This can be pre-configured in a configuration file, such as an XML or JSON format mapping table; it can also be dynamically established during system initialization via software modules, for example, by defining the correspondence and conversion rules of data fields through a programming interface; or it can be integrated into the data processing logic of the simulation end through hard coding. The joint information and inertial state information associated with the motion data typically refer to the position, velocity, acceleration, and other state data of each joint of the robot, as well as the robot's posture (such as Euler angles or quaternions), angular velocity, linear acceleration, and other data. This information collectively describes the kinematic and dynamic state of the robot in the simulation environment and can be calculated or simulated in real time by the dynamic model inside the simulation end. Encoding the virtual sensor frames corresponding to the control program means encapsulating and converting the raw joint information and inertial state information according to a specific data structure and protocol format to make it conform to the sensor data format expected by the control program. Virtual sensor frames are encoded data packets that simulate the output of real robot sensors (such as encoders and IMUs), allowing the control program to distinguish between data from real sensors and the simulation environment. The sensor output channel is part of the virtual hardware interface, serving as the logical path for transmitting virtual sensor frames from the simulation end to the control end. First shared memory is an inter-process communication (IPC) mechanism that allows multiple processes to access the same physical memory area, specifically used to store virtual sensor frames generated by the simulation end for the control end to read. It can be created and managed using shared memory APIs provided by the operating system (such as System V shared memory or POSIX shared memory under Linux) or implemented through memory-mapped files. Reading virtual sensor frames refers to the control end accessing the first shared memory to obtain the virtual sensor frames written by the simulation end. Reading operations typically require synchronization mechanisms (such as semaphores and mutexes) to ensure data consistency and integrity. The control commands corresponding to the simulation task are instructions generated by the control program based on the simulation task's objective and the currently received virtual sensor frames, used to drive the robot's actuators. Constructing actuator commands refers to the control program calculating and generating specific instructions to drive the robot actuators based on its internal control algorithm, combined with received virtual sensor frames and simulation task control commands. The control input channel is another part of the virtual hardware interface, serving as the logical path for transmitting actuator commands from the control end to the simulation end. The second shared memory is also used for inter-process communication, but in the opposite direction; it is specifically used to store actuator commands generated by the control end for the simulation end to read. Reading actuator commands means that the simulation end accesses the second shared memory to retrieve the actuator commands written by the control end.The state of updating the dynamic model based on actuator commands refers to the process by which the simulation end receives actuator commands, parses them, and applies them to its internal dynamic model. For example, it calculates the acceleration of the joints based on the actuator commands, and then updates the velocity and position of the joints to simulate the motion of the robot.

[0081] Based on this, by introducing a shared memory mechanism, the efficiency of high-frequency data interaction between the simulation end and the control end is significantly optimized, thus solving problems such as data transmission delay, imperfect synchronization mechanism, and excessive communication overhead that may be caused by traditional communication methods. Specifically, in simulation mode, the simulation end first determines the robot's motion data according to the simulation task corresponding to the control program. This data includes joint information and inertial state information. To enable the control program to directly process this data, the simulation end encodes this motion data into virtual sensor frames that the control program can recognize, according to a preset mapping relationship. Subsequently, the simulation end writes these virtual sensor frames directly into the pre-allocated first shared memory, without using conventional network protocols or message queues. This direct memory-level write operation greatly reduces data transmission overhead and latency. At the same time, the control end reads these virtual sensor frames from the first shared memory with extremely high efficiency through its sensor output channel. After receiving the virtual sensor frames, the control program combines them with the control commands associated with the simulation task to construct actuator commands for driving the robot actuators. To efficiently transmit these actuator commands back to the simulation end, the control end writes these commands into the second shared memory through its control input channel. The simulation end reads these actuator commands from the second shared memory and applies them to its internal dynamics model, thereby updating the state of the dynamics model. Through this bidirectional shared memory architecture, data flow between the simulation and control ends achieves memory-level speeds, avoiding time-consuming operations such as serialization, deserialization, and network transmission in traditional communication protocols. This allows the updates to the simulation's dynamics model to closely follow the commands from the control end, ensuring the real-time performance and accuracy of the simulation loop. This mechanism enables the control program to obtain real-time data streams highly consistent with the physical robot's operating environment in the simulation environment, providing a solid foundation for subsequently switching the virtual hardware interface to a physical hardware interface and driving the target robot through the control program.

[0082] For example, a POSIX shared memory mechanism can be used to implement the first and second shared memory. The simulation end can create a POSIX shared memory object named " / robot_sensor_shm" as the first shared memory and map it to the address spaces of both the simulation and control ends. The simulation end packages joint information and inertial state information (e.g., joint angles, angular velocities, linear acceleration, etc.) into a predefined structure (such as SensorFrame_t) and writes it to this shared memory region. To ensure data synchronization, POSIX semaphores or mutexes can be used. For example, the simulation end acquires the lock before writing data, releases the lock after writing, and sends a signal to the control end to notify that the data has been updated. The control end can open and map the same " / robot_sensor_shm" shared memory object and read the SensorFrame_t structure from this shared memory through the sensor output channel. When reading data, the control end also needs to acquire the corresponding synchronization lock. Based on the read virtual sensor frames and simulation task control commands, the control program calculates the actuator commands (e.g., target joint torque or position) and encapsulates them into another predefined structure (such as ActuatorCommand_t). Subsequently, the control end can create another POSIX shared memory object named " / robot_actuator_shm" as a second shared memory, write the ActuatorCommand_t structure into it, and synchronize it using semaphores or mutexes. The simulation end opens and maps the " / robot_actuator_shm" shared memory through the control input channel, reads the ActuatorCommand_t structure, parses it, applies it to its dynamics model, and updates the robot's state.

[0083] In practice, the simulator can establish bidirectional shared memory and synchronization signals for each robot. The simulator encodes joint and inertial states into sensor protocol frames that the control program can recognize and writes them into the first shared memory; the control program reads these frames, calculates the joint targets and gains, and then writes the control protocol frames into the second shared memory. The simulator parses the control protocol frames and writes them to the physical actuators. The virtual hardware interface can use ROS2 or DDS to publish sensor states and subscribe to actuator commands. The control program continues to use its original message types and namespaces. The simulator publishes a simulation clock, causing control and application nodes that depend on robot time to run according to the simulation time.

[0084] In summary, the above-described methods effectively solve the problems of transmission delay, synchronization deviation, and excessive communication overhead in high-frequency data interaction between the simulation and control ends. By introducing first and second shared memory, data transmission between the simulation and control ends is made in-memory, greatly improving the real-time performance and efficiency of data exchange. This allows the control program to acquire sensor data from the simulation environment and send actuator commands at speeds approaching those of physical hardware, thereby establishing a highly real-time and accurate dynamic closed loop within the simulation environment, involving real control logic. This efficient data interaction mechanism ensures the reliability of simulation results and provides a solid technical guarantee for the seamless switching of the control program from simulation mode to physical robot drive mode, significantly reducing development and debugging risks. When dealing with multi-robot collaboration or multi-task parallel scenarios, if the control system only supports the control logic of a single robot, conflicts or confusion may occur in the control commands, status data, and hardware interface configurations between different robots, making it impossible to achieve independent and precise control of multiple target robots. Therefore, in this embodiment, when there are multiple target robots, each target robot is associated with a corresponding control process, configuration file, and virtual hardware interface of the control program.

[0085] Specifically, the control process of a control program refers to a program instance that runs independently at the operating system level to execute specific control logic. Each control process has its own independent memory space, system resources, and execution context, thereby ensuring that the control logic between different robots does not interfere with each other. Its implementation can include, but is not limited to: starting an independent operating system process for each target robot to run its control program; or using containerization technology to create an independent container instance for each control program to achieve lightweight isolation and resource management. A configuration file is a collection of text or binary data used to store the parameters, settings, and mapping relationships required for the operation of the control program. By providing each target robot with an independent configuration file, customized configurations can be made for the physical characteristics, task requirements, or environmental parameters of different robots without affecting the operation of other robots. The implementation of configuration files can include, but is not limited to: maintaining an independent XML, JSON, or YAML format configuration file for each target robot; or storing the configuration information in a database and distinguishing and loading it using a unique robot identifier. The virtual hardware interface is a software abstraction layer that simulates the behavior of the real robot hardware interface. It provides the control program with an input / output mechanism consistent with the physical hardware in the simulation environment. By assigning a separate virtual hardware interface to each target robot, it can be ensured that each control process can receive its own sensor data and send its own actuator commands, thereby achieving precise routing and isolation of data flows. The implementation of the virtual hardware interface can include, but is not limited to: utilizing a shared memory region to allocate a separate memory segment for each robot for exchanging sensor data and actuator commands; or establishing a separate communication channel for each robot through message queues or named pipes.

[0086] Based on this, a multi-instance isolation mechanism is introduced to achieve refined management of multi-robot systems. In multi-robot scenarios, by independently configuring a dedicated control process, configuration file, and virtual hardware interface for each target robot, it is ensured that the control logic and data flow between different robots do not interfere with each other. Specifically, by assigning an independent control process to each robot, parallel processing of multiple tasks can be achieved, avoiding potential blocking or logical errors that may occur when a single process handles multiple tasks. Simultaneously, by configuring independent configuration files, differentiated parameter settings can be made for the physical characteristics or task requirements of different robots, ensuring the flexibility and adaptability of the control program. Furthermore, by establishing an independent virtual hardware interface, precise routing of sensor data and actuator commands is achieved, ensuring that each robot can accurately receive its own control instructions and provide correct status information. This design gives the system good scalability, effectively supporting complex multi-robot collaborative operation environments and solving the problems of resource contention and data coupling in multi-robot control. In other words, the system creates an independent control program process, configuration file, robot name, and virtual channel for each virtual robot. When a single robot is reset, only the corresponding control program and channel are rebuilt, without affecting the closed loop of other robots.

[0087] For example, suppose there is a scenario with three target robots (Robot A, Robot B, and Robot C). To achieve independent control and simulation of these three robots, the system can launch a separate control program process for each robot. For example, in a Linux operating system, three independent processes can be launched, each running the same robot control program code, but they are isolated from each other in memory. Each process loads an associated configuration file. For example, Robot A's configuration file might contain specific calibration parameters and a dynamic model for its joints; Robot B's configuration file might define its unique end effector type and force sensor thresholds; and Robot C's configuration file might specify its path planning parameters for a specific task. These configuration files can be separate YAML files, named robotA_config.yaml, robotB_config.yaml, and robotC_config.yaml, respectively. Furthermore, each control process communicates with a separate virtual hardware interface. For example, a shared memory region named ` / dev / shm / robotA_virtual_hw` can be allocated to robot A as its virtual hardware interface, ` / dev / shm / robotB_virtual_hw` to robot B, and ` / dev / shm / robotC_virtual_hw` to robot C. When the control terminal needs to send actuator commands to robot A, it writes the commands to `robotA_virtual_hw`; when the simulation terminal of robot A generates sensor data, it writes the data to `robotA_virtual_hw`, which is then read by the control process of robot A. This independent configuration and communication mechanism ensures that even if the control program of robot A fails or needs to be reset, only its corresponding control process and virtual hardware interface need to be restarted, without affecting the normal operation of robots B and C.

[0088] In summary, the above approach effectively solves the problem of conflicting or confusing control commands, status data, and hardware interface configurations caused by the single control logic in traditional control systems during multi-robot collaboration or multi-task parallel scenarios. By providing each target robot with an independent control process, configuration file, and virtual hardware interface, fine-grained management and complete isolation of the multi-robot system are achieved. This not only ensures that each robot can independently and accurately receive and execute control commands, avoiding resource contention and data coupling, but also significantly improves the system's robustness and scalability. When a robot needs maintenance, debugging, or reset, its operation is limited to its own independent instance and will not affect other running robots, thus ensuring the stability and continuous operation of the entire multi-robot system.

[0089] Furthermore, the same motion control executable program supports both live mode and simulation mode. Live mode connects to physical hardware drivers, while simulation mode connects to virtual hardware interfaces. Both modes load compatible control libraries and configuration systems, and provide the same SDK or service interfaces to upper-layer applications. In this embodiment, the control program is a user-mode motion control program on the robot's main control computing platform and is not required to be embedded firmware on the motor control board. Embedded firmware can also be used as an implementation of the control program when instruction set simulation, board-level virtualization, or hardware-in-the-loop conditions are available. Figure 2 The diagram shows that on the same robot application, the target operating environment can be selected according to the needs. Whether it is simulation or actual machine, the same source control program can be started. In simulation mode, communication can be achieved by connecting virtual hardware endpoints, and in actual machine mode, communication can be achieved by connecting physical hardware endpoints. After that, by starting the same source control program and providing the same SDK or message interface, the corresponding task execution and control operation can be realized.

[0090] The following is in conjunction with the appendix Figure 3 Taking the application of the control system provided in this specification in a simulation scenario as an example, the control system will be further explained. Figure 3 A flowchart illustrating the processing procedure of a control system according to an embodiment of this specification is shown, specifically including the following steps.

[0091] Step S302: The simulation terminal loads a dynamic model containing robot joints, actuators, and sensors, and obtains the joint identifiers and joint sequences associated with the robot joints, the actuator index information and inertial measurement unit associated with the actuators, and the sensor information associated with the sensors in the dynamic model; obtains the hardware interface description information corresponding to the control program, and determines the control program joints, control program sensors, and control program output channels based on the hardware interface description information; establishes a first mapping relationship between the control program joints and the joint identifiers and joint sequences, a second mapping relationship between the control program sensors and sensor information, and a third mapping relationship between the control program output channels and the actuator index information and inertial measurement unit; and uses the first, second, and third mapping relationships as preset mapping relationships.

[0092] In step S304, the control terminal starts a control program that is the same as the target robot and loads the runtime library, configuration information and robot middleware associated with the target robot; the control program connects to the virtual hardware interface through preset communication configuration parameters.

[0093] Step S306: The simulation terminal determines the simulation task corresponding to the control program, and acquires joint position information, joint velocity information, body posture information, angular velocity information, linear acceleration information, contact state information and / or force information based on the simulation task as motion data; the motion data is mapped according to a preset mapping relationship to obtain an initial virtual sensor frame; the initial virtual sensor frame is encapsulated according to the protocol structure corresponding to the control program to obtain a virtual sensor frame, and sent to the control terminal through the sensor output channel.

[0094] In step S308, the control terminal receives the virtual sensor frame and determines the control command corresponding to the simulation task; the control program processes the virtual sensor frame and the control command to obtain the actuator command, and sends it to the simulation terminal through the control input channel.

[0095] In step S310, the simulation terminal converts the actuator commands into model control information and writes the model control information into the actuator corresponding to the dynamic model. The actuator then updates the state of the dynamic model based on the model control information.

[0096] In step S312, if the control terminal determines that the control program meets the simulation conditions of the simulation task based on the updated state, it switches the virtual hardware interface to the physical hardware interface between the control terminal and the target robot, which is used to drive the target robot through the control program.

[0097] In summary, by establishing a closed-loop communication mechanism between the control end and the simulation end through a virtual hardware interface, lossless conversion of motion data and virtual sensor frames is achieved using a preset mapping relationship, and seamless switching to the physical hardware interface is possible when simulation conditions are met. This allows the control program to drive the target robot without modification, solving the problem of inconsistency between the simulation environment and the physical robot control program, avoiding repeated adaptation work, shortening the development cycle, and reducing the risk of operational failures.

[0098] Corresponding to the above system embodiments, this specification also provides embodiments of control methods. Figure 4 This is a flowchart illustrating a control method provided in one embodiment of this specification. Figure 4 As shown, this method is applied to a control terminal, which is configured with a control program. The control terminal connects to a simulation terminal via a virtual hardware interface to achieve a simulation mode. The simulation terminal is configured with a dynamic model of the associated target robot. The virtual hardware interface includes a control input channel and a sensor output channel, including: Step S402: Receive the virtual sensor frame sent by the simulation terminal through the sensor output channel, wherein the virtual sensor frame is obtained by converting motion data according to a preset mapping relationship, and the motion data is determined according to the simulation task corresponding to the control program.

[0099] Step S404: The control program constructs actuator commands based on the control commands associated with the simulation task and the virtual sensor frame, and sends them to the simulation terminal through the control input channel.

[0100] Step S406: Based on the state of the dynamic model updated by the simulation terminal according to the actuator command, and after determining that the control program meets the simulation conditions of the simulation task, the virtual hardware interface is switched to the physical hardware interface between the control terminal and the target robot, for driving the target robot through the control program.

[0101] The above is an illustrative scheme of a control method according to this embodiment. It should be noted that the technical solution of this control method and the technical solution of the control system described above belong to the same concept. For details not described in detail in the technical solution of the control method, please refer to the description of the technical solution of the control system described above.

[0102] Corresponding to the above method embodiments, this specification also provides control device embodiments. Figure 5 A schematic diagram of a control device according to one embodiment of this specification is shown. Figure 5 As shown, the device is applied to the control terminal, which is configured with a control program. The control terminal connects to the simulation terminal in simulation mode via a virtual hardware interface. The simulation terminal is configured with a dynamic model of the associated target robot. The virtual hardware interface includes a control input channel and a sensor output channel, including: The receiving module 502 is configured to receive a virtual sensor frame sent by the simulation terminal through the sensor output channel, wherein the virtual sensor frame is obtained by converting motion data according to a preset mapping relationship, and the motion data is determined according to the simulation task corresponding to the control program. The construction module 504 is configured to construct actuator commands through the control program based on the control commands associated with the simulation task and the virtual sensor frame, and send them to the simulation terminal through the control input channel; The determination module 506 is configured to, based on the state of the dynamic model updated by the simulation terminal according to the actuator command, and if the control program satisfies the simulation conditions of the simulation task, switch the virtual hardware interface to the physical hardware interface between the control terminal and the target robot, for driving the target robot through the control program.

[0103] The above is a schematic representation of a control device according to this embodiment. It should be noted that the technical solution of this control device and the technical solution of the control method described above belong to the same concept. Details not described in detail in the technical solution of the control device can be found in the description of the technical solution of the control method described above.

[0104] Figure 6 A structural block diagram of a computing device 600 according to one embodiment of this specification is shown. The components of the computing device 600 include, but are not limited to, a memory 610 and a processor 620. The processor 620 is connected to the memory 610 via a bus 630, and a database 650 is used to store data.

[0105] The computing device 600 also includes an access device 640, which enables the computing device 600 to communicate via one or more networks 660. Examples of these networks include Public Switched Telephone Network (PSTN), Local Area Network (LAN), Wide Area Network (WAN), Personal Area Network (PAN), or combinations of communication networks such as the Internet. The access device 640 may include one or more of any type of wired or wireless network interface (e.g., a network interface card (NIC)), such as an IEEE 802.11 Wireless Local Area Network (WLAN) wireless interface, a Wi-MAX (Worldwide Interoperability for Microwave Access) interface, an Ethernet interface, a Universal Serial Bus (USB) interface, a cellular network interface, a Bluetooth interface, or a Near Field Communication (NFC) interface.

[0106] In one embodiment of this specification, the above-described components of the computing device 600 and Figure 6 Other components, not shown, can also be connected to each other, for example, via a bus. It should be understood that... Figure 6 The block diagram of the computing device shown is for illustrative purposes only and is not intended to limit the scope of this specification. Those skilled in the art can add or replace other components as needed.

[0107] The computing device 600 can be any type of stationary or mobile computing device, including mobile computers or mobile computing devices (e.g., tablet computers, personal digital assistants, laptop computers, notebook computers, netbooks, etc.), mobile phones (e.g., smartphones), wearable computing devices (e.g., smartwatches, smart glasses, etc.) or other types of mobile devices, or stationary computing devices such as desktop computers or personal computers (PCs). The computing device 600 can also be a mobile or stationary server.

[0108] The processor 620 is configured to execute the following computer-executable instructions, which, when executed by the processor, implement the steps of the control method described above.

[0109] The above is an illustrative scheme of a computing device according to this embodiment. It should be noted that the technical solution of this computing device and the technical solution of the control method described above belong to the same concept. For details not described in detail in the technical solution of the computing device, please refer to the description of the technical solution of the control method described above.

[0110] An embodiment of this specification also provides a computer-readable storage medium storing computer-executable instructions that, when executed by a processor, implement the steps of the control method described above.

[0111] The above is an illustrative scheme of a computer-readable storage medium according to this embodiment. It should be noted that the technical solution of this storage medium and the technical solution of the control method described above belong to the same concept. For details not described in detail in the technical solution of the storage medium, please refer to the description of the technical solution of the control method described above.

[0112] An embodiment of this specification also provides a computer program product, including a computer program or instructions that, when executed by a processor, implement the steps of the control method described above.

[0113] The above is an illustrative scheme of a computer program product according to this embodiment. It should be noted that the technical solution of this computer program product and the technical solution of the control method described above belong to the same concept. For details not described in detail in the technical solution of the computer program product, please refer to the description of the technical solution of the control method described above.

[0114] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0115] The computer instructions include computer program code, which may be in the form of source code, object code, executable file, or certain intermediate forms. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording media, USB flash drive, portable hard drive, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium may be appropriately added or removed according to the requirements of patent practice. For example, in some regions, according to patent practice, computer-readable media may not include electrical carrier signals and telecommunication signals.

[0116] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that the embodiments in this specification are not limited to the described order of actions, because according to the embodiments in this specification, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in this specification are all preferred embodiments, and the actions and modules involved are not necessarily essential to the embodiments in this specification.

[0117] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0118] The preferred embodiments disclosed above are merely illustrative of this specification. The optional embodiments do not exhaustively describe all details, nor do they limit the invention to the specific implementations described. Clearly, many modifications and variations can be made based on the embodiments described in this specification. These embodiments are selected and specifically described in this specification to better explain the principles and practical applications of the embodiments, thereby enabling those skilled in the art to better understand and utilize this specification.

Claims

1. A control system, characterized in that, The system includes a control terminal and a simulation terminal. The control terminal is configured with a control program, and the simulation terminal is configured with a dynamic model of the associated target robot. The simulation terminal and the control terminal are connected in simulation mode through a virtual hardware interface, which includes a control input channel and a sensor output channel. The simulation terminal is used to determine motion data according to the simulation task corresponding to the control program; convert the motion data into virtual sensor frames according to a preset mapping relationship, and send them to the control terminal through the sensor output channel; The control terminal is used to construct actuator commands based on the control commands associated with the simulation task and the virtual sensor frame through the control program, and send them to the simulation terminal through the control input channel. The simulation terminal is used to update the state of the dynamic model based on the executor commands; The control terminal is used to switch the virtual hardware interface to the physical hardware interface between the control terminal and the target robot when the control program meets the simulation conditions of the simulation task based on the updated state, so as to drive the target robot through the control program.

2. The control system according to claim 1, characterized in that, The simulation terminal is also used to load a dynamic model containing robot joints, actuators, and sensors, and to obtain the joint identifiers and joint order associated with the robot joints, the actuator index information and inertial measurement unit associated with the actuators, and the sensor information associated with the sensors in the dynamic model; to obtain the hardware interface description information corresponding to the control program, and to determine the control program joints, control program sensors, and control program output channels based on the hardware interface description information; Establish a first mapping relationship between the control program joints and the joint identifiers and the joint sequence, a second mapping relationship between the control program sensors and the sensor information, and a third mapping relationship between the control program output channels and the actuator index information and the inertial measurement unit; use the first mapping relationship, the second mapping relationship, and the third mapping relationship as preset mapping relationships.

3. The control system according to claim 1, characterized in that, The control terminal is also used to start a control program that is of the same origin as the target robot, and load the runtime library, configuration information and robot middleware associated with the target robot; and to connect the control program with the virtual hardware interface through preset communication configuration parameters.

4. The control system according to claim 1 or 2, characterized in that, The simulation terminal is also used to determine the simulation task corresponding to the control program, and to acquire joint position information, joint velocity information, body posture information, angular velocity information, linear acceleration information, contact state information and / or force information based on the simulation task as the motion data; and to perform mapping processing on the motion data according to a preset mapping relationship to obtain an initial virtual sensor frame; The initial virtual sensor frame is encapsulated according to the protocol structure corresponding to the control program to obtain the virtual sensor frame, and then sent to the control terminal through the sensor output channel.

5. The control system according to claim 1, characterized in that, The control terminal is also used to receive the virtual sensor frame and determine the control command corresponding to the simulation task; process the virtual sensor frame and the control command through the control program to obtain the actuator command, and send it to the simulation terminal through the control input channel; The simulation terminal is also used to convert the actuator commands into model control information, write the model control information into the actuator corresponding to the dynamic model, and update the state of the dynamic model through the actuator based on the model control information.

6. The control system according to claim 1, characterized in that, When the virtual hardware interface is switched to the physical hardware interface, the target robot maintains the same calling method and control data structure, while the running position, running mode, hardware interface or network address of the control program are switched.

7. The control system according to claim 1, characterized in that, The simulation terminal is also used to encode the joint information and inertial state information associated with the motion data into virtual sensor frames corresponding to the control program according to a preset mapping relationship, and write them into the first shared memory through the sensor output channel. The control terminal is also used to read the virtual sensor frame from the first shared memory and construct actuator commands based on the virtual sensor frame and the control commands corresponding to the simulation task; The executor command is written into the second shared memory through the control input channel; The simulation terminal is also used to read the executor commands from the second shared memory and update the state of the dynamic model based on the executor commands.

8. The control system according to claim 1, characterized in that, When there are multiple target robots, each target robot is associated with the control process, configuration file and virtual hardware interface of the control program.

9. A control method, characterized in that, Applied to the control end, the control end is configured with a control program. The control end connects to the simulation end in simulation mode through a virtual hardware interface. The simulation end is configured with a dynamic model of the associated target robot. The virtual hardware interface includes a control input channel and a sensor output channel, including: The simulation terminal receives a virtual sensor frame sent through the sensor output channel, wherein the virtual sensor frame is obtained by converting motion data according to a preset mapping relationship, and the motion data is determined according to the simulation task corresponding to the control program. The control program constructs actuator commands based on the control commands associated with the simulation task and the virtual sensor frame, and sends them to the simulation terminal through the control input channel. Based on the state of the dynamic model updated by the simulation terminal according to the actuator command, and after determining that the control program meets the simulation conditions of the simulation task, the virtual hardware interface is switched to the physical hardware interface between the control terminal and the target robot, for driving the target robot through the control program.

10. A computing device, characterized in that, include: Memory and processor; The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions, which, when executed by the processor, implement the steps of the method of claim 9.

11. A computer-readable storage medium, characterized in that, It stores computer-executable instructions that, when executed by a processor, implement the steps of the method of claim 9.

12. A computer program product, characterized in that, It includes a computer program or instructions that, when executed by a processor, implement the steps of the method of claim 9.

Images