A real-time simulation and visualization method of robotic arm based on UE4

Abstract

The present application provides a UE4-based real-time simulation and visualization method for a robotic arm, which belongs to the field of virtual simulation and robotics technology, comprising: establishing a robotic arm model based on a UE4 editor, configuring parameters to generate a transformation matrix and a tool coordinate system matrix; the UE4 editor establishes communication with a robotic arm controller, and receives the current robotic arm joint angle and output data sent by the robotic arm controller at a preset frequency; the UE4 editor solves the current robotic arm joint angle through a forward kinematics algorithm based on D-H parameters to obtain position information and posture information of the robotic arm end effector; based on the tool coordinate system matrix, the position information and the posture information, the transformation matrix is ​​updated to control the position and posture of the robotic arm joint of the robotic arm model, a motion trajectory of the robotic arm end effector is generated based on the position information, and the color of the motion trajectory is changed based on the output data.

Description

Technical Field

[0001] The present application belongs to the field of virtual simulation and robotics technology, and specifically relates to a real-time simulation and visualization method of a robotic arm based on UE4. Background Art

[0002] With the widespread application of industrial robots in intelligent manufacturing, automated assembly, and flexible production, visual simulation systems for robotic arms have become important tools for R&D, teaching, debugging, and remote monitoring. Traditional robotic arm development and debugging often rely on physical prototypes, which are not only costly and time-consuming, but also lack the ability to reflect the robotic arm's operating status in real time, limiting its application in complex tasks.

[0003] In recent years, virtual simulation technology has been gradually introduced into the field of robotics. In particular, 3D visualization platforms built on high-performance graphics engines like Unreal Engine 4 (UE4) provide excellent support for robotic arm motion simulation, trajectory planning, and human-machine interaction. Existing research and applications have included communicating UE4 with a robotic arm controller and driving a virtual robotic arm model through DH parameter modeling.

[0004] However, existing technologies usually use fixed models or hard-coding methods to establish robotic arm models, which cannot be flexibly adapted to different types of robotic arms, resulting in poor versatility and scalability of the system. Most systems fail to achieve stable and high-frequency communication with the robotic arm controller, resulting in delays or deviations between the virtual model and the real robotic arm movement, affecting the accuracy of the simulation. When calculating the position of the end effector, existing systems often ignore the influence of the tool coordinate system, resulting in inaccurate positioning of the work point, affecting path analysis and interactive operations. Most systems can only generate static trajectories and cannot dynamically adjust the trajectory color or style according to the output data of the robotic arm, lacking intuitive feedback on the operating status of the robotic arm. Some systems have not implemented the function of updating the joint posture step by step based on the transformation matrix, resulting in stiff and distorted movements of the virtual robotic arm, and unable to truly restore the actual motion state. Summary of the Invention

[0005] In order to solve at least one technical problem in the background technology, the present application provides a real-time simulation and visualization method of a robotic arm based on UE4.

[0006] The technical solutions adopted in this application are:

[0007] The first embodiment of the present application provides a UE4-based real-time simulation and visualization method for a robotic arm, comprising:

[0008] Build a robotic arm model based on the UE4 editor and configure parameters to generate the transformation matrix and tool coordinate system matrix;

[0009] The UE4 editor establishes communication with the robotic arm controller, and receives the current robotic arm joint angle and output data sent by the robotic arm controller at a preset frequency;

[0010] The UE4 editor calculates the current joint angle of the robotic arm by a forward kinematics algorithm based on DH parameters to obtain position information and posture information of the end effector of the robotic arm;

[0011] Based on the tool coordinate system matrix, the position information and the posture information, the transformation matrix is ​​updated to control the position and posture of the robotic arm joints of the robotic arm model, the motion trajectory of the robotic arm end effector is generated based on the position information, and the color of the motion trajectory is changed based on the output data.

[0012] According to one embodiment of the present application, the robotic arm model is established based on the UE4 editor, and parameters are configured to generate a transformation matrix and a tool coordinate system matrix, specifically:

[0013] Configure DH parameters for each joint in the UE4 blueprint editor. The DH parameters include link length a, offset d, and torsion angle α;

[0014] Generate the transformation matrix of each joint based on the input DH parameters;

[0015] Configure the tool center point in the UE4 blueprint editor, that is, the position and posture of the end effector relative to the last joint;

[0016] A tool coordinate system matrix is ​​generated according to the configuration of the tool center point.

[0017] According to one embodiment of the present application, the UE4 editor establishes communication with the robotic arm controller and receives the current robotic arm joint angle and output data sent by the robotic arm controller at a preset frequency, specifically:

[0018] A Socket server is established on the robotic arm controller side, and a Socket client is established in the UE4 editor. The Socket client initiates a connection request to the IP address and port number of the robotic arm controller to establish a stable two-way communication link.

[0019] The Socket client receives data from the robotic arm controller at a preset frequency, where the data includes angle values ​​of six joints and other output data.

[0020] According to one embodiment of the present application, the UE4 editor calculates the current robotic arm joint angle using a forward kinematics algorithm based on DH parameters to obtain position information and posture information of the robotic arm end effector, specifically:

[0021] According to the above DH parameters, the corresponding transformation matrix is ​​constructed for each joint;

[0022] Apply the FK algorithm to calculate the position and posture of the end effector of the robot arm relative to the base by multiplying the transformation matrices of all joints;

[0023] After calculating the final transformation matrix of the end-effector position and posture, it is combined with the tool coordinate system matrix to obtain the exact position and posture of the end-effector.

[0024] According to one embodiment of the present application, the updating of the transformation matrix based on the tool coordinate system matrix, the position information, and the posture information to control the position and posture of the manipulator joint of the manipulator model is specifically as follows:

[0025] Based on the position information and posture information of the end effector of the robotic arm, the transformation matrix of each robotic arm joint is updated, and the posture of the robotic arm in the virtual environment is updated by setting the transformation matrix of each robotic arm joint model; according to the currently calculated tool center point position, the specific position of the end effector of the robotic arm in space is determined.

[0026] According to one embodiment of the present application, generating the motion trajectory of the robotic arm end effector based on the position information is specifically as follows:

[0027] Use ULineBatchComponent to create line objects. Each time the robot moves, a new line segment is generated based on the new tool center point position and added to the previous line to form a continuous trajectory curve.

[0028] According to one embodiment of the present application, the changing the color of the motion trajectory based on the output data is specifically:

[0029] Define a set of rules to change the color of ULineBatchComponent according to different output data.

[0030] A computer program product comprising instructions, when executed on a device, causes the device to execute the steps in the method.

[0031] A computer-readable storage medium stores a program, which implements the steps of the method when executed by a processor.

[0032] An electronic device comprises a memory, a processor and a program stored in the memory and executable on the processor, wherein the steps in the method are implemented when the processor executes the program.

[0033] Due to the adoption of the above technical solution, the beneficial effects achieved by this application are as follows:

[0034] This application implements parametric modeling of the robotic arm structure by configuring DH parameters for each joint in the UE4 Blueprint Editor. This allows the system to flexibly adapt to a variety of robotic arm brands and models, improving the system's versatility and scalability. By generating a transformation matrix for each joint and a tool coordinate system matrix, the position of the end effector can be accurately described, improving modeling accuracy and simulation realism.

[0035] By establishing a socket client-server communication link, the stability and real-time nature of data transmission between the UE4 editor and the robotic arm controller are ensured. Data is received at a preset frequency, ensuring the virtual robotic arm's movements are highly synchronized with the real one. Joint angle and output data are also received, providing a rich data foundation for subsequent pose updates and trajectory visualization.

[0036] Using a forward kinematics algorithm to calculate joint angles in real time, the spatial position and posture of the end effector are quickly determined, providing key data support for subsequent model posture updates and trajectory drawing. This process integrates DH parameters with the tool coordinate system matrix, improving the accuracy of end effector positioning and ensuring precise operation and path verification.

[0037] By dynamically updating the transformation matrix of each joint, the virtual robot arm achieves a step-by-step actuation of each joint's posture, making the entire robot model's movements smoother and more natural, closely resembling the kinematic characteristics of a real robot arm. Combined with tool center position information, this ensures precise spatial positioning of the end effector, enhancing the system's applicability for complex tasks.

[0038] The ULineBatchComponent dynamically creates straight line segments and connects them into continuous trajectory curves, enabling a visual display of the end-effector's motion path, helping users intuitively understand the robot's working path. This continuous trajectory generation ensures the integrity and real-time nature of the path display, facilitating teaching demonstrations, path optimization, and troubleshooting.

[0039] By defining color change rules based on different output data and applying them to trajectory display, visual feedback on the robot's operating status is achieved. For example, color changes can reflect gripper pressure, temperature changes, or task status, enhancing the system's interactivity and intelligent perception capabilities. This dynamic color feedback mechanism provides an auxiliary means for remote monitoring and abnormality warnings, enhancing the system's practicality and safety. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0041] Figure 1 A flowchart of a UE4-based real-time simulation and visualization method for a robotic arm is provided in an embodiment of the present application. DETAILED DESCRIPTION

[0042] In order to more clearly illustrate the overall concept of the present application, a detailed description is given below in an illustrative manner in conjunction with the accompanying drawings.

[0043] The following description sets forth many specific details to facilitate a thorough understanding of the present application. However, the present application may also be implemented in other ways than those described herein, and therefore, the scope of protection of the present application is not limited by the specific embodiments disclosed below. It should be noted that the embodiments of the present application and the features of each embodiment may be combined with each other unless there is a conflict.

[0044] In this application, unless otherwise expressly specified and limited, a first feature "above" or "below" a second feature may be that the first and second features are in direct contact, or the first and second features are in indirect contact through an intermediate medium. In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in an appropriate manner in any one or more embodiments or examples.

[0045] Example 1

[0046] like Figure 1 As shown, the first embodiment of the present application provides a real-time simulation and visualization method for a robotic arm based on UE4, comprising:

[0047] Create a robotic arm model based on the UE4 editor and configure parameters to generate the transformation matrix and tool coordinate system matrix.

[0048] As described above, create a new project in Unreal Engine 4 (UE4) or open an existing one. Next, use the tools provided by UE4 to import a 3D model of the six-axis robotic arm. This model represents the actual physical structure of the robotic arm, including its joints and their connections.

[0049] In UE4, the Blueprint system is used to configure the parameters of the robotic arm. These parameters primarily include DH parameters (link length, offset, and twist angle), which are key to describing the geometric relationships between the robotic arm's joints. Users can set these parameter values ​​for each joint in the Blueprint Editor. This step allows the system to automatically generate a transformation matrix for each joint's position and orientation relative to the previous joint based on the input parameters.

[0050] Next, configure the Tool Coordinate System (TCP) in the Blueprint Editor. This defines the specific position and orientation of the end effector (such as a gripper or other tool) relative to the last joint. This step is crucial for accurately calculating the actual working position of the end effector of the robotic arm. By configuring the TCP, the system generates a Tool Coordinate System matrix that describes the position and orientation of the end effector relative to the last joint.

[0051] Based on the previously defined DH parameters, the system automatically generates transformation matrices for each joint. These matrices detail the positional transformation and rotation angles from one joint to the next. This means that once the angles of all joints are known, the pose of the entire robotic arm can be accurately calculated using these transformation matrices.

[0052] For example, let's say you're developing a simulation system for a manufacturing company that simulates the workings of a six-axis welding robot. First, create a new project in UE4 and import the 3D model of the welding robot. This model includes the detailed design of the robot's joints, linkages, and end effector (such as a welding gun).

[0053] In the UE4 blueprint editor, we need to set the DH parameters for each joint. For example:

[0054] For the first joint (base), assume link length a = 0 (since it is a rotating base), offset d = 250 mm (the distance from the base to the center of the first joint), and twist angle α = −90°.

[0055] The next few joints are similarly set according to the actual design parameters of the robot arm. For example, the second joint may have a=612mm, d=0, α=0°, and so on.

[0056] Based on the DH parameters set above, the system automatically generates a transformation matrix that describes the position and rotation angle of each joint relative to the previous joint. For example, for the first joint, its transformation matrix describes its specific position and orientation relative to the base; for the second joint, it describes its position and orientation relative to the first joint, and so on. In this way, we obtain a series of transformation matrices that describe the entire robotic arm structure.

[0057] Next, you need to configure the tool coordinate system—the position and orientation of the end effector relative to the last joint. In this welding robot example, assume the welding gun is mounted at the front of the last joint, slightly off-center and tilted at an angle to better contact the weld. In the Blueprint Editor, set the specific position and orientation of the welding gun relative to the last joint to generate the tool coordinate system matrix (Tool_Transform). This matrix will be used to calculate the actual working position of the end effector.

[0058] It should be noted that in specific implementation scenarios, based on the above solution, users can be allowed to input or adjust the DH parameters of each joint through a graphical interface, rather than just the preset values. This can be achieved by developing an intuitive parameter input panel, allowing users to easily modify parameters such as link length, offset, torsion angle, etc., so as to quickly adapt to different robot arm models. The function of adjusting joint parameters in real time is provided, so that the impact of different parameters on the robot arm posture can be immediately observed during the simulation process. For example, when the user changes the link length of a joint, the system should be able to immediately update the transformation matrix of the joint and reflect it in the robot arm posture in the virtual environment.

[0059] In specific implementation scenarios, based on the above solution, a mechanism can also be designed to enable users to easily change the end effector (such as switching from a welding gun to a gripper) during simulation. For each tool, its Tool_Transform matrix relative to the last joint is pre-set. When the user selects a different tool, the system automatically loads the corresponding matrix for calculation. Provide users with a tool calibration function to help them accurately set the actual installation position and orientation of the end effector. By guiding the user to complete a series of simple operations (such as moving the robotic arm to a specific point), the system can automatically calculate the correct Tool_Transform matrix, improving the accuracy and efficiency of the settings.

[0060] The UE4 editor establishes communication with the robotic arm controller and receives the current robotic arm joint angle and output data sent by the robotic arm controller at a preset frequency.

[0061] As mentioned above, a background program is run on the robotic arm controller, which is responsible for creating a socket server. This server will be used to listen for connection requests from UE4 clients. Make sure the server is configured with the correct IP address and port number so that the UE4 client can find and connect to it.

[0062] Add the necessary network programming modules or plugins to the UE4 project to support socket communication. Create a socket client instance that will attempt to connect to the socket server of the above robotic arm controller when the application starts.

[0063] Configure the client to connect using the robot controller's IP address and port number to ensure a stable two-way communication link between the two.

[0064] In the UE4 client, set a reasonable data reception frequency, such as 50 data updates per second. This depends on the specific application requirements and network bandwidth conditions. You can set a timer or use a loop mechanism to request the latest status data from the robot controller at a fixed interval.

[0065] Once the connection is established, the UE4 client begins receiving data packets from the robotic arm controller at the specified frequency. Each packet contains the angle values ​​of the six joints and other possible output data (such as sensor readings, switch status, etc.).

[0066] The UE4 client needs to parse this raw data to extract the specific angle values ​​and output information of each joint. For example, the data format may be [J1_Angle][J2_Angle]...[J6_Angle][DO], which contains the angles of six joints and additional data output (DO).

[0067] The received joint angle values, combined with the previously configured DH parameters, are used to calculate the current robotic arm's posture using a forward kinematics (FK) algorithm. This means determining the position and orientation of each joint based on the joint angles, ultimately determining the exact position and posture of the end effector.

[0068] Based on the calculation results, the posture of the robotic arm model is dynamically adjusted in the virtual environment to accurately reflect the state of the actual robotic arm.

[0069] For example, a background program runs on the robotic arm controller, which creates a socket server. This server listens for connection requests from UE4 clients. For example, the robotic arm controller is configured to listen on a specific port (such as 54321) on the IP address 192.168.1.100, waiting for connections from UE4 clients.

[0070] In the UE4 project, the necessary network modules or plugins are added to support socket communication. A socket client instance is created, which attempts to connect to the socket server of the above robotic arm controller.

[0071] When the UE4 application starts, it automatically attempts to connect to port 54321 of the IP address 192.168.1.100 to establish a stable communication link with the robotic arm controller.

[0072] A timer is set in the UE4 client to fire 10 times per second (once every 0.1 seconds). This means that UE4 will request the latest state data from the robot controller every 0.1 seconds. This frequency can be adjusted based on actual needs to balance the timeliness of data updates with network bandwidth consumption.

[0073] Once a stable communication link is established, the UE4 client begins receiving data packets from the robotic arm controller at a set frequency. Each packet contains the angle values ​​of the six joints and other output data (such as temperature and pressure).

[0074] Assume that the received data format is as follows: "J1=30°; J2=45°; J3=60°; J4=75°; J5=90°; J6=105°; Temp=45; Pressure=10". Here, "J1" to "J6" represent the angle values ​​of the six joints (in degrees), while "Temp" and "Pressure" are additional sensor readings.

[0075] After receiving this data, the UE4 client first parses it and extracts the angle values ​​and other output data of each joint. For example, from the data above, we can know that the angle of the first joint is 30 degrees, the angle of the second joint is 45 degrees, and so on.

[0076] Using these joint angle values, combined with the previously configured DH parameters, the current posture of the robotic arm is calculated using a forward kinematics algorithm. This means determining the position and orientation of each joint, and ultimately the exact position and posture of the end effector.

[0077] Based on the calculation results, the robot arm model's posture is dynamically adjusted in the UE4 virtual environment. For example, if the received data indicates that the robot arm should raise the welding gun, the position and orientation of the robot arm model are adjusted accordingly in the virtual environment to accurately reflect the actual robot arm's movements.

[0078] For additional output data, such as temperature and pressure, you can display this information on the UI or change certain properties based on this data. For example, when the temperature exceeds a certain threshold, you can alert the user by changing the color of the track.

[0079] It should be noted that in specific implementation scenarios, threshold conditions can be set based on the above solution. When certain key parameters (such as temperature and pressure) exceed the safe range, an alarm mechanism will be triggered. Users can be notified of potential problems through sound alarms, flashing colors, or pop-up warning boxes.

[0080] In specific implementation scenarios, the above solution can also be used to automatically record all received data. These records can be saved as text files or database entries for subsequent review and analysis. For example, detailed information such as joint angle changes and sensor readings during each run can be recorded. Tools are provided to allow users to review and analyze past run data. This can help identify long-term trends, optimize workflows, or diagnose the cause of a fault. For example, users can select data from a specific time period for playback to observe changes in the robot's motion patterns and related parameters.

[0081] In specific implementation scenarios, building on the above solution, users can also remotely control the robot arm's operation over the network. Designing an intuitive interactive interface within the UE4 environment allows users to adjust the robot arm's posture or perform tasks even when not on-site. Building a multi-user collaborative environment allows operators and technicians in different locations to simultaneously access and control the same virtual robot arm model. This facilitates collaborative problem solving or collaborative planning of complex operational steps.

[0082] The UE4 editor calculates the current robotic arm joint angles using a forward kinematics algorithm based on DH parameters to obtain position information and posture information of the robotic arm end effector.

[0083] As mentioned above, first, we need to define the DH parameters for each joint of the robot arm. These parameters describe the specific position and orientation of each joint relative to its previous joint, including:

[0084] Link length (a): the distance between two consecutive joint axes.

[0085] Offset (d): The distance moved along the previous joint axis.

[0086] Twist angle (α): The angle between two consecutive joint axes.

[0087] These parameters are set according to the design specifications of the robot arm and are fixed for a specific model of robot arm.

[0088] Receive the current angle data for each joint from the robot controller. For example, suppose the received data indicates that the angle of the first joint is 30 degrees, the angle of the second joint is 45 degrees, and so on, up to the sixth joint. These angles represent the actual rotation state of each joint in the robot.

[0089] Next, the forward kinematics algorithm is used to combine the DH parameters and the received joint angle data to calculate the position and posture of the robot arm end effector. The specific process is as follows:

[0090] Generate transformation matrix: For each joint, generate a transformation matrix based on its DH parameters and current angle. This matrix describes the position transformation and rotation of the joint relative to the previous joint.

[0091] Cumulative transformation: Apply the transformation matrix of each joint in turn, starting from the base and working your way up to the end effector. This means multiplying the transformation matrices of all joints together to get the final position and pose of the entire robotic arm end effector relative to the base.

[0092] Because the end effector may be mounted at different positions or orientations relative to the last joint, the tool coordinate system (TCP) must also be considered. The tool coordinate system matrix (Tool_Transform) describes the position and orientation of the end effector relative to the last joint. By multiplying the previously calculated transformation with Tool_Transform, the actual position and orientation of the end effector can be accurately determined.

[0093] After completing the above steps, we obtain the exact position and attitude information of the robot arm's end effector. This includes the coordinates of the end effector in three-dimensional space (x, y, z) and its orientation relative to the global coordinate system (usually expressed as Euler angles or quaternions).

[0094] For example, imagine you're using a six-axis industrial robot for handling. This robot is installed in an automated production line, moving materials from one location to another. You need to accurately simulate the robot's movements in the UE4 environment to train operators or optimize their workflow.

[0095] First, set the DH parameters for each joint of the six-axis robot arm. These parameters are determined according to the design specifications of the robot arm and are used to describe the specific position and orientation of each joint relative to the previous joint. For example:

[0096] For the first joint (the base), assume link length a = 0 (since it is a rotating base), offset d = 250 mm (the distance from the base to the center of the first joint), and twist angle α = −90°. For the second joint, assume link length a = 612 mm, offset d = 0, and twist angle α = 0°.

[0097] The remaining joints are similarly set according to the actual design parameters.

[0098] Assume that the data received from the robotic arm controller indicates that the current angles of each joint are as follows:

[0099] Joint Angle: J1 = 30°. Joint Angle: J2 = 45°. Joint Angle: J3 = 60°. Joint Angle: J4 = 75°. Joint Angle: J5 = 90°. Joint Angle: J6 = 105°.

[0100] These angles represent the actual rotational states of the various joints of the robotic arm.

[0101] Next, the forward kinematics algorithm is used to combine the above DH parameters and the received joint angle data to calculate the position and posture of the end effector of the robotic arm:

[0102] Generate transformation matrix: For each joint, generate a transformation matrix based on its DH parameters and current angle. This matrix describes the position transformation and rotation of the joint relative to the previous joint.

[0103] Cumulative transformation: Apply the transformation matrix of each joint in turn, starting from the base and working your way up to the end effector. This means multiplying the transformation matrices of all joints together to get the final position and pose of the entire robotic arm end effector relative to the base.

[0104] Given that the end effector may be mounted at different positions or orientations of the last joint, the tool coordinate system (TCP) also needs to be considered. Let's assume our end effector is a gripper with a specific offset and orientation relative to the sixth joint. By multiplying the previously calculated transformation with the Tool_Transform, we can accurately determine the actual position and orientation of the gripper.

[0105] After completing the above steps, we obtain the exact position and orientation of the robotic arm's end effector. For example, in this transport task, calculations show that the gripper's current position is (x=1200mm, y=800mm, z=500mm), and that it is oriented toward the next workstation on the production line. This gives us the gripper's exact position in space and its orientation.

[0106] It should be noted that in specific implementation scenarios, building on the above solution, after calculating the position and posture of the robot arm's end effector, it is possible to further develop functionality to predict its future path. For example, based on a preset mission objective or a user-entered target point, the system can dynamically generate an optimal path from the current position to the target location and display this path in real time within a virtual environment. Incorporating environmental models (such as workstations and obstacles), collision risks that may be encountered during the robot arm's movement can be detected in real time. Once a potential collision is detected, the system can automatically adjust the path or prompt the user to make manual adjustments to ensure operational safety.

[0107] In specific implementation scenarios, based on the above solution, a mechanism can also be provided to allow users to dynamically adjust certain DH parameters or other related parameters (such as the tool coordinate system) during runtime. This allows users to fine-tune the behavior of the robot arm according to actual conditions without having to restart the entire simulation process. For example, if the angle of a joint is found to be deviated from the expected value during actual operation, the user can directly modify the relevant parameters of the joint on the interface and immediately see the effect. Provide users with a simple calibration tool to help them accurately set or calibrate the DH parameters and tool coordinate system of the robot arm. For example, guiding users through a series of simple operations (such as moving the robot arm to a specific reference point) to automatically calculate the correct parameter values, thereby improving the accuracy and efficiency of the settings.

[0108] Based on the tool coordinate system matrix, the position information and the posture information, the transformation matrix is ​​updated to control the position and posture of the robotic arm joints of the robotic arm model, the motion trajectory of the robotic arm end effector is generated based on the position information, and the color of the motion trajectory is changed based on the output data.

[0109] As mentioned above, the position and orientation of the end effector relative to the base are first calculated using the forward kinematics (FK) algorithm. This process combines the DH parameters with the current angle values ​​of each joint and takes into account the tool coordinate system matrix (Tool_Transform) to obtain the exact position and orientation of the end effector.

[0110] Based on the above calculation results, the system generates or updates the transformation matrix (Transform) for each robotic arm joint. These transformation matrices contain the specific position and rotation angle of each joint relative to its previous joint.

[0111] Then, by setting the Transform of each robotic arm joint model, the system updates the pose of the robotic arm in the virtual environment. This means that the robotic arm model in the virtual environment will be adjusted accordingly based on the actual robotic arm's state, ensuring that the virtual model is consistent with the real-world robotic arm. For example, if the calculation results indicate that the first joint needs to rotate 30 degrees clockwise, the angle of that joint will be adjusted accordingly in the virtual environment.

[0112] Based on the currently calculated position of the tool center point (TCP), the system can determine the specific position of the robot arm's end effector in space. For example, at a certain moment, the coordinates of the end effector in three-dimensional space are calculated to be (x=100mm, y=200mm, z=300mm).

[0113] To visualize the path of the robotic arm's end effector, UE4 uses a ULineBatchComponent to create line objects. Each time the robotic arm moves, a new line segment is generated based on the new TCP position and added to the previous line, forming a continuous trajectory curve. For example, as the robotic arm moves from its starting point to its end point, the system records the position of the end effector at each time point and connects these points to form a smooth curve.

[0114] The data received from the robotic arm controller includes not only joint angles but also other output data (such as sensor readings and switch states), often referred to as DO (Data Output). For example, suppose a temperature sensor is installed on the robotic arm, which can provide real-time feedback on the current operating temperature.

[0115] Based on different DO data, you can define a set of rules to change the color of the ULineBatchComponent. For example, when the temperature is below 50 degrees Celsius, the track will be blue; when the temperature is between 50 and 100 degrees Celsius, the track will turn yellow; when the temperature exceeds 100 degrees Celsius, the track will turn red. The purpose of this is to provide an intuitive way to display the different working states or operating modes of the robot arm.

[0116] This approach not only enhances the user’s understanding and ability to monitor the robot’s operation, but also helps quickly identify potential problems. For example, if the track suddenly turns red, it may mean that the robot is overheating and requires immediate action to prevent damage.

[0117] For example, imagine you're using a six-axis industrial robot to assemble electronic components. Its task is to pick components from trays and precisely place them on circuit boards. You need to simulate the robot's movements in UE4 to train operators or optimize their workflow.

[0118] After receiving the current joint angle data from the robot controller, the forward kinematics algorithm combines the DH parameters and the tool coordinate system matrix (Tool_Transform) to calculate the exact position and posture of the robot's end effector (such as the gripper). For example, the calculation results show that the gripper is located at the coordinates in three-dimensional space (x=150mm, y=200mm, z=300mm) and is oriented vertically downward.

[0119] Based on these calculations, the system generates or updates a transformation matrix for each joint in the robotic arm. These transformation matrices describe the specific position and rotation angle of each joint relative to the previous joint. For example, if the calculations indicate that the first joint needs to rotate 45 degrees clockwise, the angle of that joint is adjusted accordingly in the virtual environment.

[0120] Then, by setting the Transform of each joint model, the system updates the pose of the robot in the virtual environment. This means that the robot model in the virtual environment will adjust accordingly based on the actual robot's state, ensuring that the virtual model is consistent with the real-world robot. In this example, the virtual robot will adjust the positions of all joints according to the calculated results, allowing the gripper to move accurately to the desired position.

[0121] As the robot moves from its starting point to its end point, the system records the position of the end effector at each time point. For example, suppose the robot starts at an initial position (x=0mm, y=0mm, z=0mm), passes through a series of intermediate points, and finally reaches the target position (x=150mm, y=200mm, z=300mm).

[0122] Use a ULineBatchComponent to create line objects in UE4. Each time the arm moves, a new line segment is generated based on the new TCP position and added to the previous line, forming a continuous trajectory curve. In this example, as the arm moves, a smooth curve is dynamically generated from the starting point to the end point, representing the actual movement path of the gripper.

[0123] The data received from the robotic arm controller includes not only joint angles but also other output data such as sensor readings, switch states, etc. For example, suppose the robotic arm is equipped with a force sensor that provides real-time feedback on the pressure currently applied to the gripper.

[0124] Based on different DO data, you can define a set of rules to change the color of the ULineBatchComponent. For example, when the gripper pressure is less than 5 Newtons, the track will be green; when the pressure is between 5 and 10 Newtons, the track will turn yellow; and when it exceeds 10 Newtons, the track will turn red. This provides an intuitive way to display the different working states or operating modes of the robot arm.

[0125] In this example, assuming the gripper gradually increases pressure as it approaches the target position, you can see the track color change from green to yellow and finally to red, indicating that the gripper has successfully grasped the component and applied enough pressure to ensure that the component does not fall.

[0126] It should be noted that in specific implementation scenarios, it is also possible to add a path optimization algorithm on the basis of the above solution, so that the system can dynamically adjust the movement path of the robot arm according to real-time feedback data. For example, if an obstacle or a new target point is detected, the system can automatically recalculate an optimal path and instantly update the robot arm model path in the virtual environment. Provide users with an intuitive interface that allows them to adjust DH parameters, tool coordinate system parameters, and other related parameters at runtime. This can help users quickly respond to problems encountered in actual operations, such as robot arm accuracy deviation, without having to restart the entire simulation process.

[0127] In specific implementation scenarios, the above solution can also be used to implement detailed logging of each solution result and all relevant parameters (including position, posture, output data, etc.). This logging should not only be limited to the final result but also include data from intermediate steps to facilitate subsequent analysis and troubleshooting. Dedicated tools should be provided to allow users to review and analyze past operation data. For example, users can select data from a specific time period for playback to observe the robot arm's motion patterns and the changing trends of its related parameters, thereby identifying potential issues or optimizing operational processes.

[0128] According to one embodiment of the present application, the robotic arm model is established based on the UE4 editor, and parameters are configured to generate a transformation matrix and a tool coordinate system matrix, specifically:

[0129] Configure DH parameters for each joint in the UE4 blueprint editor. The DH parameters include link length a, offset d, and torsion angle α;

[0130] Generate the transformation matrix of each joint based on the input DH parameters;

[0131] Configure the tool center point in the UE4 blueprint editor, that is, the position and posture of the end effector relative to the last joint;

[0132] A tool coordinate system matrix is ​​generated according to the configuration of the tool center point.

[0133] As mentioned above, in the UE4 Blueprint Editor, set the DH parameters for each joint of the robotic arm. These parameters are used to describe the basic geometric information of the robotic arm structure, mainly including the following three contents:

[0134] Link length a: represents the shortest distance between the rotation axes of two adjacent joints.

[0135] Offset d: represents the distance from the origin of the current coordinate system to the origin of the next coordinate system along the direction of the previous joint rotation axis.

[0136] Twist angle α (alpha): represents the angle between the rotation axes of two adjacent joints.

[0137] These parameters are set based on the physical structure of the actual robot arm. Different robot arm models have different DH parameters. In Blueprint, users can enter or adjust these parameters through a graphical interface, allowing for flexible adaptation to a variety of robot arm models.

[0138] After completing the DH parameter configuration, the system automatically generates a transformation matrix for each joint based on these parameters. This transformation matrix describes the position and posture changes of the current joint relative to its previous joint, that is, the transformation process of "moving and rotating from the previous joint to the current joint".

[0139] These transformation matrices form the foundation of the entire kinematic calculation. They are applied sequentially, starting from the base and passing down to the end effector, to construct the spatial structure of the entire robotic arm. In other words, the position and posture of each joint depends on the previous one, and this hierarchical relationship is expressed through the transformation matrix.

[0140] Next, in the Blueprint Editor, you need to define the Tool Center Point (TCP), which is the position and orientation of the end effector relative to the last joint. For example, if the end effector is a gripper, it might be mounted at the tip of the sixth joint with a certain offset and tilt angle.

[0141] This step ensures that the simulation system can accurately locate the "working point" of the robot arm, that is, the position where it actually works (such as welding point, gripping point, etc.). Users can set this offset and direction in an intuitive way in Blueprint without having to write any code manually.

[0142] Finally, the system generates a tool coordinate system matrix (Tool_Transform Matrix) based on the configuration of the tool center point. This matrix describes the specific transformation relationship of the end effector relative to the last joint.

[0143] In the subsequent forward kinematics solution process, this tool coordinate system matrix will be used to multiply the transformation matrices of all previous joints to ultimately determine the position and posture of the end effector in the global space.

[0144] According to one embodiment of the present application, the UE4 editor establishes communication with the robotic arm controller and receives the current robotic arm joint angle and output data sent by the robotic arm controller at a preset frequency, specifically:

[0145] A Socket server is established on the robotic arm controller side, and a Socket client is established in the UE4 editor. The Socket client initiates a connection request to the IP address and port number of the robotic arm controller to establish a stable two-way communication link.

[0146] The Socket client receives data from the robotic arm controller at a preset frequency, where the data includes angle values ​​of six joints and other output data.

[0147] As mentioned above, first, a background program is run on the robotic arm controller device. This program is responsible for creating and starting a socket server. The role of this server is to listen for connection requests from external clients (i.e., the UE4 editor) and continuously send the client the real-time status data of the robotic arm.

[0148] This socket server will bind a fixed IP address and a specified port number as the entry point for its network communication. Once successfully started, it can wait for connection requests from the UE4 editor.

[0149] At the same time, a socket client module is developed or integrated into the Unreal Engine 4 (UE4) editor. This client has the ability to actively initiate a network connection.

[0150] When the user starts the simulation system in the UE4 editor, the socket client automatically attempts to connect to the IP address and corresponding port number of the robot controller. Once the connection is successful, a stable two-way communication link is established between the two parties, and data exchange can begin.

[0151] This two-way communication mechanism not only allows the UE4 to receive data from the robotic arm, but also supports sending control instructions (such as pause, restart, etc.) from the UE4 to the robotic arm controller when necessary.

[0152] After the communication link is established, the Socket client in the UE4 editor will receive data packets from the robotic arm controller at a set fixed frequency (for example, 10 times or 50 times per second).

[0153] The angle values ​​of the six joints represent the actual rotation angle of each joint, which is used to drive the posture update of the virtual robotic arm model in UE4.

[0154] Other output data (DO): This may include the robot's status signals, sensor readings, digital output signals (e.g., whether the gripper is closed), temperature, pressure, and other additional information. This data can be used to enhance the realism and functionality of the simulation, such as track color changes, status prompts, and alarm triggering.

[0155] The data reception frequency can be configured according to actual needs. The higher the frequency, the closer the virtual model's movements are to the real robotic arm, but it will also bring higher network load and computing overhead.

[0156] According to one embodiment of the present application, the UE4 editor calculates the current robotic arm joint angle using a forward kinematics algorithm based on DH parameters to obtain position information and posture information of the robotic arm end effector, specifically:

[0157] According to the above DH parameters, the corresponding transformation matrix is ​​constructed for each joint;

[0158] Apply the FK algorithm to calculate the position and posture of the end effector of the robot arm relative to the base by multiplying the transformation matrices of all joints;

[0159] After calculating the final transformation matrix of the end-effector position and posture, it is combined with the tool coordinate system matrix to obtain the exact position and posture of the end-effector.

[0160] As mentioned above, the DH parameters of each joint of the robotic arm (including link length a, offset d, and torsion angle α) have been pre-configured in the UE4 editor. These parameters are the basic geometric information that describes the structure of the robotic arm.

[0161] Based on these parameters, the system generates a corresponding transformation matrix for each joint of the robotic arm. This transformation matrix expresses the spatial transformation relationship of each joint relative to the previous joint, including positional movement and directional rotation. In other words, each joint's transformation matrix describes the changes from the previous joint to the current joint.

[0162] Next, the system uses the Forward Kinematics (FK) algorithm to superimpose the transformation matrices of each joint in turn, that is, multiply them in turn.

[0163] Starting from the base of the robotic arm, the transformation matrix for each joint is applied step by step, ultimately resulting in the overall transformation of the end effector relative to the base. This final transformation matrix contains the end effector's specific position in 3D space (x, y, z coordinates) and its pose (i.e., orientation, typically expressed as Euler angles or quaternions).

[0164] In other words, in this way, the system can accurately know the actual working point and orientation of the robot arm's end effector (such as grippers, welding guns, etc.) in the virtual space.

[0165] In order to further improve the accuracy, after obtaining the position and posture of the end effector, the system will also combine it with the previously set tool coordinate system matrix (Tool_Transform).

[0166] The tool coordinate system matrix describes the actual installation position and orientation of the end effector relative to the last joint. For example, if the gripper is not installed exactly at the center point of the sixth axis, but has a certain offset and tilt, then this matrix will record these deviations.

[0167] By multiplying the result calculated by FK with the tool coordinate system matrix, the exact position and posture of the end effector in the global coordinate system can be obtained, ensuring that the movement of the virtual model is completely consistent with that of the real robotic arm.

[0168] According to one embodiment of the present application, the updating of the transformation matrix based on the tool coordinate system matrix, the position information, and the posture information to control the position and posture of the manipulator joint of the manipulator model is specifically as follows:

[0169] Based on the position information and posture information of the end effector of the robotic arm, the transformation matrix of each robotic arm joint is updated, and the posture of the robotic arm in the virtual environment is updated by setting the transformation matrix of each robotic arm joint model; according to the currently calculated tool center point position, the specific position of the end effector of the robotic arm in space is determined.

[0170] As mentioned above, first, the system will reversely deduce the spatial transformation state that each intermediate joint should be in based on the previously calculated position and posture information of the robot end effector, combined with the set DH parameters and tool coordinate system matrix.

[0171] This process does not directly use the position of the end effector to "move" the model. Instead, it uses the kinematic link relationship to gradually determine the correct position of each joint in space. In other words, the system regenerates a new transformation matrix for each joint to describe its new position and orientation relative to the previous joint.

[0172] In the UE4 editor, each joint of the robotic arm is a model component with independent space coordinates. Once the new transformation matrix of each joint is obtained, the system will apply these matrices to the corresponding joint model.

[0173] This step is equivalent to telling the UE4 engine: "Now, please move this joint here and rotate it to this angle." In this way, the posture of the entire robotic arm model will change accordingly and remain consistent with the real robotic arm.

[0174] For example, if the angle of a joint changes, its transformation matrix is ​​updated accordingly, and the model can perform corresponding actions in the virtual environment, such as raising its arm or rotating its wrist.

[0175] After the update is complete, the system also pays special attention to the position of the tool center point (TCP), which represents the actual working position of the end effector, such as the contact point of the gripper grasping the object.

[0176] The system calculates the exact three-dimensional position of the tool center point in the global coordinate system based on the latest transformation matrix chain and tool coordinate system matrix. This ensures that not only the overall posture of the robot arm is correctly restored, but also the working point at the end can be accurately positioned.

[0177] This position information can also be used for subsequent functions, such as drawing the motion trajectory of the end effector, determining whether the target point has been reached, triggering interactive events, etc.

[0178] According to one embodiment of the present application, generating the motion trajectory of the robotic arm end effector based on the position information is specifically as follows:

[0179] Use ULineBatchComponent to create line objects. Each time the robot moves, a new line segment is generated based on the new tool center point position and added to the previous line to form a continuous trajectory curve.

[0180] As mentioned above, in Unreal Engine 4 (UE4), ULineBatchComponent is a component used to efficiently render lines and shapes. It allows developers to dynamically add and manage large amounts of line data with low performance overhead.

[0181] Every time the robotic arm moves, the system uses this component to create new line segments that represent the path the end effector takes from one position to the next.

[0182] After each movement of the robotic arm, the system calculates the current position of the tool center point (TCP). The TCP is the actual working point on the end effector, such as the actual gripping point of a gripper or the welding point of a welding gun.

[0183] Based on the current and previously recorded TCP positions, the system creates a new straight line segment between them. This straight line segment represents the direct path of the end effector between these two positions.

[0184] Newly generated line segments are automatically added to the collection of pre-existing lines, forming a continuous trajectory curve. In this way, as the robot arm moves, the entire motion path of its end effector gradually emerges.

[0185] The system receives data from the robot controller at every time interval (e.g., 50 times per second) and updates the trajectory based on the latest TCP position. This means the trajectory reflects the actual movement of the robot in real time, providing immediate visual feedback.

[0186] If the robot arm changes direction or speed, the trajectory will adjust accordingly. This allows users to clearly see any subtle changes, helping them better understand and optimize the robot arm's operation process.

[0187] According to one embodiment of the present application, the changing the color of the motion trajectory based on the output data is specifically:

[0188] Define a set of rules to change the color of ULineBatchComponent according to different output data.

[0189] As mentioned above, during the operation of the robotic arm, in addition to joint angles and end-effector position information, other types of output data (DO) are also generated, such as sensor readings (such as temperature and pressure), status signals (such as whether the gripper is closed), velocity, acceleration, etc. This data reflects the current operating status or environmental conditions of the robotic arm.

[0190] To help operators more intuitively understand the status of the robotic arm, the system can visualize the output data by changing the color of the motion trajectory. Specifically, a set of rules is defined based on different output data to dynamically adjust the color of the trajectory.

[0191] First, we need to determine which output data is most important for monitoring the working status of the robotic arm. For example, suppose we are interested in the force sensor reading on the robotic arm's end effector, which can help us determine whether the force used to grasp the object is appropriate.

[0192] For the selected output data, set several key threshold ranges. Each range corresponds to a specific color. For example:

[0193] When the force sensor reading is below a low threshold (such as 5 Newtons), the trace appears green, indicating that the force is low and safe.

[0194] When the reading is between two thresholds (such as 5 to 10 Newtons), the trace turns yellow, alerting the user that the force has increased but is still within the normal range.

[0195] If the reading exceeds a high threshold (e.g. greater than 10 Newtons), the trace turns red to warn of a possible overload risk.

[0196] Create a simple mapping rule table to map different ranges of output data values ​​to corresponding colors. This rule can be linear (for example, the color changes from green to red as the value increases) or non-linear (flexibly set according to actual needs).

[0197] Whenever new output data is received, the system automatically calculates the corresponding track color according to the rules defined above and updates the line color in ULineBatchComponent.

[0198] If the output data changes frequently (e.g., multiple updates per second), the system needs to be able to respond quickly and adjust the track color instantly to ensure that the color always accurately reflects the current output data state.

[0199] To avoid visual discontinuity caused by sudden color changes, smooth transitions can be implemented between adjacent segments. For example, when the force sensor reading suddenly increases from 9 Newtons to 11 Newtons, the track color doesn’t instantly jump from yellow to red. Instead, the transition gradually creates a smoother visual effect.

[0200] A computer program product comprising instructions, when executed on a device, causes the device to execute the steps in the method.

[0201] A computer-readable storage medium stores a program, which implements the steps of the method when executed by a processor.

[0202] An electronic device comprises a memory, a processor and a program stored in the memory and executable on the processor, wherein the steps in the method are implemented when the processor executes the program.

[0203] Anything not described in this application can be achieved by adopting or drawing on existing technologies.

[0204] The various embodiments in this specification are described in a progressive manner, and the same or similar parts between the various embodiments can be referred to each other. Each embodiment focuses on the differences from other embodiments.

[0205] The foregoing is merely an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various modifications and variations. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application should all be included in the protection scope of the present application.

Claims

1. A real-time simulation and visualization method for a robotic arm based on UE4, characterized in that: include: Build a robotic arm model based on the UE4 editor and configure parameters to generate a transformation matrix and a tool coordinate system matrix. Specifically, configure DH parameters for each joint in the UE4 blueprint editor. The DH parameters include the link length a, offset d, and torsion angle α. Generate the transformation matrix of each joint based on the input DH parameters; Configure the tool center point in the UE4 blueprint editor, that is, the position and posture of the end effector relative to the last joint; Generate a tool coordinate system matrix based on the configuration of the tool center point; The UE4 editor establishes communication with the robotic arm controller, and receives the current robotic arm joint angle and output data sent by the robotic arm controller at a preset frequency; The UE4 editor solves the current robotic arm joint angles using a forward kinematics algorithm based on DH parameters to obtain position information and posture information of the robotic arm end effector, specifically by constructing a corresponding transformation matrix for each joint based on the DH parameters; Apply the FK algorithm to calculate the position and posture of the end effector of the robot arm relative to the base by multiplying the transformation matrices of all joints; After calculating the final transformation matrix of the end effector position and posture, it is combined with the tool coordinate system matrix to obtain the exact position and posture of the end effector; The transformation matrix is ​​updated based on the tool coordinate system matrix, the position information, and the posture information to control the position and posture of the manipulator joints of the manipulator model, specifically: based on the position information and posture information of the manipulator end effector, the transformation matrix of each manipulator joint is updated, and by setting the transformation matrix of each manipulator joint model, the posture of the manipulator in the virtual environment is updated; Determine the specific position of the robot arm end effector in space based on the currently calculated tool center point position; The motion trajectory of the end effector of the robotic arm is generated based on the position information, and the color of the motion trajectory is changed based on the output data. Specifically, a set of rules are defined to change the color of ULineBatchComponent according to different output data. The output data includes: sensor readings and switch status.

2. The method according to claim 1, characterized in that The UE4 editor establishes communication with the robotic arm controller and receives the current robotic arm joint angle and output data sent by the robotic arm controller at a preset frequency, specifically: A Socket server is established on the robotic arm controller side, and a Socket client is established in the UE4 editor. The Socket client initiates a connection request to the IP address and port number of the robotic arm controller to establish a stable two-way communication link. The Socket client receives data from the robotic arm controller at a preset frequency, where the data includes angle values ​​of six joints and other output data.

3. The method according to claim 1, characterized in that The motion trajectory of the end effector of the robotic arm is generated based on the position information, specifically: Use ULineBatchComponent to create line objects. Each time the robot moves, a new line segment is generated based on the new tool center point position and added to the previous line to form a continuous trajectory curve.

4. A computer program product comprising instructions, which, when executed on a device, is characterized in that The device is enabled to execute the steps in the method according to any one of claims 1 to 3.

5. A computer-readable storage medium having a program stored thereon, characterized in that: When the program is executed by a processor, the steps in the method according to any one of claims 1 to 3 are implemented.

6. An electronic device comprising a memory, a processor, and a program stored in the memory and executable on the processor, wherein: When the processor executes the program, the steps in the method according to any one of claims 1 to 3 are implemented.

Images