Mechanical arm real-time simulation and visualization method based on UE4

Abstract

The invention provides a UE4-based mechanical arm real-time simulation and visualization method, and belongs to the technical field of virtual simulation and robots, and the method comprises the steps: building a mechanical arm model based on a UE4 editor, and configuring parameters to generate a conversion matrix and a tool coordinate system matrix; the UE4 editor establishes communication with a mechanical arm controller, and receives a current mechanical arm joint angle and output data sent by the mechanical arm controller according to a preset frequency; the UE4 editor resolves the current mechanical arm joint angle through a forward kinematics algorithm based on D-H parameters, and position information and attitude information of a mechanical arm end effector are obtained; and the conversion matrix is updated based on the tool coordinate system matrix, the position information and the attitude information so as to control the position and the attitude of a mechanical arm joint of the mechanical arm model, the motion trail of a mechanical arm end effector is generated based on the position information, and the color of the motion trail is changed based on the output data.

Description

Technical Field

[0001] This application belongs to the field of virtual simulation and robotics technology, and specifically relates to a real-time simulation and visualization method for robotic arms based on UE4. Background Art

[0002] With the wide application of industrial robots in intelligent manufacturing, automated assembly, and flexible production, the visual simulation system of robotic arms has become an important tool for research and development, teaching, debugging, and remote monitoring. Traditional robotic arm development and debugging mostly rely on physical prototypes, which not only have high costs and long cycles but also are difficult to reflect the operating state of the robotic arm in real time, limiting their application in complex tasks.

[0003] In recent years, virtual simulation technology has gradually been introduced into the field of robotics. In particular, three-dimensional visualization platforms built based on high-performance graphics engines such as Unreal Engine 4 (UE4) provide good support for the motion simulation, trajectory planning, and human-machine interaction of robotic arms. In the prior art, there have been studies and applications on communicating UE4 with robotic arm controllers and driving virtual robotic arm models through D-H parameter modeling.

[0004] However, the prior art usually establishes robotic arm models using fixed models or hard-coded methods, which cannot flexibly adapt to different models of robotic arms, resulting in poor versatility and scalability of the system. Most systems fail to achieve stable high-frequency communication with robotic arm controllers, resulting in delays or deviations between the virtual model and the actions of the real robotic arm, affecting the accuracy of the simulation. When calculating the position of the end effector in existing systems, the influence of the tool coordinate system is often ignored, resulting in inaccurate positioning of the working point and affecting path analysis and interactive operations. Most systems can only generate static trajectories and cannot dynamically adjust the trajectory color or style according to the output data of the robotic arm, lacking intuitive feedback on the operating state of the robotic arm. Some systems do not implement the function of gradually updating joint postures based on transformation matrices, resulting in rigid and distorted actions of the virtual robotic arm and being unable to truly restore the actual motion state. Summary of the Invention

[0005] To solve at least one aspect of the technical problems in the background art, this application provides a real-time simulation and visualization method for robotic arms based on UE4.

[0006] The technical solution adopted in this application is as follows: The first aspect embodiment of this application provides a real-time simulation and visualization method for robotic arms based on UE4, including: Establishing a robotic arm model based on the UE4 editor and configuring parameters to generate a transformation matrix and a tool coordinate system matrix; The UE4 editor establishes communication with the robotic arm controller and receives the current robotic arm joint angles and output data sent by the robotic arm controller at a preset frequency; The UE4 editor resolves the current robotic arm joint angles through a forward kinematics algorithm based on D-H parameters to obtain the position information and attitude information of the end effector of the robotic arm; Based on the tool coordinate system matrix, the position information, and the attitude information, update the transformation matrix to control the position and attitude of the robotic arm joints of the robotic arm model, generate the motion trajectory of the end effector of the robotic arm based on the position information, and change the color of the motion trajectory based on the output data.

[0007] According to an embodiment of the present application, establishing a robotic arm model based on the UE4 editor and configuring parameters to generate a transformation matrix and a tool coordinate system matrix specifically includes: Configure D-H parameters for each joint in the UE4 blueprint editor, where the D-H parameters include link length a, offset d, and twist angle α; Generate the transformation matrix for each joint based on the input D-H parameters; Configure the tool center point in the UE4 blueprint editor, that is, the position and attitude of the end effector relative to the last joint; Generate a tool coordinate system matrix according to the configuration of the tool center point.

[0008] According to an embodiment of the present application, the UE4 editor establishes communication with the robotic arm controller and receives the current robotic arm joint angles and output data sent by the robotic arm controller at a preset frequency, specifically including: Establish a Socket server at the robotic arm controller end, establish a Socket client in the UE4 editor, and the Socket client initiates a connection request to the IP address and port number of the robotic arm controller to establish a stable two-way communication link; The Socket client receives the data of the robotic arm controller at a preset frequency, and the data includes the angle values of six joints and other output data.

[0009] According to an embodiment of the present application, the UE4 editor resolves the current robotic arm joint angles through a forward kinematics algorithm based on D-H parameters to obtain the position information and attitude information of the end effector of the robotic arm, specifically including: Construct a corresponding transformation matrix for each joint according to the above D-H parameters; Apply the FK algorithm to calculate the position and attitude of the end effector of the robotic arm relative to the base by multiplying the transformation matrices of all joints; After calculating the final transformation matrix of the end effector's position and orientation, it is combined with the tool coordinate system matrix to obtain the exact position and orientation of the end effector.

[0010] According to an embodiment of the present application, updating the transformation matrix based on the tool coordinate system matrix, the position information, and the orientation information to control the position and orientation of the robotic arm joints of the robotic arm model specifically includes: Based on the position information and orientation information of the end effector of the robotic arm, update the transformation matrix of each robotic arm joint. By setting the transformation matrix of each robotic arm joint model, update the orientation of the robotic arm in the virtual environment; determine the specific position of the end effector of the robotic arm in space according to the currently calculated position of the tool center point.

[0011] According to an embodiment of the present application, generating the motion trajectory of the end effector of the robotic arm based on the position information specifically includes: Use ULineBatchComponent to create a straight line object. Each time the robotic arm moves, generate a new straight line segment based on the new position of the tool center point and add it to the previous line to form a continuous trajectory curve.

[0012] According to an embodiment of the present application, changing the color of the motion trajectory based on the output data specifically includes: Define a set of rules to change the color of ULineBatchComponent according to different output data.

[0013] A computer program product containing instructions, when running on a device, enables the device to execute the steps in the method.

[0014] A computer-readable storage medium, on which a program is stored, and when the program is executed by a processor, the steps in the method are implemented.

[0015] An electronic device includes a memory, a processor, and a program stored on the memory and executable on the processor. When the processor executes the program, the steps in the method are implemented.

[0016] Due to the adoption of the above technical solution, the beneficial effects obtained by the present application are: In the present application, by configuring D-H parameters for each joint in the UE4 blueprint editor, parametric modeling of the robotic arm structure is achieved, enabling the system to flexibly adapt to robotic arms of various brands and models, and improving the versatility and scalability of the system. By generating the transformation matrix of each joint and the tool coordinate system matrix, it is ensured that the position of the end effector can be accurately described, improving the modeling accuracy and simulation authenticity.

[0017] By establishing a Socket client-server communication link, the stability and real-time performance of data transmission between the UE4 editor and the robotic arm controller are ensured; receiving data at a preset frequency enables the actions of the virtual robotic arm to be highly synchronized with the real robotic arm. At the same time, receiving joint angles and output data provides a rich data basis for subsequent attitude updates and trajectory visualization.

[0018] Using the forward kinematics algorithm to perform real-time calculations on joint angles, the spatial position and attitude of the end effector are quickly obtained, providing key data support for subsequent model attitude updates and trajectory drawing. This process integrates D-H parameters and the tool coordinate system matrix, improving the accuracy of end effector positioning and providing guarantee for precise operations and path verification.

[0019] By dynamically updating the transformation matrix of each joint, the hierarchical driving of the attitudes of the joints of the virtual robotic arm is realized, making the actions of the entire robotic arm model smoother and more natural, and closer to the motion characteristics of the real robotic arm. Combining the position information of the tool center point ensures the precise positioning of the end effector in space and enhances the applicability of the system in complex tasks.

[0020] By dynamically creating line segments through the ULineBatchComponent component and connecting them into a continuous trajectory curve, the visualization display of the motion path of the end effector is realized, which helps users intuitively understand the working path of the robotic arm. The continuous generation method of the trajectory ensures the integrity and real-time performance of the path display, facilitating teaching demonstrations, path optimization, and fault troubleshooting.

[0021] Defining color change rules according to different output data and applying them to the trajectory display realizes the visual feedback of the operating state of the robotic arm. For example, reflecting the gripper pressure, temperature change, or task status through color changes improves the interactivity and intelligent perception ability of the system. This dynamic color feedback mechanism provides an auxiliary means for remote monitoring and abnormal warning, enhancing the practicality and safety of the system. Description of the Drawings

[0022] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The schematic embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation of the present application. In the drawings: Figure 1 It is a schematic flow chart of a robotic arm real-time simulation and visualization method based on UE4 provided by an embodiment of the present application. Detailed Embodiments

[0023] To more clearly explain the overall concept of the present application, the following will be described in detail by way of examples in conjunction with the drawings of the specification.

[0024] In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present application. However, the present application may be implemented in other ways different from those described herein. Therefore, the scope of protection of the present application is not limited by the specific embodiments disclosed below. It should be noted that, without conflict, the embodiments of the present application and the features in each embodiment may be combined with each other.

[0025] In the present application, unless otherwise clearly specified and limited, the first feature being "on" or "under" the second feature may be that the first and second features are in direct contact, or the first and second features are indirectly in contact through an intermediate medium. In the description of this specification, the description with reference to terms such as "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials, or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0026] Embodiment 1 As Figure 1 shown, an embodiment of the first aspect of the present application provides a method for real-time simulation and visualization of a robotic arm based on UE4, including: Establish a robotic arm model based on the UE4 editor and configure parameters to generate a transformation matrix and a tool coordinate system matrix.

[0027] As described above, create a new project or open an existing project in Unreal Engine 4 (UE4). Then, use the tools provided by UE4 to import the 3D model of the six-axis robotic arm. These models represent the actual physical structure of the robotic arm, including each joint and its connecting parts.

[0028] In UE4, configure parameters for the robotic arm through the blueprint system. These parameters mainly include D-H parameters (link length, offset, twist angle), which are the key to describing the geometric relationship between the joints of the robotic arm. The user can set these parameter values for each joint in the blueprint editor. This step allows the system to automatically generate a transformation matrix for the position and orientation of each joint relative to the previous joint according to the input parameters.

[0029] Next, configure the Tool Center Point (TCP) in the blueprint editor. This is to define the specific position and orientation of the end effector (such as a gripper or other tool) relative to the last joint. This step is crucial for accurately calculating the actual working position of the end effector of the robotic arm. By configuring the TCP, the system generates a tool coordinate system matrix that describes the position and orientation of the end effector relative to the last joint.

[0030] Based on the previously set D-H parameters, the system automatically generates transformation matrices for each joint. These matrices detail the position transformation and rotation angles from one joint to the next. This means that when the angles of all joints are determined, the pose of the entire robotic arm can be accurately calculated through these transformation matrices.

[0031] For example, suppose we are developing a simulation system for a manufacturing enterprise that needs to simulate the working process of a six-axis welding robotic arm. First, create a new project in UE4 and then import the 3D model files of this welding robotic arm. These model files contain the detailed designs of each joint, link, and end effector (such as a welding torch) of the robotic arm.

[0032] In the UE4 blueprint editor, we need to set the D-H parameters for each joint. For example: For the first joint (base), assume the link length a = 0 (since it is a rotating base), the offset d = 250 mm (the distance from the base to the center of the first joint), and the twist angle α = -90°.

[0033] The next few joints are similarly set according to the design parameters of the actual robotic arm. For example, the second joint may have a = 612 mm, d = 0, α = 0°, and so on.

[0034] According to the above-set D-H parameters, the system automatically generates transformation matrices for the position transformation and rotation angles of each joint relative to the previous joint. For example, for the first joint, its transformation matrix describes its specific position and orientation relative to the base; for the second joint, it describes its position and orientation relative to the first joint, and so on. In this way, we obtain a series of transformation matrices that describe the structure of the entire robotic arm.

[0035] Next, it is necessary to configure the tool coordinate system, that is, the position and orientation of the end effector relative to the last joint. In the example of this welding robot arm, it is assumed that the welding torch is installed at the front end of the last joint, slightly offset from the center and tilted at a certain angle to better contact the welding point. In the blueprint editor, set the specific position and orientation of the welding torch relative to the last joint to generate the tool coordinate system matrix (Tool_Transform). This matrix will be used for subsequent calculations of the actual working position of the end effector.

[0036] It should be noted that in a specific implementation scenario, based on the above solution, it is also possible to allow users to input or adjust the D-H parameters of each joint through a graphical interface, rather than just preset values. This can be achieved by developing an intuitive parameter input panel that enables users to easily modify parameters such as link length, offset, and twist angle, so as to quickly adapt to different robot arm models. Providing the function of real-time adjustment of joint parameters allows observing the impact of different parameters on the robot arm's posture during the simulation process. For example, when the user changes the link length of a certain joint, the system should be able to immediately update the transformation matrix of that joint and reflect it in the robot arm's posture in the virtual environment.

[0037] In a specific implementation scenario, based on the above solution, a mechanism can also be designed to enable users to conveniently replace the end effector in the simulation (such as switching from a welding torch to a gripper). For each tool, the Tool_Transform matrix relative to the last joint is preset in advance. When the user selects a different tool, the system automatically loads the corresponding matrix for calculation. Provide users with a tool calibration function to help them accurately set the actual installation position and orientation of the end effector. By guiding the user to complete a series of simple operations (such as moving the robot arm to a specific point), the system can automatically calculate the correct Tool_Transform matrix, improving the accuracy and efficiency of the setting.

[0038] The UE4 editor establishes communication with the robot arm controller and receives the current robot arm joint angles and output data sent by the robot arm controller at a preset frequency.

[0039] As described above, run a background program on the robot arm controller, and this program is responsible for creating a socket server. This server will be used to listen for connection requests from the UE4 client. Ensure that the server is configured with the correct IP address and port number so that the UE4 client can find and connect to it.

[0040] In the UE4 project, add the necessary network programming modules or plugins to support socket communication. Create a socket client instance that will attempt to connect to the socket server of the above robotic arm controller when the application starts.

[0041] Configure the client to connect using the IP address and port number of the robotic arm controller, ensuring a stable two-way communication link can be established between them.

[0042] Set a reasonable data reception frequency in the UE4 client, such as receiving 50 data updates per second. This depends on the specific application requirements and network bandwidth conditions. You can use a timer or a loop mechanism to request the latest status data from the robotic arm controller at fixed intervals.

[0043] After a successful connection, the UE4 client starts receiving data packets from the robotic arm controller at the set frequency. Each data packet contains the angle values of six joints and other possible output data (such as sensor readings, switch states, etc.).

[0044] The UE4 client needs to parse these raw data to extract the specific angle values and output information of each joint. For example, the data format may be [J1_Angle][J2_Angle]...[J6_Angle][DO], which includes the angles of six joints and additional data output (DO).

[0045] Using the received joint angle values and combining with the previously configured D-H parameters, calculate the current pose of the robotic arm through the forward kinematics (FK) algorithm. This means determining the position and orientation of each joint based on the joint angles and ultimately obtaining the exact position and pose of the end effector.

[0046] According to the calculation results, dynamically adjust the pose of the robotic arm model in the virtual environment to accurately reflect the actual state of the robotic arm.

[0047] For example, run a background program on the robotic arm controller that creates a socket server. This server listens for connection requests from the UE4 client. For example, the robotic arm controller is configured to listen on a specific port (such as 54321) on the IP address 192.168.1.100, waiting for a connection from the UE4 client.

[0048] In the UE4 project, the necessary network modules or plugins are added to support socket communication. A socket client instance is created that attempts to connect to the socket server of the above robotic arm controller.

[0049] When the UE4 application starts, it automatically attempts to connect to port 54321 of the IP address 192.168.1.100 to establish a stable communication link with the robotic arm controller.

[0050] A timer is set in the UE4 client, which triggers 10 times per second (i.e., once every 0.1 seconds). This means that UE4 will request the latest status data from the robotic arm controller every 0.1 seconds. This frequency can be adjusted according to actual needs to balance the timeliness of data updates and the consumption of network bandwidth.

[0051] Once a stable communication link is established, the UE4 client starts receiving data packets from the robotic arm controller at the set frequency. Each data packet contains the angle values of six joints and other output data (such as temperature, pressure, etc.).

[0052] Suppose the received data format is as follows: "J1=30°; J2=45°; J3=60°; J4=75°; J5=90°; J6=105°; Temp=45; Pressure=10". Here, "J1" to "J6" represent the angle values of six joints (in degrees), and "Temp" and "Pressure" are additional sensor readings.

[0053] After the UE4 client receives this data, it first parses it to extract the angle values of each joint and other output data. For example, from the above data, it can be known that the angle of the first joint is 30 degrees, the angle of the second joint is 45 degrees, and so on.

[0054] Using these joint angle values and combining with the previously configured D-H parameters, the pose of the current robotic arm is calculated through the forward kinematics algorithm. This means determining the position and orientation of each joint and finally obtaining the exact position and pose of the end effector.

[0055] According to the calculation results, the pose of the robotic arm model is dynamically adjusted in the UE4 virtual environment. For example, if it is calculated from the received data that the robotic arm should lift the welding torch upward, the position and orientation of the robotic arm model are adjusted accordingly in the virtual environment to accurately reflect the actions of the actual robotic arm.

[0056] For additional output data such as temperature and pressure, this information can be displayed on the UI interface or certain properties can be changed according to these data. For example, when the temperature exceeds a certain threshold, the trajectory color can be changed to alert the user.

[0057] It should be noted that in a specific implementation scenario, based on the above solution, threshold conditions can also be set. When certain key parameters (such as temperature, pressure) exceed the safe range, an alarm mechanism is triggered. The user can be notified of potential problems through methods such as sound alarms, flashing colors, or pop-up warning boxes.

[0058] In a specific implementation scenario, based on the above solution, the function of automatically recording all received data can also be realized. These records can be saved as text files or database entries for easy subsequent reference and analysis. For example, detailed information such as the joint angle changes and sensor readings during each operation are recorded. Tools are provided for the user to view and analyze past operation data. This can help identify long-term trends, optimize work processes, or diagnose the causes of faults. For example, the user can select data for a specific time period for playback to observe the motion patterns of the robotic arm and the changes in its related parameters.

[0059] In a specific implementation scenario, based on the above solution, the user can be allowed to remotely control the operation of the robotic arm through the network. An intuitive interactive interface is designed in the UE4 environment so that the user can adjust the posture of the robotic arm or execute tasks even when not on-site. A multi-user collaboration environment is built to enable operators and technicians at different locations to simultaneously access and control the same virtual robotic arm model. This helps the team to cooperate in problem-solving or jointly plan complex operation steps.

[0060] The UE4 editor calculates the current robotic arm joint angles through the forward kinematics algorithm based on D-H parameters to obtain the position information and attitude information of the end effector of the robotic arm.

[0061] As described above, first, D-H parameters need to be defined for each joint of the robotic arm. These parameters describe the specific position and orientation of each joint relative to its previous joint, including: Link length (a): The distance between two consecutive joint axes.

[0062] Offset (d): The distance moved along the previous joint axis.

[0063] Twist angle (α): The angle between two consecutive joint axes.

[0064] These parameters are set according to the design specifications of the robotic arm and are fixed for a specific model of the robotic arm.

[0065] Receive the angle data of each current joint from the robotic arm controller. For example, assume that the received data indicates that the angle of the first joint is 30 degrees, the angle of the second joint is 45 degrees, and so on until the angle of the sixth joint. These angles represent the actual rotation states of the various joints of the robotic arm.

[0066] Next, use the forward kinematics algorithm combined with D-H parameters and the received joint angle data to calculate the position and orientation of the end effector of the robotic arm. The specific process is as follows: Generate transformation matrices: For each joint, generate a transformation matrix based on its D-H parameters and the current angle. This matrix describes the position transformation and rotation of this joint relative to the previous joint.

[0067] Accumulate transformations: Apply the transformation matrices of each joint in sequence, starting from the base and going all the way to the end effector. This means multiplying all the joint transformation matrices to obtain the final position and orientation of the entire end effector of the robotic arm relative to the base.

[0068] Since the end effector may be installed at different positions or orientations on the last joint, the tool coordinate system (TCP) also needs to be considered. The tool coordinate system matrix (Tool_Transform) describes the position and orientation of the end effector relative to the last joint. By multiplying the previously calculated transformation result by Tool_Transform, the actual position and orientation of the end effector can be accurately determined.

[0069] After the above steps, we obtain the exact position and orientation information of the end effector of the robotic arm. This includes the coordinates (x, y, z) of the end effector in three-dimensional space, as well as its orientation relative to the global coordinate system (usually represented by Euler angles or quaternions).

[0070] For example, assume that a six-axis industrial robotic arm is being used for a handling operation. This robotic arm is installed on an automated production line and is responsible for moving materials from one location to another. It is necessary to accurately simulate the movements of this robotic arm in the UE4 environment for training operators or optimizing its workflow.

[0071] First, set the D-H parameters for each joint of this six-axis robotic arm. These parameters are determined according to the design specifications of the robotic arm and are used to describe the specific position and orientation of each joint relative to the previous joint. For example: For the first joint (base), assume the link length a = 0 (since it is a rotating base), the offset d = 250 mm (the distance from the base to the center of the first joint), and the twist angle α = -90°. For the second joint, assume the link length a = 612 mm, the offset d = 0, and the twist angle α = 0°.

[0072] The remaining joints are similarly set according to the actual design parameters.

[0073] Assume that the data received from the robotic arm controller indicates that the current angles of each joint are as follows: First joint angle: J1 = 30°. Second joint angle: J2 = 45°. Third joint angle: J3 = 60°. Fourth joint angle: J4 = 75°. Fifth joint angle: J5 = 90°. Sixth joint angle: J6 = 105°.

[0074] These angles represent the actual rotation states of the various joints of the robotic arm.

[0075] Next, use the forward kinematics algorithm to combine the above D-H parameters and the received joint angle data to calculate the position and orientation of the end effector of the robotic arm: Generate transformation matrices: For each joint, generate a transformation matrix based on its D-H parameters and the current angle. This matrix describes the position transformation and rotation of the joint relative to the previous joint.

[0076] Accumulate transformations: Apply the transformation matrices of each joint in sequence, starting from the base and going all the way to the end effector. This means multiplying all the joint transformation matrices to obtain the final position and orientation of the end effector of the entire robotic arm relative to the base.

[0077] Considering that the end effector may be installed at different positions or orientations on the last joint, the tool coordinate system (TCP) also needs to be considered. Assume that our end effector is a gripper, which has a specific offset and orientation relative to the sixth joint. By multiplying the previously calculated transformation result by Tool_Transform, the actual position and orientation of the gripper can be accurately determined.

[0078] After the above steps, we obtain the exact position and orientation information of the end effector of the robotic arm. For example, in this handling task, it is calculated that the current position of the gripper is (x = 1200mm, y = 800mm, z = 500mm), and its orientation is towards the next station on the production line. In this way, we know the specific position of the gripper in space and its pointing direction.

[0079] It should be noted that in a specific implementation scenario, on the basis of the above solution, after calculating the position and orientation of the end effector of the robotic arm, functions can be further developed to predict its future path. For example, according to the preset task objective or the target point input by the user, the system can dynamically generate an optimal path from the current position to the target position and display this path in real time in a virtual environment. Combining with the environment model (such as workstations, obstacles, etc.), the system can detect the potential collision risks during the movement of the robotic arm in real time. Once a potential collision is detected, the system can automatically adjust the path or prompt the user for manual adjustment to ensure the safety of the operation.

[0080] In a specific implementation scenario, based on the above solution, a mechanism can be provided to allow users to dynamically adjust certain D-H parameters or other relevant parameters (such as the tool coordinate system) during runtime. This enables users to fine-tune the behavior of the robotic arm according to the actual situation without restarting the entire simulation process. For example, if it is found during actual operation that the angle of a certain joint deviates from the expected value, the user can directly modify the relevant parameters of that joint on the interface and immediately see the effect. Provide users with a simple calibration tool to help them accurately set or calibrate the D-H parameters and tool coordinate system of the robotic arm. For example, guide users to automatically calculate the correct parameter values through a series of simple operations (such as moving the robotic arm to specific reference points) to improve the accuracy and efficiency of the settings.

[0081] Update the transformation matrix based on the tool coordinate system matrix, the position information, and the attitude information to control the position and attitude of the robotic arm joints of the robotic arm model, generate the motion trajectory of the end effector of the robotic arm based on the position information, and change the color of the motion trajectory based on the output data.

[0082] As described above, first, the position and attitude of the end effector of the robotic arm relative to the base are calculated through the forward kinematics (FK) algorithm. This process combines the D-H parameters and the current angle values of each joint and takes into account the tool coordinate system matrix (Tool_Transform) to obtain the exact position and orientation of the end effector.

[0083] Based on the above calculation results, the system generates or updates the transformation matrix (Transform) for each robotic arm joint. These transformation matrices contain the specific position and rotation angle of each joint relative to its previous joint.

[0084] Then, by setting the Transform of each robotic arm joint model, the system updates the attitude of the robotic arm in the virtual environment. This means that the robotic arm model in the virtual environment will be adjusted accordingly according to the state of the actual robotic arm, so that the virtual model is consistent with the robotic arm in reality. For example, if the calculation results show that the first joint needs to rotate 30 degrees clockwise, the angle of that joint is adjusted accordingly in the virtual environment.

[0085] According to the currently calculated position of the tool center point (TCP), the system can determine the specific position of the end effector of the robotic arm in space. For example, at a certain moment, it is calculated that the end effector is located at the coordinates (x = 100mm, y = 200mm, z = 300mm) in three-dimensional space.

[0086] To visualize the path of the end effector of the robotic arm, ULineBatchComponent is used in UE4 to create line objects. Each time the robotic arm moves, a new line segment is generated based on the new TCP position and added to the previous lines, thus forming a continuous trajectory curve. For example, when the robotic arm moves from the starting point to the ending point, the system records the position of the end effector at each time point and connects these points to form a smooth curve.

[0087] The data received from the robotic arm controller includes not only joint angles but also other output data (such as sensor readings, switch states, etc.), which is usually abbreviated as DO (Data Output). For example, assume a temperature sensor is installed on the robotic arm, which can provide real-time feedback on the current operating temperature.

[0088] According to different DO data, a set of rules can be defined to change the color of the ULineBatchComponent. For example, it is set that when the temperature is below 50 degrees Celsius, the trajectory is displayed in blue; when the temperature is between 50 and 100 degrees Celsius, the trajectory turns yellow; when it exceeds 100 degrees Celsius, the trajectory becomes red. The purpose of this is to provide an intuitive way to display different working states or operation modes of the robotic arm.

[0089] This method can not only enhance the user's understanding and monitoring ability of the robotic arm's operation but also help quickly identify potential problems. For example, if it is found that the trajectory suddenly turns red, it may mean that the robotic arm is overheating and immediate measures need to be taken to prevent damage.

[0090] For example, assume that a six-axis industrial robotic arm is being used for the assembly operation of electronic components. The task of this robotic arm is to pick up components from the tray and accurately place them at the specified positions on the circuit board. It is necessary to simulate the actions of this robotic arm in the UE4 environment to facilitate operator training or optimize its workflow.

[0091] After receiving the angle data of each joint from the robotic arm controller, through the forward kinematics algorithm combined with D-H parameters and the tool coordinate system matrix (Tool_Transform), the exact position and orientation of the end effector of the robotic arm (such as the gripper) are calculated. For example, the calculation results show that the gripper is located at the coordinates (x = 150mm, y = 200mm, z = 300mm) in the three-dimensional space, and its direction is vertically downward.

[0092] Based on the above calculation results, the system generates or updates the transformation matrix for each robotic arm joint. These transformation matrices describe the specific position and rotation angle of each joint relative to the previous joint. For example, if the calculation results show that the first joint needs to rotate 45 degrees clockwise, the angle of that joint is adjusted accordingly in the virtual environment.

[0093] Then, by setting the Transform of each robotic arm joint model, the system updates the pose of the robotic arm in the virtual environment. This means that the robotic arm model in the virtual environment is adjusted according to the actual state of the real robotic arm, so that the virtual model is consistent with the real-world robotic arm. In this example, the virtual robotic arm will adjust the positions of all joints according to the calculation results, so that the gripper accurately moves to the predetermined position.

[0094] When the robotic arm moves from the starting point to the ending point, the system records the position of the end effector at each time point. For example, assume the robotic arm starts moving from the initial position (x = 0mm, y = 0mm, z = 0mm), passes through a series of intermediate points, and finally reaches the target position (x = 150mm, y = 200mm, z = 300mm).

[0095] Use ULineBatchComponent to create a straight line object in UE4. Each time the robotic arm moves, a new line segment is generated based on the new TCP position and added to the previous line, thus forming a continuous trajectory curve. In this example, as the robotic arm moves, the system dynamically generates a smooth curve from the starting point to the ending point, showing the actual movement path of the gripper.

[0096] The data received from the robotic arm controller includes not only joint angles but also other output data (such as sensor readings, switch states, etc.). For example, assume a force sensor is installed on the robotic arm, which can provide real-time feedback on the pressure currently applied to the gripper.

[0097] According to different DO data, a set of rules can be defined to change the color of the ULineBatchComponent. For example, it is set that when the pressure on the gripper is less than 5 Newtons, the trajectory is displayed in green; when the pressure is between 5 and 10 Newtons, the trajectory turns yellow; when it exceeds 10 Newtons, the trajectory becomes red. The purpose of this is to provide an intuitive way to display different working states or operation modes of the robotic arm.

[0098] In this example, assume the gripper gradually increases the pressure when approaching the target position. Then it can be seen that the trajectory color changes from green to yellow and finally to red, indicating that the gripper has successfully grasped the component and applied enough pressure to ensure that the component will not fall off.

[0099] It should be noted that in a specific implementation scenario, based on the above solution, a path optimization algorithm can also be added on the existing basis, enabling the system to dynamically adjust the movement path of the robotic arm according to the real-time feedback data. For example, if an obstacle or a new target point is detected, the system can automatically recalculate an optimal path and instantaneously update the path of the robotic arm model in the virtual environment. Provide an intuitive interface for users to allow them to adjust D-H parameters, tool coordinate system parameters, and other relevant parameters during runtime. This can help users quickly respond to problems encountered in actual operations, such as robotic arm precision deviation, without having to restart the entire simulation process.

[0100] In a specific implementation scenario, based on the above solution, it is also possible to implement detailed logging of each solution result and all relevant parameters (including position, orientation, output data, etc.). These records are not limited to the final results but should also include the data of intermediate steps for subsequent analysis and problem troubleshooting. Provide a dedicated tool for users to be able to view and analyze past run data. For example, users can select data for a specific time period for playback, observe the motion patterns of the robotic arm and the changing trends of its relevant parameters, so as to identify potential problems or optimize the operation process.

[0101] According to an embodiment of the present application, the robotic arm model is established based on the UE4 editor, and parameters are configured to generate a transformation matrix and a tool coordinate system matrix. Specifically: Configure D-H parameters for each joint in the UE4 blueprint editor. The D-H parameters include the link length a, the offset d, and the twist angle α. Based on the input D-H parameters, generate a transformation matrix for each joint. Configure the tool center point in the UE4 blueprint editor, that is, the position and orientation of the end effector relative to the last joint. According to the configuration of the tool center point, generate a tool coordinate system matrix.

[0102] As described above, in the UE4 blueprint editor, set D-H parameters for each joint of the robotic arm. These parameters are the basic geometric information used to describe the structure of the robotic arm, mainly including the following three items: Link length a: Represents the shortest distance between the rotation axes of two adjacent joints.

[0103] Offset d: Represents the distance from the origin of the current coordinate system to the origin of the next coordinate system along the direction of the rotation axis of the previous joint.

[0104] Twist angle α (alpha): Represents the angle between the rotation axes of two adjacent joints.

[0105] These parameters are set according to the physical structure of the actual robotic arm. Different models of robotic arms have different D-H parameters. In the blueprint, users can input or adjust these parameters through a graphical interface, so as to flexibly adapt to a variety of robotic arm models.

[0106] After completing the D-H parameter configuration, the system will automatically generate a transformation matrix for each joint based on these parameters. This transformation matrix describes the position and attitude change relationship of the current joint relative to its previous joint, that is, the transformation process of "moving and rotating from the previous joint to the current joint".

[0107] These transformation matrices are the basis of the entire kinematic calculation. They will be applied sequentially, starting from the base and gradually transmitted to the end effector, thereby constructing the spatial structure of the entire robotic arm. That is to say, the position and attitude of each joint depend on the previous joint, and this hierarchical relationship is expressed through the transformation matrix.

[0108] Next, in the blueprint editor, it is necessary to define the Tool Center Point (TCP), that is, the position and attitude of the end effector relative to the last joint. For example, if the end effector is a gripper, it may be installed at the front end of the sixth joint and has a certain offset and tilt angle.

[0109] This step is to ensure that the simulation system can accurately locate the "working point" of the robotic arm, that is, the position where it actually works (such as the welding point, the grasping point, etc.). Users can set this offset and direction in an intuitive way in the blueprint without manually writing any code.

[0110] Finally, the system will generate a Tool_Transform Matrix according to the above configuration of the tool center point. This matrix describes the specific transformation relationship of the end effector relative to the last joint.

[0111] In the subsequent forward kinematic solution process, this tool coordinate system matrix will be used to multiply with the transformation matrices of all previous joints, so as to finally determine the position and attitude of the end effector in the global space.

[0112] According to an embodiment of the present application, the UE4 editor establishes communication with the robotic arm controller and receives the current robotic arm joint angles and output data sent by the robotic arm controller at a preset frequency. Specifically: Establish a Socket server at the robotic arm controller end, establish a Socket client at the UE4 editor, and the Socket client initiates a connection request to the IP address and port number of the robotic arm controller to establish a stable two-way communication link; The Socket client receives the data from the robotic arm controller at a preset frequency, and the data includes the angle values of six joints and other output data.

[0113] As described above, first, run a background program on the robotic arm controller device, and this program is responsible for creating and starting a Socket server. The role of this server is to listen for connection requests from external clients (i.e., the UE4 editor) and continuously send the real-time status data of the robotic arm to the client.

[0114] This Socket server will bind a fixed IP address and a specified port number as the entry for its network communication. Once started successfully, it can wait for connection requests from the UE4 editor.

[0115] Meanwhile, develop or integrate a Socket client module in the Unreal Engine 4 (UE4) editor. This client has the ability to initiate a network connection actively.

[0116] When the user starts the simulation system in the UE4 editor, the Socket client will automatically attempt to connect to the IP address and the corresponding port number of the robotic arm controller. Once the connection is successful, a stable two-way communication link is established between the two parties, and data interaction can start.

[0117] This two-way communication mechanism not only allows the UE4 side to receive the data of the robotic arm, but also supports sending control instructions (such as pause, restart, etc.) from the UE4 side to the robotic arm controller when necessary.

[0118] After the communication link is established, the Socket client in the UE4 editor will receive data packets from the robotic arm controller at a set fixed frequency (such as 10 times per second, 50 times per second, etc.).

[0119] The angle values of six joints: represent the actual rotation angle of each joint currently, and are used to drive the posture update of the virtual robotic arm model in UE4 subsequently.

[0120] Other output data (DO): may include additional information such as the status signal of the robotic arm, sensor readings, digital output signals (such as whether the gripper is closed), temperature, pressure, etc. These data can be used to enhance the authenticity and functionality of the simulation, such as for trajectory color change, status prompt, alarm triggering, etc.

[0121] The data reception frequency can be configured according to actual needs. The higher the frequency, the closer the actions of the virtual model are to the real robotic arm, but it will also bring higher network load and computational overhead.

[0122] According to an embodiment of the present application, the UE4 editor calculates the current robotic arm joint angles through a forward kinematics algorithm based on D-H parameters to obtain the position information and attitude information of the end effector of the robotic arm. Specifically: Based on the above D-H parameters, a corresponding transformation matrix is constructed for each joint; Apply the FK algorithm to calculate the position and attitude of the end effector of the robotic arm relative to the base by multiplying the transformation matrices of all joints; After calculating the final transformation matrix of the end effector position and attitude, it is combined with the tool coordinate system matrix to obtain the exact position and attitude of the end effector.

[0123] As described above, in the UE4 editor, the D-H parameters (including link length a, offset d, and twist angle α) of each joint of the robotic arm have been pre-configured. These parameters are the basic geometric information describing the structure of the robotic arm.

[0124] The system will generate a corresponding transformation matrix for each joint of the robotic arm according to these parameters. This transformation matrix expresses the spatial transformation relationship of the joint relative to its previous joint, including position movement and direction rotation. That is to say, the transformation matrix of each joint describes the change from the previous joint to the current joint.

[0125] Next, the system uses the forward kinematics (FK) algorithm to sequentially perform superposition operations on the transformation matrices of each joint, that is, multiply them sequentially.

[0126] Starting from the base of the robotic arm, applying the transformation matrix of each joint step by step, the overall transformation result of the end effector relative to the base can be finally obtained. This final transformation matrix contains the specific position (x, y, z coordinates) of the end effector in three-dimensional space and its attitude (i.e., orientation, usually represented by Euler angles or quaternions).

[0127] In other words, in this way, the system can accurately know the actual working point and its direction of the end effector (such as gripper, welding torch, etc.) of the robotic arm in the virtual space.

[0128] To further improve the accuracy, after obtaining the position and attitude of the above end effector, the system will also combine it with the previously set tool coordinate system matrix (Tool_Transform).

[0129] The tool coordinate system matrix describes the actual installation position and direction of the end effector relative to the last joint. For example, if the gripper is not exactly installed at the center point of the sixth axis but has a certain offset and tilt, then this matrix will record this deviation information.

[0130] After multiplying the result calculated by FK with the tool coordinate system matrix, the exact position and orientation of the end effector in the global coordinate system can be obtained, ensuring that the actions of the virtual model are exactly the same as those of the real robotic arm.

[0131] According to an embodiment of the present application, updating the transformation matrix based on the tool coordinate system matrix, the position information, and the orientation information to control the positions and orientations of the robotic arm joints of the robotic arm model specifically includes: Based on the position information and orientation information of the end effector of the robotic arm, update the transformation matrix of each robotic arm joint. By setting the transformation matrix of each robotic arm joint model, update the orientation of the robotic arm in the virtual environment; determine the specific position of the end effector of the robotic arm in space according to the currently calculated position of the tool center point.

[0132] As described above, first, the system will, based on the previously calculated position and orientation information of the end effector of the robotic arm, in combination with the already set D-H parameters and the tool coordinate system matrix, inversely deduce the spatial transformation states that each intermediate joint should be in.

[0133] This process does not directly use the position of the end effector to "move" the model, but rather determines the correct pose of each joint in space step by step through the kinematic link relationship. In other words, the system will generate a new transformation matrix for each joint to describe its new position and new direction relative to the previous joint.

[0134] In the UE4 editor, each joint of the robotic arm is a model component with independent spatial coordinates. Once the new transformation matrices of each joint are obtained, the system will apply these matrices to the corresponding joint models.

[0135] This operation is equivalent to telling the UE4 engine: "Now, please move this joint here and rotate it to this angle." In this way, the orientation of the entire robotic arm model will change accordingly and be consistent with the real robotic arm.

[0136] For example, if the angle of a certain joint changes, then its transformation matrix will also be updated accordingly, and the model will make corresponding actions in the virtual environment, such as raising the arm or rotating the wrist.

[0137] After the entire update is completed, the system will also pay special attention to the position of the tool center point (TCP). This point represents the position where the end effector actually functions, such as the contact point where the gripper grasps an object.

[0138] The system will calculate the exact three-dimensional position of the tool center point in the global coordinate system based on the latest transformation matrix chain and the tool coordinate system matrix. This can ensure that not only the overall posture of the robotic arm is correctly restored, but also the working point at its end can be accurately positioned.

[0139] This position information can also be used for subsequent functions, such as drawing the motion trajectory of the end effector, determining whether the target point is reached, triggering interaction events, etc.

[0140] According to an embodiment of the present application, generating the motion trajectory of the end effector of the robotic arm based on the position information specifically is: Use ULineBatchComponent to create a line object. Each time the robotic arm moves, a new line segment is generated based on the new position of the tool center point and added to the previous line to form a continuous trajectory curve.

[0141] As described above, in Unreal Engine 4 (UE4), ULineBatchComponent is a component for efficiently rendering lines and shapes. It allows developers to dynamically add and manage a large amount of line data with relatively low performance overhead.

[0142] Whenever the robotic arm moves, the system uses this component to create new line segments, which represent the path of the end effector moving from one position to the next.

[0143] After each movement of the robotic arm, the system calculates the position of the current tool center point (TCP). TCP is the point on the end effector that actually functions, such as the actual grasping point of the gripper or the welding point of the welding torch.

[0144] Based on the current and the previously recorded TCP positions, the system creates a new line segment between the two. This line segment represents the direct path of the end effector between these two positions.

[0145] The newly generated line segment is automatically added to the existing line set, forming a continuous trajectory curve. In this way, as the robotic arm continues to move, the entire motion path of its end effector will gradually emerge.

[0146] The system receives data from the robotic arm controller at each time interval (for example, 50 times per second) and updates the trajectory based on the latest TCP position. This means that the trajectory can reflect the actual actions of the robotic arm in real time and provide instant visual feedback.

[0147] If the robotic arm changes its direction or speed, the trajectory will be adjusted accordingly. This allows users to clearly see any subtle changes, helping them better understand and optimize the operation process of the robotic arm.

[0148] According to an embodiment of the present application, changing the color of the motion trajectory based on the output data specifically includes: Define a set of rules to change the color of the ULineBatchComponent according to different output data.

[0149] As mentioned above, during the operation of the robotic arm, in addition to the joint angles and the position information of the end effector, there will be other types of output data (Output Data, DO), such as sensor readings (e.g., temperature, pressure), status signals (e.g., whether the gripper is closed), speed, acceleration, etc. These data reflect the current working state or environmental conditions of the robotic arm.

[0150] To enable the operator to more intuitively understand the state of the robotic arm, the system can visualize this output data by changing the color of the motion trajectory. Specifically, a set of rules is defined according to different output data to dynamically adjust the color of the trajectory.

[0151] First, it is necessary to determine which output data is most important for monitoring the working state of the robotic arm. For example, assume that we are concerned about the force sensor reading on the end effector of the robotic arm, which can help us determine whether the force during object grasping is appropriate.

[0152] For the selected output data, set several key threshold ranges. Each range corresponds to a specific color. For example: When the force sensor reading is below a certain low threshold (e.g., 5 Newtons), the trajectory is displayed in green, indicating a small and safe force.

[0153] When the reading is between two thresholds (e.g., 5 to 10 Newtons), the trajectory turns yellow, prompting the user to note that the force has increased but is still within the normal range.

[0154] If the reading exceeds the high threshold (e.g., greater than 10 Newtons), the trajectory then turns red, warning of a possible overload risk.

[0155] Establish a simple mapping rule table to map output data values in different ranges to the corresponding colors. This rule can be linear (e.g., the color changes from green to red as the value increases) or non-linear (flexibly set according to actual needs).

[0156] Whenever new output data is received, the system automatically calculates the corresponding trajectory color according to the rules defined above and updates the line color in the ULineBatchComponent.

[0157] If the output data changes frequently (such as being updated multiple times per second), the system needs to be able to respond quickly and adjust the trajectory color immediately to ensure that the color always accurately reflects the current state of the output data.

[0158] To avoid a visually jarring sense of incoherence caused by sudden color changes, a smooth transition can be applied between adjacent segments. For example, when the force sensor reading suddenly rises from 9 Newtons to 11 Newtons, the trajectory color does not instantaneously jump from yellow to red but instead transitions gradually, providing a more seamless visual effect.

[0159] A computer program product containing instructions, when run on a device, causes the device to perform the steps implemented in the said method.

[0160] A computer-readable storage medium having stored thereon a program which, when executed by a processor, implements the steps in the said method.

[0161] An electronic device comprising a memory, a processor, and a program stored on the memory and executable on the processor, wherein the processor, when executing the program, implements the steps in the said method.

[0162] What is not described in this application can be implemented by adopting or referring to existing technologies.

[0163] Each embodiment in this specification is described in a progressive manner. For the same or similar parts among the embodiments, reference can be made to each other, and each embodiment focuses on the differences from other embodiments.

[0164] The above are only the embodiments of this application and are not used to limit this application. For those skilled in the art, various changes and modifications can be made to this application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of this application shall be included within the protection scope of this application.

Claims

1. A real-time simulation and visualization method for a robotic arm based on UE4, characterized in that include: Build a robotic arm model based on the UE4 editor and configure parameters to generate the transformation matrix and tool coordinate system matrix; The UE4 editor establishes communication with the robotic arm controller, and receives the current robotic arm joint angle and output data sent by the robotic arm controller at a preset frequency; The UE4 editor calculates the current joint angle of the robotic arm by a forward kinematics algorithm based on DH parameters to obtain position information and posture information of the end effector of the robotic arm; Based on the tool coordinate system matrix, the position information and the posture information, the transformation matrix is ​​updated to control the position and posture of the robotic arm joints of the robotic arm model, the motion trajectory of the robotic arm end effector is generated based on the position information, and the color of the motion trajectory is changed based on the output data.

2. The method according to claim 1, characterized in that, The mechanical arm model is established based on the UE4 editor, and the parameters are configured to generate the transformation matrix and the tool coordinate system matrix, specifically: Configure DH parameters for each joint in the UE4 blueprint editor, where the DH parameters include link length a, offset d, and torsion angle α; Generate the transformation matrix of each joint based on the input DH parameters; Configure the tool center point in the UE4 blueprint editor, that is, the position and posture of the end effector relative to the last joint; A tool coordinate system matrix is ​​generated according to the configuration of the tool center point.

3. The method according to claim 1, wherein The UE4 editor establishes communication with the robotic arm controller, and receives the current robotic arm joint angle and output data sent by the robotic arm controller at a preset frequency, specifically: A Socket server is established on the robotic arm controller, and a Socket client is established on the UE4 editor. The Socket client initiates a connection request to the IP address and port number of the robotic arm controller to establish a stable two-way communication link. The Socket client receives data from the robotic arm controller at a preset frequency, wherein the data includes angle values ​​of six joints and other output data.

4. The method according to claim 1, wherein The UE4 editor calculates the current robot arm joint angle through a forward kinematics algorithm based on DH parameters to obtain the position information and posture information of the robot arm end effector, specifically: According to the above DH parameters, the corresponding transformation matrix is ​​constructed for each joint; Apply the FK algorithm to calculate the position and posture of the robot end effector relative to the base by multiplying the transformation matrices of all joints; After calculating the final transformation matrix of the end-effector position and attitude, it is combined with the tool coordinate system matrix to obtain the exact position and attitude of the end-effector.

5. The method according to claim 1, characterized in that, The updating of the transformation matrix based on the tool coordinate system matrix, the position information and the posture information to control the position and posture of the mechanical arm joint of the mechanical arm model is specifically: Based on the position information and posture information of the end effector of the robotic arm, the transformation matrix of each robotic arm joint is updated, and the posture of the robotic arm in the virtual environment is updated by setting the transformation matrix of each robotic arm joint model; According to the currently calculated tool center point position, the specific position of the robot arm end effector in space is determined.

6. The method according to claim 1, wherein Generating the motion trajectory of the end effector of the robotic arm based on the position information specifically includes: Using the ULineBatchComponent to create a straight line object. Each time the robotic arm moves, a new straight line segment is generated based on the new position of the tool center point and added to the previous line to form a continuous trajectory curve.

7. The method according to claim 1, characterized in that, Changing the color of the motion trajectory based on the output data specifically includes: Defining a set of rules to change the color of the ULineBatchComponent according to different output data.

8. A computer program product comprising instructions, when it runs on a device, characterized in that, Enabling the device to execute the steps in the method according to any one of claims 1-7.

9. A computer-readable storage medium having a program stored thereon, characterized in that, When the program is executed by the processor, it implements the steps in the method according to any one of claims 1-7.

10. An electronic device, comprising a memory, a processor, and a program stored on the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps in the method according to any one of claims 1-7.

Images