Manipulator touch visualization system based on unreal engine

Abstract

The invention provides a manipulator touch visualization system based on an unreal engine, which relates to the technical field of virtual reality and comprises an input layer, a processing layer and an output layer, the input layer comprises a manipulator body, a pressure sensor array, a controller and a lower computer network communication module; the processing layer comprises a network communication module, a touch visualization module and a posture synchronization module; the pressure sensor array is used for acquiring pressure data borne by the manipulator body, and the controller is used for driving fingers of the manipulator body to rotate and acquiring steering engine angle data of the manipulator body; the tactile visualization module is used for constructing a tactile visualization virtual hand and applying pressure data rendering to the tactile visualization virtual hand; the attitude synchronization module is used for constructing an attitude synchronization virtual hand and applying the attitude synchronization virtual hand to the attitude synchronization virtual hand based on the joint angle data so as to realize synchronous rotation of the attitude synchronization virtual hand; and the output layer is used for displaying the tactile visualization virtual hand and the posture synchronization virtual hand.

Description

Technical Field

[0001] The present invention relates to the field of virtual reality technology, and in particular to a manipulator tactile visualization system based on an Unreal Engine. Background Art

[0002] In existing technologies, tactile visualization systems typically utilize tactile sensors to present pressure information collected by tactile points on a tactile sensor array within a host computer software platform. Alternatively, these systems employ heatmaps, using grayscale gradients or pseudo-color coding to represent the spatial distribution of pressure intensity. However, these tactile visualization methods have significant limitations: first, they poorly reproduce the actual tactile distribution and fail to capture the complex tactile characteristics of the real world. Second, the information presented is too limited, encompassing only the pressure at discrete points, making them inadequate for complex applications such as tactile feedback and remote surgical operations. Summary of the Invention

[0003] The purpose of the present invention is to provide a manipulator tactile visualization system based on Unreal Engine, and to build a manipulator tactile visualization system with good real-time performance and good tactile visualization effects in a virtual environment by combining Unreal Engine technology. 1. Since the system has low network latency and good real-time performance, it can quickly obtain sensor data of the manipulator during various operations, and process and present the data in real time. 2. Utilizing the material blueprint system of Unreal Engine, based on the sensor pressure point data, tactile visualization results close to the real tactile distribution are obtained through interpolation calculation, vividly presenting tactile information. 3. Meet the subsequent tactile feedback application requirements and enhance people's acquisition of touch in some special scenarios.

[0004] A manipulator tactile visualization system based on Unreal Engine, comprising: an input layer, a processing layer, and an output layer; The input layer includes a manipulator body, a pressure sensor array, a controller and a lower computer network communication module; the processing layer includes a network communication module, a tactile visualization module and a posture synchronization module; The pressure sensor array is provided on the manipulator body, and is used to obtain pressure data on the manipulator body. The controller is used to drive the rotation of the fingers of the manipulator body and obtain the steering gear angle data of the manipulator body. The lower computer network communication module is used to send the pressure data and the steering gear angle data to the network communication module; The tactile visualization module is used to construct a tactile visualization virtual hand and apply the pressure data rendering to the tactile visualization virtual hand; The posture synchronization module is used to construct a posture synchronization virtual hand and calculate the rotation amount through the FinterpTo node based on the servo angle data and apply it to the posture synchronization virtual hand to achieve synchronous rotation of the posture synchronization virtual hand; The output layer is used to display the tactile visualization virtual hand and the posture synchronization virtual hand.

[0005] Optionally, the lower computer network communication module communicates with the network communication module based on the UDP network protocol.

[0006] Optionally, the network communication module is implemented based on a UDP Wrapper plug-in.

[0007] Optionally, the tactile visualization module is specifically: Building a tactile visualization virtual hand blueprint based on the Unreal Engine, creating a dynamic material instance using the Create Dynamic Material Instance blueprint node in the constructor of the tactile visualization virtual hand blueprint, adding a palm static mesh component to the tactile visualization virtual hand blueprint, customizing the material blueprint, and binding the custom material to the material slot of the mesh; Based on the arrangement of the pressure sensor array, set the UV coordinates of the tactile points in the material blueprint; Determine the pressure influence radius based on COMSOL simulation results; Based on the UV coordinates and pressure influence radius, and by inputting the center and edge color vectors, write a custom Blueprint node that outputs tactile visualization results based on real-time pressure values; Based on the custom blueprint node, Gaussian blur interpolation calculation is performed on the pressure data received by the network communication module, and rendering is achieved.

[0008] Optionally, the posture synchronization module is specifically: Build a pose-synchronized virtual hand blueprint based on the Unreal Engine and add a Poseable Mesh component to the blueprint; Convert the static mesh of the pose-synced virtual hand blueprint into a skeletal mesh, and use the bone editor to build the bone tree and perform skinning. Bind the Skeletal Mesh to the Poseable Mesh component added in the Pose Sync Virtual Hand Blueprint; Based on the servo angle data, the rotation amount is calculated through the Finterp To node, and the Set BoneRotation By Name node is used to achieve the synchronous rotation of the posture-synchronized virtual hand.

[0009] The effects of the present invention are as follows: The present invention provides a manipulator tactile visualization system based on the Unreal Engine, establishes a network communication between the virtual environment based on the Unreal Engine and the manipulator system, and realizes stable and fast sensor data transmission.

[0010] The present invention is based on the Unreal Engine-based manipulator tactile visualization system, builds a virtual manipulator model on the virtual end, and realizes the synchronization of the real manipulator and the virtual manipulator posture.

[0011] The present invention is based on the Unreal Engine-based manipulator tactile visualization system, combined with the Unreal Engine's material blueprint system, to perform real-time interpolation calculation and rendering of original pressure information, presenting a realistic tactile visualization effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 This is a module flow chart of the manipulator tactile visualization system based on Unreal Engine of the present invention; Figure 2 This is a schematic diagram of the implementation process of the network communication module of the present invention; Figure 3 It is a schematic diagram of the process of customizing materials of the present invention; Figure 4 This is a schematic diagram of the tactile rendering effect of the present invention; Figure 5 This is a schematic diagram of the real-time rotation results of the posture-synchronized virtual hand based on joint angle data of the present invention.

[0013] In the figure: 1. Input layer; 2. Processing layer; 3. Output layer; 11. Robot body; 12. Pressure sensor array; 13. Lower computer network communication module; 21. Network communication module; 22. Tactile visualization module; 23. Posture synchronization module. DETAILED DESCRIPTION

[0014] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings.

[0015] Figure 1 This is a flow chart of the module of the manipulator tactile visualization system based on Unreal Engine. Figure 1 As shown, the present invention provides a manipulator tactile visualization system based on Unreal Engine, characterized in that it includes: an input layer 1, a processing layer 2 and an output layer 3.

[0016] The input layer 1 includes a manipulator body 11, a pressure sensor array 12, a controller, and a lower-level network communication module 13. The processing layer 2 includes a network communication module 21, a tactile visualization module 22, and a posture synchronization module 23. Preferably, the controller uses an STM32F407ZET6 chip.

[0017] The pressure sensor array 12 is provided on the manipulator body 11 , and is used to obtain pressure data on the manipulator body 11 . The controller is used to drive the rotation of the fingers of the manipulator body 11 and obtain the servo angle data of the manipulator body 11 .

[0018] The lower computer network communication module 13 is used to send the pressure data and the servo angle data to the network communication module 21. Specifically, the lower computer network communication module 13 communicates with the network communication module 21 based on the UDP network protocol. The network communication module 21 is implemented based on the UDP Wrapper plug-in.

[0019] like Figure 2 As shown, the network communication module 21 uses the functionality of the UDP Wrapper plug-in and adds multiple UDP components to the blueprint class. The Event Begin Play event is executed when the game starts. When the Event Begin Play event is executed, the program sets the IP addresses and port numbers of all UDP components through the Open Receive Socket blueprint node to enable network communication for all UDP components. The On Receive Byte event is triggered after the UDP component receives byte data. All data is stored in a byte array. Subsequent program blocks decode and classify the initial data to obtain: pressure data, which is applied to the tactile visualization module 22 through material parameters; and servo angle data, which is applied to the posture synchronization module 23 through GameInstance. Figure 2 In the example, UDP stands for User Datagram Protocol, Event Begin Play indicates that the event starts running, Open Receive Socket indicates that the receiving socket is opened, OnReceive Byte indicates that the byte is received, and Game Instance indicates the game instance.

[0020] The tactile visualization module 22 is used to construct a tactile visualization virtual hand and apply pressure data rendering to the tactile visualization virtual hand.

[0021] The attitude synchronization module 23 is used to construct an attitude synchronization virtual hand and calculate the rotation amount through the Finterp To node based on the servo angle data and apply it to the attitude synchronization virtual hand to achieve synchronous rotation of the attitude synchronization virtual hand; The output layer 3 is used to display the tactile visualization virtual hand and the posture synchronization virtual hand.

[0022] Preferably, if Figure 3 As shown, the tactile visualization module 22 is specifically: Build a tactile visualization virtual hand blueprint based on the Unreal Engine. Create a dynamic material instance by using the Create Dynamic Material Instance blueprint node in the constructor of the tactile visualization virtual hand blueprint. Add a palm static mesh component to the tactile visualization virtual hand blueprint, customize the material blueprint, and bind the customized material to the material slot of the mesh.

[0023] Based on the arrangement of the pressure sensor array 12 , the UV coordinates of the tactile points are set in the material blueprint.

[0024] Based on the COMSOL simulation results, the pressure influence radius is determined.

[0025] Based on the UV coordinates and pressure influence radius, and by inputting center and edge color vectors, a custom Blueprint node was written to output haptic visualization results based on real-time pressure values. The center color was set to a vector of (1, 0, 0) and the edge color was set to a vector of (0, 0, 0) to create an effect that shifts from red in the center to white at the edges.

[0026] Based on the custom blueprint node, Gaussian blur difference calculation is performed on the pressure data received by the network communication module 21, and rendering is achieved.

[0027] Figure 4 Demonstrates the effects of tactile rendering in different situations. The single tactile point rendering effect is a single tactile point rendering effect displayed by setting the UV coordinates of the pressure point to (0.5, 0.5); the rectangular arrangement of multiple tactile point rendering effect is a multi-point superposition effect produced by setting the UV coordinates of the pressure points to (0.4, 0.4), (0.4, 0.6), (0.6, 0.4), and (0.6,0.6) respectively; the horizontal arrangement of multiple tactile point rendering effect is a multi-point superposition effect produced by setting the UV coordinates of the pressure points to (0.2, 0.5), (0.4, 0.5), (0.6, 0.5), and (0.8, 0.5) respectively; the diagonal arrangement of multiple tactile point rendering effect is a multi-point superposition effect produced by setting the UV coordinates of the pressure points to (0.2, 0.2), (0.4, 0.4), (0.6, 0.6), and (0.8, 0.8) respectively. The influence radius of the above four cases is set to 0.2.

[0028] Furthermore, if Figure 5 As shown, the posture synchronization module 23 is specifically: Build a pose-synchronized virtual hand blueprint based on the Unreal Engine and add a Poseable Mesh component to the blueprint.

[0029] Convert the static mesh of the Pose Sync Virtual Hand Blueprint into a skeletal mesh, and use the Skeleton Editor to build a skeleton tree and perform skinning.

[0030] Bind the Skeletal Mesh to the Poseable Mesh component added in the Pose Sync Virtual Hand Blueprint.

[0031] Based on the servo angle data, the rotation amount is calculated through the Finterp To node, and the Set Bone Rotation By Name node is used to achieve the synchronous rotation of the pose-synchronized virtual hand. Figure 5 In , Poseable Mesh represents a manipulable mesh.

[0032] The embodiments described above are merely descriptions of preferred implementations of the present invention and are not intended to limit the scope of the present invention. Without departing from the spirit of the present invention, various modifications and improvements made to the technical solutions of the present invention by ordinary technicians in this field should fall within the scope of protection determined by the claims of the present invention.

Claims

1. A tactile visualization system for manipulators based on Unreal Engine, characterized in that: It includes: Input layer, processing layer and output layer; The input layer includes a manipulator body, a pressure sensor array, a controller and a lower computer network communication module; the processing layer includes a network communication module, a tactile visualization module and a posture synchronization module; The pressure sensor array is provided on the manipulator body, and is used to obtain pressure data on the manipulator body. The controller is used to drive the rotation of the fingers of the manipulator body and obtain the steering gear angle data of the manipulator body. The lower computer network communication module is used to send the pressure data and the steering gear angle data to the network communication module; The tactile visualization module is used to construct a tactile visualization virtual hand and apply the pressure data rendering to the tactile visualization virtual hand; The posture synchronization module is used to construct a posture synchronization virtual hand and calculate the rotation amount through the Finterp To node based on the servo angle data and apply it to the posture synchronization virtual hand to achieve synchronous rotation of the posture synchronization virtual hand; The output layer is used to display the tactile visualization virtual hand and the posture synchronization virtual hand.

2. The tactile visualization system for manipulators based on Unreal Engine according to claim 1, characterized in that: The lower computer network communication module communicates with the network communication module based on the UDP network protocol.

3. The tactile visualization system for manipulators based on Unreal Engine according to claim 1, characterized in that: The network communication module is implemented based on the UDP Wrapper plug-in.

4. The tactile visualization system for manipulators based on Unreal Engine according to claim 1, characterized in that: The tactile visualization module is specifically: Building a tactile visualization virtual hand blueprint based on the Unreal Engine, creating a dynamic material instance using the Create Dynamic Material Instance blueprint node in the constructor of the tactile visualization virtual hand blueprint, adding a palm static mesh component to the tactile visualization virtual hand blueprint, customizing the material blueprint, and binding the custom material to the material slot of the mesh; Based on the arrangement of the pressure sensor array, set the UV coordinates of the tactile points in the material blueprint; Determine the pressure influence radius based on COMSOL simulation results; Based on the UV coordinates and pressure influence radius, and by inputting the center and edge color vectors, write a custom Blueprint node that outputs tactile visualization results based on real-time pressure values; Based on the custom blueprint node, Gaussian blur interpolation calculation is performed on the pressure data received by the network communication module, and rendering is achieved.

5. The tactile visualization system for manipulators based on Unreal Engine according to claim 1, characterized in that: The posture synchronization module is specifically: Build a pose-synchronized virtual hand blueprint based on the Unreal Engine and add a Poseable Mesh component to the blueprint; Convert the static mesh of the pose-synced virtual hand blueprint into a skeletal mesh, and use the bone editor to build the bone tree and perform skinning. Bind the Skeletal Mesh to the Poseable Mesh component added in the Pose Sync Virtual Hand Blueprint; Based on the servo angle data, the rotation amount is calculated through the Finterp To node, and the Set Bone RotationBy Name node is used to achieve the synchronous rotation of the posture-synchronized virtual hand.

Images