A Virtual Reality Visualization Method and System for Contact Surfaces Based on High-Density Flexible Tactile Sensors

Abstract

This invention discloses a virtual reality visualization method and system for contact surfaces based on a high-density flexible tactile sensor, belonging to the field of robot tactile perception and virtual reality human-computer interaction technology. The method acquires raw contact data output by a high-density flexible tactile sensor in real time, including the true three-dimensional position information of each tactile unit and its corresponding deformation or force information. It generates a three-dimensional point cloud with color attributes, encapsulates the point cloud into a message and publishes it through middleware. After coordinate system transformation, scaling, or remapping, the message is transmitted to a virtual reality device via a network protocol, ultimately rendering a three-dimensional morphology consistent with the geometric features of the sensor's contact surface. This invention achieves real-time three-dimensional visualization of tactile contact surfaces, intuitively presenting micron-level geometric morphology, and meeting the low-latency interaction requirements of scenarios such as precision assembly, remote operation teaching, and medical puncture.

Description

Technical Field

[0001] This invention belongs to the field of robot tactile perception and virtual reality human-computer interaction technology, and particularly relates to a virtual reality visualization method and system for contact surfaces based on high-density flexible tactile sensors. Background Technology

[0002] In the field of precision robotic manipulation, such as industrial assembly, remote control, and medical intervention, operators need to accurately perceive the contact state between the end effector and the target object to ensure the accuracy and safety of the operation. The main goal in this field is to capture contact information through sensor technology and translate it into a form understandable to the operator, thereby improving the efficiency and reliability of human-robot interaction. Currently, visual sensors or force feedback devices are widely used to assist in operation. These technologies play an important role in providing basic perceptual information, but they still present certain challenges when dealing with complex environments or micrometer-level details.

[0003] In existing technologies, high-density flexible tactile sensors have become an important sensing tool, enabling them to directly adhere to the surface of objects and acquire high-resolution contact data. However, current processing methods typically rely on indirect data conversion, such as calculating force distribution after capturing the displacement of marker points through images. This results in the final information failing to intuitively reflect the actual geometric characteristics of the contact surface. Especially in virtual reality environments, operators cannot obtain an immersive and realistic sensory experience, which increases the difficulty of operation and reduces efficiency.

[0004] To address the aforementioned issues, this invention proposes a virtual reality visualization method and system for contact surfaces based on high-density flexible tactile sensors. Summary of the Invention

[0005] To address the aforementioned technical problems, this invention provides a virtual reality visualization method for contact surfaces based on high-density flexible tactile sensors, comprising the following steps: Real-time acquisition of raw contact data output by high-density flexible tactile sensors, the raw contact data including the real three-dimensional position information of each tactile unit and the corresponding deformation information or force information; Based on the actual three-dimensional position information, a three-dimensional point cloud with color attributes is generated by combining the deformation information or force information. The 3D point cloud is encapsulated into a message and published via middleware; Subscribe to the message and perform coordinate system transformation, scaling or remapping processing on the 3D point cloud in the message to obtain a 3D point cloud that adapts to the display requirements of virtual reality devices; The processed 3D point cloud is transmitted to the virtual reality device via a network protocol; The virtual reality device renders a three-dimensional shape that matches the geometric features of the contact surface of the high-density flexible tactile sensor based on the processed three-dimensional point cloud.

[0006] Optionally, the true three-dimensional position information is calculated based on the calibrated three-dimensional position and real-time deformation of each tactile unit.

[0007] Optionally, generating a three-dimensional point cloud with color attributes based on the real three-dimensional position information and the deformation information or force information involves using the real three-dimensional position of each tactile unit as the spatial coordinates of the point cloud, and color-coding the point cloud based on the deformation information or force information.

[0008] Optionally, encapsulating the 3D point cloud into a message and publishing it through middleware means encapsulating the 3D point cloud into a standard point cloud message format and publishing it to a specified topic.

[0009] Optionally, the coordinate system transformation, scaling, or remapping process based on the 3D point cloud in the message involves extracting the center point of the 3D point cloud, scaling and adjusting the radial coordinates, and setting a base height to preserve the height difference of the contact surfaces.

[0010] Optionally, the network protocol is UDP, TCP, or WebSocket.

[0011] Optionally, the virtual reality device renders a three-dimensional shape from the processed three-dimensional point cloud, which renders the three-dimensional point cloud as geometric primitives selected from any one or more combinations of the following: scattered point cloud, triangular mesh, line segment, cylinder, sphere, normal disk, or triangular facet.

[0012] Optionally, the surface profile of the high-density flexible tactile sensor is a spherical segment, a cylindrical segment, a saddle shape, or other deformable curved surface, and the three-dimensional morphology is consistent with the real-time concave shape of the surface profile.

[0013] The present invention also provides a virtual reality visualization system for contact surfaces based on a high-density flexible tactile sensor, including a data acquisition module, a point cloud conversion module, a data publishing module, a data processing and transmission module, and a virtual reality rendering module; The data acquisition module is used to acquire raw contact data output by the high-density flexible tactile sensor in real time. The raw contact data includes the real three-dimensional position information of each tactile unit and the corresponding deformation information or force information. The point cloud conversion module is used to generate a three-dimensional point cloud with color attributes based on the real three-dimensional position information and the deformation information or force information. The data publishing module is used to encapsulate the 3D point cloud into a message and publish it through middleware; The data processing and transmission module is used to subscribe to the message, and perform coordinate system transformation, scaling or remapping processing on the 3D point cloud in the message to obtain a 3D point cloud that adapts to the display requirements of the virtual reality device, and transmit the processed 3D point cloud to the virtual reality device through a network protocol. The virtual reality rendering module is used by the virtual reality device to render a three-dimensional shape consistent with the geometric features of the contact surface of the high-density flexible tactile sensor based on the processed three-dimensional point cloud.

[0014] Compared with the prior art, the present invention has the following advantages and technical effects: The technical solution provided by this invention enables real-time 3D visualization of tactile contact surfaces, allowing operators to intuitively perceive the micron-level geometric features of the touched object in a virtual reality environment, such as slot boundaries, chamfers, steps, and other fine structures, significantly enhancing the immersiveness and realism of tactile feedback. This method supports low-latency transmission and rendering of high-density flexible tactile sensor data, ensuring smooth real-time interaction and effectively reducing the operator's cognitive burden and operational error rate. Simultaneously, this solution possesses good versatility and scalability, adaptable to various tactile sensors and virtual reality platforms, and suitable for scenarios with high requirements for contact geometry information, such as precision assembly, remote operation teaching, and medical puncture, thereby significantly improving the accuracy and efficiency of precision operations. Attached Figure Description

[0015] The accompanying drawings, which form part of this application, are used to provide a further understanding of this application. The illustrative embodiments and descriptions of this application are used to explain this application and do not constitute an undue limitation of this application. In the drawings: Figure 1 This is a schematic diagram of the method flow according to an embodiment of the present invention; Figure 2 This is a flowchart illustrating the color point cloud generation and publishing process according to an embodiment of the present invention.

[0016] Figure 3 This is a flowchart illustrating point cloud data processing and virtual reality transmission in an embodiment of the present invention. Detailed Implementation

[0017] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.

[0018] It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.

[0019] Example 1 like Figure 1 As shown, this embodiment provides a virtual reality visualization method for contact surfaces based on high-density flexible tactile sensors, including the following steps: Real-time acquisition of raw contact data output by high-density flexible tactile sensors, the raw contact data including the real three-dimensional position information of each tactile unit and the corresponding deformation information or force information; Based on the actual three-dimensional position information, a three-dimensional point cloud with color attributes is generated by combining the deformation information or force information. The 3D point cloud is encapsulated into a message and published via middleware; Subscribe to the message and perform coordinate system transformation, scaling or remapping processing on the 3D point cloud in the message to obtain a 3D point cloud that adapts to the display requirements of virtual reality devices; The processed 3D point cloud is transmitted to the virtual reality device via a network protocol; The virtual reality device renders a three-dimensional shape that matches the geometric features of the contact surface of the high-density flexible tactile sensor based on the processed three-dimensional point cloud.

[0020] Furthermore, the high-density flexible tactile sensor has an output frequency of 30 Hz to 1000 Hz and more than 200 tactile units.

[0021] Furthermore, the true three-dimensional position information is calculated based on the calibrated three-dimensional position and real-time deformation of each tactile unit.

[0022] Furthermore, the process of generating and publishing colored point clouds is as follows: Figure 2 As shown. Generating a 3D point cloud with color attributes based on the actual 3D position information and the deformation or force information involves using the actual 3D position of each tactile unit as the spatial coordinates of the point cloud, and color-coding the point cloud according to the deformation or force information.

[0023] Furthermore, encapsulating the 3D point cloud into a message and publishing it through middleware involves encapsulating the 3D point cloud into a standard point cloud message format and publishing it to a specified topic.

[0024] Furthermore, the point cloud data processing and virtual reality transmission process is as follows: Figure 3 As shown. Based on the 3D point cloud in the message, coordinate system transformation, scaling, or remapping is performed. This involves extracting the center point of the 3D point cloud, scaling and adjusting the radial coordinates, and setting the base height to preserve the height difference of the contact surfaces.

[0025] Furthermore, the network protocol is UDP, TCP, or WebSocket. Data packets contain location, size, and color information, with end-to-end latency less than 100 ms.

[0026] Furthermore, the virtual reality device renders a three-dimensional shape based on the processed three-dimensional point cloud, which renders the three-dimensional point cloud as geometric primitives selected from any one or more combinations of the following: scattered point cloud, triangular mesh, line segment, cylinder, sphere, normal disk or triangular facet.

[0027] Specifically, the rendering method can be selected from any one or more combinations of the following: Directly render scattered point clouds; Real-time surface reconstruction followed by rendering of triangular meshes; Convert point clouds into geometric primitives such as line segments, cylinders, spheres, normal disks, or triangular facets; Furthermore, the position and color of all geometric primitives change with the real-time concavity of the flexible surface, or they can be left unchanged. Additionally, the surface profile of the high-density flexible tactile sensor is a spherical segment, cylindrical segment, saddle shape, or other deformable surface, and the three-dimensional morphology is consistent with the real-time concavity morphology of the surface profile.

[0028] Furthermore, virtual reality devices use native applications such as Unity, Unreal Engine, Godot, or OpenXR as rendering frameworks and support 3D voxel rendering of flexible sensor surface deformation.

[0029] This method is applicable to scenarios involving transparent workpieces, mirror-like workpieces, or precision assembly, remote operation teaching, and high-precision tactile data acquisition with micron-level features.

[0030] Example 2 This embodiment proposes a virtual reality visualization system for contact surfaces based on a high-density flexible tactile sensor, including a data acquisition module, a point cloud conversion module, a data publishing module, a data processing and transmission module, and a virtual reality rendering module; The data acquisition module is used to acquire raw contact data output by the high-density flexible tactile sensor in real time. The raw contact data includes the real three-dimensional position information of each tactile unit and the corresponding deformation information or force information. The point cloud conversion module is used to generate a three-dimensional point cloud with color attributes based on the real three-dimensional position information and the deformation information or force information. The data publishing module is used to encapsulate the 3D point cloud into a message and publish it through middleware; The data processing and transmission module is used to subscribe to the message, and perform coordinate system transformation, scaling or remapping processing on the 3D point cloud in the message to obtain a 3D point cloud that adapts to the display requirements of the virtual reality device, and transmit the processed 3D point cloud to the virtual reality device through a network protocol. The virtual reality rendering module is used to render a three-dimensional shape consistent with the geometric features of the contact surface of the high-density flexible tactile sensor based on the processed three-dimensional point cloud using a virtual reality device.

[0031] This embodiment is implemented in an Ubuntu 22.04 and ROS2 Humble environment. First, raw data from a high-density flexible tactile sensor (400 tactile units) is acquired in real time through the sensor interface, including but not limited to the calibrated three-dimensional position, real-time deformation, and force information of each tactile unit. The data acquisition node initializes the sensor connection, sets the port to 9988 and configures the maximum queue size to 10. After waiting for frame data to arrive, these array information are extracted for each frame to ensure smooth data transmission without losing keyframes, while processing possible noise or outliers to improve data reliability.

[0032] Next, the extracted original positions and deformation variables are added point by point to calculate the true 3D point cloud of the current contact surface. Simultaneously, the point cloud is color-coded according to the force magnitude to highlight the concave features of the contact area; for example, the greater the force, the more reddish the color. This serves as auxiliary visual information without affecting geometric calculations. The encoding process considers force value normalization to avoid extreme values ​​affecting the overall visual effect. The point cloud publishing node encapsulates the calculation results into a standard point cloud message format, including x, y, and z coordinates and RGB color fields, and publishes it to the topic / tac3d / points3d. The publishing frequency is maintained at 60 Hz to ensure real-time performance. The message header sets the frame_id to "tac3d_sensor" for easy reference in subsequent coordinate systems, and a timestamp can be added to support synchronous processing.

[0033] The data processing and broadcasting nodes subscribe to and publish point cloud messages, performing coordinate system extraction, centering calculation, radial scaling, and height mapping on the point cloud. The specific process is as follows: extract the 3D coordinates of the point cloud, calculate the average center point of all points, scale and adjust the radial coordinates (x and y components) to match the scale of virtual reality, uniformly set the base height in virtual reality to raise the overall surface while preserving the actual height difference of the depressions, thereby generating a starting point coordinate set (fixed base height) and an ending point coordinate set (varying with the depressions). These processes ensure that the point cloud adapts to the requirements of virtual reality display without introducing additional transformations. Simultaneously, the color highlighting through X-axis displacement makes the depressions more visually distinguishable. Downsampling or filtering can also be included in the processing to optimize performance, making it suitable for different hardware configurations.

[0034] After processing, the start and end coordinates are packaged into a standardized data structure and broadcast to the virtual reality device via the UDP protocol (using port 9001). The data packet mainly contains point cloud position, size, and color information, supports real-time transmission, and includes rendering shape identifiers such as points, triangles, or cylinders to allow the virtual reality client to flexibly select geometric primitives. The data packet format is designed to be compact to reduce latency, and checksums can be added to ensure transmission integrity.

[0035] The virtual reality terminal receives UDP data packets based on the Unity engine and XR technology stack. After parsing and format verification, it dynamically constructs a mesh or uses components such as LineRenderer and Particle System to render 3D point clouds, achieving low-latency reconstruction and visualization of contact surfaces. The specific rendering process includes data stream reception, integrity verification, prefab core logic construction, and multi-morphological 3D reconstruction to ensure high stability. For example, it renders 400 short line segments to reflect height differences and supports user interaction such as zooming or rotating. The measured end-to-end latency is approximately 27 ms. In practical applications, such as inserting a SIM card into a phone's frame, the 0.06 mm step boundary, R0.15 chamfer, and spring deformation can be clearly seen in VR. Through 20 experiments, the operator can determine that the insertion is in place 90% of the time, which is a 41% improvement in success rate compared to traditional solutions. This application verifies the effectiveness of the system in precision assembly scenarios and can be extended to similar operating environments to improve overall efficiency.

[0036] The system can be scaled to alternative implementations, such as replacing the sensor with other high-density flexible tactile sensors (300–10,000 tactile units), while maintaining the data acquisition, point cloud computing, color encoding, publishing, processing, and UDP transmission processes. Geometric rendering can utilize primitives such as spatial points, cylinders, normal disks, or triangular facets. Middleware can be plugged in with shared memory, ZeroMQ, or DDS to optimize data flow. Broadcast protocols can be TCP, WebSocket, or gRPC to adapt to different network conditions. Rendering engines can be native applications of Unity, Unreal Engine, Godot, or OpenXR. Color encoding can be based on any scalar field such as force, deformation, or temperature. The system frequency range is 30 Hz–1000 Hz, with latency relaxed to 200 ms. Rendered images can track the viewpoint or remain stationary. Applications extend to all situations requiring realistic contact surface geometry visualization, such as medical soft tissue punctures, automotive wiring harness insertion and removal, and aerospace connector assembly. These variations maintain a consistent core data processing flow, adjusting only specific components to adapt to diverse needs, thereby expanding the system's applicability without major modifications.

[0037] The above are merely preferred embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for virtual reality visualization of a contact surface based on a high-density flexible tactile sensor, characterized in that, Includes the following steps: Real-time acquisition of raw contact data output by high-density flexible tactile sensors, the raw contact data including the real three-dimensional position information of each tactile unit and the corresponding deformation information or force information; Based on the actual three-dimensional position information, a three-dimensional point cloud with color attributes is generated by combining the deformation information or force information. The 3D point cloud is encapsulated into a message and published via middleware; Subscribe to the message and perform coordinate system transformation, scaling or remapping processing on the 3D point cloud in the message to obtain a 3D point cloud that adapts to the display requirements of virtual reality devices; The processed 3D point cloud is transmitted to the virtual reality device via a network protocol; The virtual reality device renders a three-dimensional shape that matches the geometric features of the contact surface of the high-density flexible tactile sensor based on the processed three-dimensional point cloud.

2. The method of claim 1, wherein, The real three-dimensional position information is calculated based on the calibrated three-dimensional position and real-time deformation of each tactile unit.

3. The method of claim 1, wherein, The process of generating a three-dimensional point cloud with color attributes based on the actual three-dimensional position information and the deformation or force information involves using the actual three-dimensional position of each tactile unit as the spatial coordinates of the point cloud and color encoding the point cloud according to the deformation or force information.

4. The method of claim 1, wherein, Encapsulating a 3D point cloud into a message and publishing it through middleware involves encapsulating the 3D point cloud into a standard point cloud message format and publishing it to a specified topic.

5. The method of claim 1, wherein, The process of performing coordinate system transformation, scaling, or remapping based on the 3D point cloud in the message involves extracting the center point of the 3D point cloud, scaling and adjusting the radial coordinates, and setting the base height to preserve the height difference of the contact surfaces.

6. The method of claim 1, wherein, The network protocol is UDP, TCP, or WebSocket.

7. The method of claim 1, wherein, The rendering of a three-dimensional shape by a virtual reality device based on a processed three-dimensional point cloud involves rendering the three-dimensional point cloud into geometric primitives selected from one or more combinations of the following: scattered point cloud, triangular mesh, line segment, cylinder, sphere, normal disk, or triangular facet.

8. The method of claim 1, wherein, The surface profile of the high-density flexible tactile sensor is a spherical segment, cylindrical segment, saddle shape, or other deformable curved surface, and the three-dimensional morphology is consistent with the real-time concave shape of the surface profile.

9. A high-density flexible tactile sensor based contact surface virtual reality visualization system, characterized in that, It includes a data acquisition module, a point cloud conversion module, a data publishing module, a data processing and transmission module, and a virtual reality rendering module; The data acquisition module is used to acquire raw contact data output by the high-density flexible tactile sensor in real time. The raw contact data includes the real three-dimensional position information of each tactile unit and the corresponding deformation information or force information. The point cloud conversion module is used to generate a three-dimensional point cloud with color attributes based on the real three-dimensional position information and the deformation information or force information. The data publishing module is used to encapsulate the 3D point cloud into a message and publish it through middleware; The data processing and transmission module is used to subscribe to the message, and perform coordinate system transformation, scaling or remapping processing on the 3D point cloud in the message to obtain a 3D point cloud that adapts to the display requirements of the virtual reality device, and transmit the processed 3D point cloud to the virtual reality device through a network protocol. The virtual reality rendering module is configured to render a three-dimensional topography consistent with the geometry of the high-density flexible tactile sensor contact surface according to the processed three-dimensional point cloud via a virtual reality device.

Images