Point cloud driven arm-hand operating system

Abstract

This invention discloses a grouped point cloud-driven arm-hand operating system. The arm-hand operating system includes a VR-based teleoperation data acquisition module, an environmental perception and encoding unit, and a two-layer feature fusion diffusion denoising model. Specifically: the VR-based teleoperation data acquisition module consists of a posture recognizer, a posture mapper, and a data storage module; the environmental perception and encoding unit consists of a point cloud grouper, a point cloud encoding module, and a state integrator; and the two-layer feature fusion diffusion denoising model consists of a two-layer feature fusion module and a diffusion denoising network. This arm-hand operating system can respond to environmental changes in real time, achieving comprehensive scene perception while quickly formulating task execution plans. This data acquisition-algorithm deployment collaborative framework effectively solves the challenge of balancing cost, speed, and accuracy in intelligent arm-hand operation, providing a cost-effective solution for embodied intelligence, robotic operations, and other fields.

Description

Technical Field

[0001] This invention belongs to the field of robot intelligent control and human-computer interaction, and relates to an arm and hand operating system, specifically an arm and hand operating system driven by grouped point clouds that can respond to environmental changes in real time. Background Technology

[0002] In the process of robot interaction with the environment, the robotic arm and dexterous hand, as end effectors that directly contact the manipulated object, directly restrict the level of machine intelligence due to their flexibility and operability. Taking grasping tasks as an example, traditional robotic arm operating systems decompose the task into three stages: scene perception, grasping detection, and grasping execution. Scene perception is used for object recognition and pose estimation, the grasping detection module is used to calculate the grasping pose of the dexterous hand in a given coordinate system, and the grasping execution module is used to calculate the degree of each joint of the arm and control the robotic arm to perform the grasping action. However, object pose recognition is affected by factors such as object material and lighting, especially in unstructured working environments such as outdoor robotic arm operations. Studies have shown that changes in scene brightness can increase the pose estimation error based on RGB-D by about 15-20%, thus affecting the accuracy of the robotic arm operation. At the same time, when the environment changes dynamically, this operation requires the scene perception module to be re-executed, resulting in poor real-time environmental response capabilities and seriously affecting the safety and intelligence of the system.

[0003] Current advanced solutions largely rely on data-driven end-to-end algorithms to improve the system's real-time response to the environment. This approach uses images or point clouds as input, with the algorithm directly outputting the desired joint angles of the arm and hand. However, using images as input ignores depth information, resulting in a success rate of less than 50% in complex environments. While using point clouds as input significantly improves the success rate of tasks involving depth, the large data volume requires significant computing power, and the average deployment time is 70% longer than with image data, making real-time performance difficult to guarantee. Furthermore, for highly dexterous hands, acquiring accurate and continuous operational data remains a major challenge.

[0004] This invention's innovative arm-hand operating system can respond to environmental changes in real time, achieving comprehensive scene perception while rapidly formulating task execution plans. Accurate and continuous operational data is collected using virtual reality glasses, and a grouped point cloud encoder is designed to enhance scene perception capabilities and speed. Combined with a diffusion denoising algorithm based on a dual-feature encoder, the system reduces deployment time by 35% and improves environmental response speed by 60% compared to existing end-to-end algorithms. Summary of the Invention

[0005] To address the issues of slow response and poor perception in traditional robotic arm operations in complex scenarios, especially in unstructured environments such as outdoor robotic operations, where existing systems struggle to balance real-time response and rapid deployment, this invention provides a grouped point cloud-driven robotic arm operating system. This system can respond to environmental changes in real time, achieving comprehensive scene perception while rapidly formulating task execution plans. The invention utilizes virtual reality glasses to collect accurate and continuous operational data and designs a grouped point cloud encoder to enhance scene perception and speed. Combined with a diffusion denoising algorithm based on a dual-feature encoder, the system reduces deployment time by 35% and improves environmental response speed by 60% compared to existing end-to-end algorithms. This data acquisition-algorithm deployment collaborative framework effectively solves the challenge of balancing cost, speed, and accuracy in intelligent robotic arm operations, providing a cost-effective solution for embodied intelligence and robotic operations.

[0006] The objective of this invention is achieved through the following technical solution:

[0007] A grouped point cloud-driven arm-hand operating system includes a VR-based teleoperation data acquisition module, an environmental perception and coding unit, and a two-layer feature fusion diffusion denoising model, wherein:

[0008] The VR-based teleoperation data acquisition module consists of an attitude recognizer, an attitude mapper, and a data storage module.

[0009] The posture recognizer is used to capture the operator's hand key points, head posture and wrist posture in real time through VR devices, and stream this data to the server.

[0010] The posture mapper is used to map the operator's movements onto the robot: the relative position of the robot's end effector and head is matched with the relative position of the human wrist and head. A closed-loop inverse kinematics algorithm is used to convert the operator's wrist position into the position of the robot's end effector, and the input is smoothed by filtering to enhance stability. For multi-finger dexterous hands, the dex-retargeting library is used to convert human hand key points into robot joint angle commands. By optimizing the vectors from the wrist to the fingertips and between the fingertips, the retargeting problem is transformed into an optimization problem, minimizing the difference between human and robot hand vectors and ensuring the temporal consistency of the movements.

[0011] The data storage module is used to save the data of each joint of the robot and the point cloud collected by the head, record the start and end time of each demonstration, and package them into a zarr compressed file.

[0012] The environmental perception and coding unit consists of a point cloud grouper, a point cloud coding module, and a state integrator.

[0013] The point cloud grouper is used to perform farthest point sampling (FPS) on the current frame point cloud from the robot's perspective. After obtaining a specified k global points, it performs sphere query grouping to obtain the sphere neighbor points of each global point, thereby obtaining a grouped point cloud family, which is regarded as a local point cloud family.

[0014] The point cloud encoding module is used to extract features from global and local point clouds. It consists of a global point cloud encoder and a local point cloud encoder. The global point cloud encoder uses a three-layer linear fully connected structure to extract the relationship between points. Layer normalization and ReLU activation are used to improve stability and nonlinear expression. Then, max pooling is used to eliminate the influence of the point cloud arrangement order. Finally, the global point cloud features are obtained through a fully connected layer. The local point cloud encoder uses a hierarchical convolutional structure to extract spatial features. First, one-dimensional convolution and ReLU activation are performed on the local point cloud. On this basis, high-level features are extracted step by step through a convolutional module with four stacked residual structures. The outputs of the four convolutional modules are concatenated. Then, one-dimensional convolution and max pooling are used to aggregate information at different scales to finally obtain a local point cloud feature representation with spatial distribution awareness.

[0015] The state integrator is used to combine the robot's state information at the same moment with the denoised steps and point cloud features to obtain state features. To ensure the consistency of the state features, MLP and sinusoidal position markers are used to process and splice the robot's joint angle data and denoised steps respectively. Then, they are combined with global point cloud features and local point cloud features respectively to obtain global state features and local state features that can reflect the current environmental state.

[0016] The two-layer feature fusion diffusion denoising model consists of a two-layer feature fusion module and a diffusion denoising network.

[0017] The dual-layer feature fusion module receives global and local state features from the environment perception and coding unit, and maps them into two sets of feature parameters through the Mish activation function and a one-dimensional convolutional layer, respectively. , ,in , These represent the calculated global and local linear feature parameters, which are controlled by linear transformations to scale and offset the activated input data in the channel dimension.

[0018] The diffusion denoising network is a Unet network with a feature fusion module at its core. The Unet network consists of an encoder, an intermediate layer, and a decoder. The encoder extracts features by downsampling layer by layer to extract multi-scale feature representations of the input. The intermediate layer connects the encoder and the decoder. The decoder upsamples step by step to restore the compressed features to the original resolution and outputs the final prediction result.

[0019] A method for implementing grouped point cloud-driven arm manipulation based on the above system includes the following steps:

[0020] Step 1: VR-based teleoperation data acquisition:

[0021] Step 11: The human operator puts on the VR headset and is under the same IP as the robot;

[0022] After steps one and two are completed, the image from the robot's head camera will be transmitted to the VR, and at the same time, the posture recognizer will recognize the operator's head, wrist and hand postures.

[0023] Step 13: The posture mapper maps the operator's pose to the robot's operating space, enabling remote operation of the robot.

[0024] Step 14: After the operator completes the task demonstration, the data storage module stores the data of each joint of the robot and the point cloud collected by the head, and packages them into a compressed file for later use.

[0025] Step 2, Environmental Perception and Coding:

[0026] Step 2: After reading the collected data, use a point cloud grouper to group the point cloud information of each input frame to obtain a global point cloud family and a local point cloud family.

[0027] Step 22: Use the global point cloud encoder and the local point cloud encoder in the point cloud encoder to process the two families of point clouds respectively, and obtain the feature information of the two families of point clouds.

[0028] Steps 2 and 3: Using a state integrator, the robot state information and the number of denoised steps are concatenated with the point cloud feature information to obtain the global state features and local state features of each frame.

[0029] Step 3: Two-layer feature fusion and diffusion denoising:

[0030] Step 3: During the training phase, the two-layer feature fusion diffusion denoising model samples a real action sequence and adds multi-step Gaussian noise to it, degenerating it into an approximately random sequence.

[0031] Step 32: Randomly select a diffusion step, and input the noisy action sequence corresponding to this step along with the two state features calculated by the two-layer feature fusion module into the diffusion denoising network. The training objective is to minimize the difference between the diffusion denoising network prediction and the actual action, so that the two-layer feature fusion diffusion denoising model can gradually learn how to generate reasonable actions under any noise level.

[0032] Step 3: After sufficient training, the two-layer feature fusion diffusion denoising model can start from pure noise and gradually "de-noise" by combining environmental and state information during actual deployment, generating an executable high-quality action sequence to complete complex operation tasks.

[0033] Compared with the prior art, the present invention has the following advantages:

[0034] 1. This invention has low cost and portability. Compared with the traditional data acquisition method based on electromagnetic gloves, the arm and hand operating system of this invention only requires a VR and an Ubuntu computer, reducing hardware costs by 60%. Moreover, it can be quickly migrated to different types of robots and dexterous hands by changing the inverse kinematics algorithm module.

[0035] 2. This invention possesses strong environmental perception capabilities. The system uses global-local grouped point clouds as environmental input and designs two different encoders to process them. Compared to existing environmental input and encoding methods, this invention enables the system to perceive depth information while ensuring rapid environmental perception, reducing deployment time by 35%.

[0036] 3. This invention has high precision and real-time response capability to the environment. The system uses a two-layer feature fusion diffusion denoising algorithm to fully integrate environmental information into the system decision-making, enabling the robot to quickly and accurately perceive changes in the environment with an accuracy rate of up to 80% and a response time of less than 1 second.

[0037] 4. This invention uses virtual reality glasses to collect environmental data and calculates the robot's joint positions by recognizing the positions of human hand fingers and the inverse kinematics of the robotic arm. The inverse kinematics calculation frequency can reach 100Hz, with an accuracy rate exceeding 90%. Combined with a grouped point cloud diffusion denoising algorithm, the system can eliminate 50% of noise points in the arm-hand working environment, increasing training speed by 30%. Attached Figure Description

[0038] Figure 1 This is a general framework diagram of the arm and hand operating system of the present invention;

[0039] Figure 2 This is a flowchart of the data acquisition method of the present invention;

[0040] Figure 3 This is a diagram of the environmental perception and coding unit architecture of the present invention;

[0041] Figure 4 This is an architectural diagram of two point cloud encoders of the present invention;

[0042] Figure 5 This is a diagram of the architecture of the two-layer feature fusion diffusion denoising model of the present invention;

[0043] Figure 6 This is a flowchart illustrating the specific steps of the present invention. Detailed Implementation

[0044] The technical solution of the present invention will be further described below with reference to the accompanying drawings, but it is not limited thereto. Any modifications or equivalent substitutions to the technical solution of the present invention that do not depart from the spirit and scope of the technical solution of the present invention should be covered within the protection scope of the present invention.

[0045] To address the problems of long deployment time, poor real-time response, and low execution accuracy in existing arm and hand manipulation methods in unstructured scenarios, this invention provides a grouped point cloud-driven arm and hand operating system, such as... Figure 1 As shown, the arm-hand operating system includes a VR-based teleoperation data acquisition module, an environmental perception and coding unit, and a two-layer feature fusion diffusion denoising model, wherein:

[0046] The VR-based teleoperation data acquisition module uses a head-mounted VR headset as the sensing device. The image captured by the robot's head camera is projected into the VR system, and a pose recognizer calculates the wrist's pose relative to the head and the feature vectors of the fingertips. Next, a pose mapper maps this data to the robot's joint space, recording the robot's joint angles and the camera's image at regular intervals. Finally, a data storage module saves and packages the data required by the algorithm. This module enables humans to teleoperate the robot and collect task data from a first-person perspective. The entire control loop operates at a frequency of 60Hz to ensure low system latency and data continuity.

[0047] The environment perception and coding unit mainly consists of a point cloud grouper, a point cloud coding module, and a state integrator. This design can not only remove noise and irrelevant point clouds, but also follow the principle of adaptive resource allocation, enabling comprehensive and rapid perception of the environment and helping the system to respond to environmental changes in real time.

[0048] The dual-layer feature fusion diffusion denoising model consists of a dual-layer feature fusion module and a diffusion denoising network. This model receives encoded environmental information and integrates global and local environmental features into the diffusion denoising model in a hierarchical manner, enabling real-time responses to environmental changes while ensuring the continuity of robot actions. After approximately 30 minutes of training, the model allows the robot to achieve a success rate of >80% and a response speed of <1 second in unstructured working environments.

[0049] In this invention, the VR-based teleoperation data acquisition module consists of a posture recognizer, a posture mapper, and a data storage module. The posture recognizer captures the operator's hand key points, head posture, and wrist posture in real time through the VR device and streams this data to the server. The VR device's hand tracking integrates multi-sensor data, making it more stable and accurate than traditional visual tracking. To ensure the operator can complete the task smoothly, the images captured by the robot's head camera are synchronized to the VR headset. Next, the posture mapper operates at a frequency of 100Hz, mapping the operator's movements onto the robot. Specifically, the relative position of the robot's end effector to its head is matched with the relative position of the human wrist to its head, and a closed-loop inverse kinematics (CLIK) algorithm is used to convert the operator's hand root position into the position of the robot arm's end effector. Input smoothing is achieved through filtering to enhance stability. For multi-fingered dexterous hands, the dex-retargeting library is used to convert human hand key points into robot joint angle commands. By optimizing the vectors from the wrist to the fingertips and between the fingertips, the retargeting problem is transformed into an optimization problem, minimizing the difference between human and robot hand vectors and ensuring temporal consistency of movements. Finally, the data storage module saves the robot's joint data and the point cloud data collected from the head at a frequency of 60Hz, records the start and end times of each demonstration, and packages them into a single ZARR compressed file. This teleoperation data acquisition method has the advantages of high precision, high stability, and strong real-time performance. By fusing multi-sensor data through VR devices, stable capture of the operator's hand and head postures is achieved. Closed-loop inverse kinematics and optimization algorithms are used to accurately map the movements onto the robot, significantly improving the smoothness of operation and the accuracy of movement matching. Simultaneously, the data storage module efficiently records the robot's state and point cloud data, providing complete and reliable data support for subsequent analysis.

[0050] In this invention, the workflow of the VR-based teleoperation data acquisition module is as follows: A human operator puts on a VR headset and is placed under the same IP as the robot. Once ready, the image from the robot's head camera is transmitted to the VR headset, and the posture recognizer identifies the operator's head, wrist, and hand postures. Next, the posture mapper maps the operator's postures to the robot's operating space, enabling correct teleoperation of the robot. After the operator completes the task demonstration, the data storage module stores the robot's joint data and the point cloud collected from the head, packaging them into a compressed file for subsequent use.

[0051] In this invention, the environmental perception and coding unit consists of a point cloud grouper, a point cloud coding module, and a state integrator, such as... Figure 3As shown, the modules are tightly integrated, enabling high accuracy and real-time performance in environmental perception within unstructured scenarios. First, the point cloud grouper performs farthest point sampling (FPS) on the current frame point cloud from the robot's perspective. After obtaining k specified global points, it performs sphere query grouping to obtain the sphere's nearest neighbors for each global point, thus creating a grouped point cloud family, which is considered a local point cloud family. The point cloud encoder consists of a global point cloud encoder and a local point cloud encoder, used to extract features from the global and local point clouds. For example... Figure 4 As shown, the global point cloud is relatively small in number but widely distributed. The encoder employs a three-layer linear fully connected structure to extract the relationships between points, combined with layer normalization and ReLU activation to improve stability and nonlinear expressiveness. Subsequently, max pooling is used to eliminate the influence of the point cloud arrangement order, and finally, a single fully connected layer is used to map the global point cloud features. Local point clouds are characterized by their large number and complex structure; therefore, the encoder design adopts a hierarchical convolutional structure to fully extract their spatial features. First, one-dimensional convolution and ReLU activation are performed on the local point cloud. Based on this, four convolutional modules with stacked residual structures are used to progressively extract high-level features. Each module includes convolution, activation, and max pooling operations, effectively enhancing the network's nonlinear expressiveness and compressing redundant information. The outputs of the four modules are concatenated, and then one-dimensional convolution and max pooling are used to aggregate information at different scales, ultimately obtaining a local point cloud feature representation with spatial distribution awareness. The state integrator is mainly used to combine the robot's state information at the same moment with the denoised steps and point cloud features to obtain state features. To ensure feature consistency, simple MLP and sinusoidal position markers are used to process and concatenate the robot joint angle data and denoised steps, respectively. These are then combined with global and local point cloud features to obtain global and local state features reflecting the current environmental state, which are then used in a two-layer feature fusion diffusion denoising algorithm. This unit is based on two main ideas: modular grouping and adaptive resource allocation. By grouping the point cloud data and designing encoders of different sizes for point clouds of different volumes, the number of parameters is only 0.3M. This reduces interference from irrelevant environmental information and accelerates environmental processing, enabling high-speed and accurate environmental perception.

[0052] The workflow of the environmental perception and encoding unit of this invention is as follows: For each frame of input point cloud and robot state information, the point cloud is first grouped using a point cloud grouper to obtain a global point cloud family and a local point cloud family. Then, the global point cloud encoder and local point cloud encoder in the point cloud encoder process the two families of point clouds respectively to obtain feature information for the two types of point clouds. Finally, the processed robot state information and a preset number of denoising steps are concatenated with the point cloud feature information to obtain the global state features and local state features for each frame, which are then used by the two-layer feature fusion diffusion denoising algorithm.

[0053] The two-layer feature fusion diffusion denoising model of this invention consists of a two-layer feature fusion module and a diffusion denoising network, wherein the two-layer feature fusion module is the core component of this model, such as... Figure 5 The two-layer feature fusion module receives global and local state features from the environment perception and encoding unit, and maps them into two sets of feature parameters through the Mish activation function and a one-dimensional convolutional layer, respectively. , These two sets of feature parameters control the scaling and offset of the activated input data in the channel dimension through linear transformations, ensuring the model remains robust under different environments. , This represents the calculated global and local linear characteristic parameters. Indicates weight, This indicates bias. Because local features are more complex, they participate in the adjustment first to achieve deeper fusion, while global features participate in the fusion last in the module. This approach allows environmental features to adjust the data at the channel level without significantly increasing the number of parameters, and also allows the model to consider both global and local environmental features separately, thus responding to environmental changes in real time. The main structure of the diffusion denoising network is a Unet network with a feature fusion module at its core, as shown in the architecture below. Figure 5 As shown, it inputs Gaussian noise and robot state features into a Unet network composed of feature fusion modules to predict the robot's next action. The Unet network consists of an encoder, intermediate layers, and a decoder. All feature channel changes are implemented by a two-layer feature fusion module. The encoder extracts features by downsampling layer by layer, extracting multi-scale feature representations of the input. The intermediate layer connects the encoder and decoder, has the smallest spatial resolution but the most feature channels, representing global context information. The decoder upsamples progressively, restoring the compressed features to the original resolution and outputting the final prediction result. The network is generated starting from pure Gaussian noise, combining two state features, and progressively denoising through a trained Unet network to restore action data that conforms to human operation. This generation process is essentially sampling the action distribution, ensuring action diversity while also giving the model strong stability and faster convergence: only 74M parameters, and convergence in 30 minutes. This model combines a two-layer feature fusion and diffusion denoising action generation method, innovatively utilizing global and local point cloud features to adjust actions at the channel level, achieving real-time response to environmental changes. The model has a lightweight structure and high training efficiency, and combines action diversity with generation stability, making it suitable for complex operation tasks.

[0054] In this invention, the training process of the two-layer feature fusion diffusion denoising model is as follows: During the training phase, the model samples a real action sequence and adds multi-step Gaussian noise to it, degenerating it into an approximately random sequence. Then, a diffusion step is randomly selected, and the noisy action sequence corresponding to that step, along with two state features calculated by the two-layer feature fusion module, is input into Unet. The training objective is to minimize the difference between Unet's prediction and the real action, thereby enabling the model to gradually learn how to generate reasonable actions under arbitrary noise levels. After sufficient training, the model can be deployed in practice, starting from pure noise and gradually "denoising" by combining environmental and state information to generate executable, high-quality action sequences to complete complex operational tasks.

[0055] The specific steps for implementing the grouped point cloud-driven arm and hand operating system are as follows:

[0056] Step 1: A human operator uses VR to remotely operate the robot and complete the assigned tasks, recording approximately 30 sets of demonstration data.

[0057] Step 2: Train the diffusion denoising model of the integrated environmental perception and coding unit using human teleoperation data, observe the loss convergence, and save the training results.

[0058] Step 3: Deploy the trained model on the robot and control the robot's movement according to the model's output. Observe the robot's performance in completing the task. Fine-tune the model training results as needed, such as by adding demonstration data or increasing training time, to ensure the robot can accurately and safely complete the designated task.

[0059] The grouped point cloud-driven arm-hand operating system designed in this invention can be applied to robotic assembly line operation scenarios. In factory packaging tasks, a human operator first simulates the robot in VR to collect data on approximately 20 sets of items to be packaged. After about 75 training iterations, the model stabilizes. The model is then deployed in the robot, automatically recognizing the positions of items and boxes on the workbench and mimicking the operator's actions to place the items into the boxes. Furthermore, the system is not limited to robot platforms; it is also applicable to single-arm and single-dexterity hand platforms. Experiments show that the model achieves a success rate of up to 80%, a 140% improvement over traditional models, and can respond to environmental changes within 1 second, such as object movement or failed item grasping, representing a 60% improvement in response speed compared to traditional models.

Claims

1. A grouped point cloud-driven arm and hand operating system, characterized in that... The arm-hand operating system includes a VR-based teleoperation data acquisition module, an environmental perception and coding unit, and a two-layer feature fusion diffusion denoising model, wherein: The VR-based teleoperation data acquisition module consists of an attitude recognizer, an attitude mapper, and a data storage module. The posture recognizer is used to capture the operator's hand key points, head posture and wrist posture in real time through VR devices, and stream this data to the server. The posture mapper is used to map the operator's actions onto the robot; The data storage module is used to save the data of each joint of the robot and the point cloud collected by the head, record the start and end time of each demonstration, and package them into a zarr compressed file. The environmental perception and coding unit consists of a point cloud grouper, a point cloud coding module, and a state integrator. The point cloud grouper is used to sample the farthest point of the current frame point cloud from the robot's perspective. After obtaining a specified k global points, it performs a ball query grouping to obtain the ball neighbor points of each global point, thereby obtaining a grouped point cloud family, which is regarded as a local point cloud family. The point cloud encoding module is used to extract features from the global point cloud and the local point cloud. The state integrator is used to combine the robot's state information at the same moment with the denoised steps and point cloud features to obtain state features. To ensure the consistency of the state features, MLP and sinusoidal position markers are used to process and splice the robot's joint angle data and denoised steps respectively. Then, they are combined with global point cloud features and local point cloud features respectively to obtain global state features and local state features that can reflect the current environmental state. The two-layer feature fusion diffusion denoising model consists of a two-layer feature fusion module and a diffusion denoising network. The dual-layer feature fusion module receives global and local state features from the environment perception and coding unit, and maps them into two sets of feature parameters through the Mish activation function and a one-dimensional convolutional layer, respectively. , ,in , These represent the calculated global and local linear feature parameters, which are controlled by linear transformations to scale and offset the activated input data in the channel dimension. The diffusion denoising network is a Unet network with a feature fusion module at its core. The Unet network consists of an encoder, an intermediate layer, and a decoder. The encoder extracts features by downsampling layer by layer to extract multi-scale feature representations of the input. The intermediate layer connects the encoder and the decoder. The decoder upsamples step by step to restore the compressed features to the original resolution and outputs the final prediction result.

2. The grouped point cloud-driven arm and hand operating system according to claim 1, characterized in that... The VR-based teleoperation data acquisition module uses a head-mounted VR headset as the sensing device.

3. The grouped point cloud-driven arm and hand operating system according to claim 1, characterized in that... The specific steps by which the posture mapper maps the operator's movements to the robot are as follows: The relative position of the robot's end effector and head is matched with the relative position of the human wrist and head. A closed-loop inverse kinematics algorithm is used to convert the operator's wrist position into the position of the robot's end effector, and the input is smoothed by filtering to enhance stability. For multi-finger dexterous hands, the dex-retargeting library is used to convert human hand key points into robot joint angle commands. By optimizing the vectors from the wrist to the fingertips and between the fingertips, the retargeting problem is transformed into an optimization problem, minimizing the difference between human and robot hand vectors and ensuring the temporal consistency of the movements.

4. The arm-hand operating system driven by grouped point cloud according to claim 1, characterized in that... The point cloud encoding module consists of a global point cloud encoder and a local point cloud encoder. The global point cloud encoder uses a three-layer linear fully connected structure to extract the relationship between points. Layer normalization and ReLU activation are used to improve stability and nonlinear expression. Then, max pooling is used to eliminate the influence of the point cloud arrangement order. Finally, the global point cloud features are obtained through a fully connected layer mapping. The local point cloud encoder uses a hierarchical convolutional structure to extract spatial features. First, one-dimensional convolution and ReLU activation are performed on the local point cloud. On this basis, high-level features are extracted step by step through a convolutional module with four stacked residual structures. The outputs of the four convolutional modules are concatenated. Then, one-dimensional convolution and max pooling are used to aggregate information at different scales to finally obtain a local point cloud feature representation with spatial distribution perception capability.

5. The arm-hand operating system driven by grouped point clouds according to claim 1, characterized in that... To ensure the consistency of state features, the state integrator uses MLP and sinusoidal position markers to process and concatenate the robot joint angle data and denoised step count, respectively. Then, it combines them with global point cloud features and local point cloud features to obtain global state features and local state features that can reflect the current environmental state.

6. The arm-hand operating system driven by grouped point cloud according to claim 1, characterized in that... The workflow of the VR-based teleoperation data acquisition module is as follows: The human operator puts on the VR headset and is under the same IP as the robot; after preparation, the image from the robot's head camera is transmitted to the VR, and at the same time, the posture recognizer identifies the operator's head, wrist and hand postures. The posture mapper maps the operator's pose to the robot's operating space, enabling remote operation of the robot. After the operator completes the task demonstration, the data storage module stores the data of each joint of the robot and the point cloud collected by the head, and packages them into a compressed file for later use.

7. The arm-hand operating system driven by grouped point clouds according to claim 1, characterized in that... The workflow of the environment perception and coding unit is as follows: For the point cloud and robot state information of each input frame, the point cloud is first grouped using a point cloud grouper to obtain a global point cloud family and a local point cloud family; then, the global point cloud encoder and the local point cloud encoder are used to process the two families of point clouds respectively to obtain the feature information of the two types of point clouds; finally, the processed robot state information and the preset number of denoising steps are concatenated with the point cloud feature information to obtain the global state features and local state features of each frame, which are used by the two-layer feature fusion diffusion denoising algorithm.

8. The grouped point cloud-driven arm and hand operating system according to claim 1, characterized in that... The training process of the dual-layer feature fusion diffusion denoising model is as follows: During the training phase, the model samples a real action sequence and adds multi-step Gaussian noise to it, degenerating it into an approximately random sequence; then, a diffusion step is randomly selected, and the noisy action sequence corresponding to this step, along with the two state features calculated by the dual-layer feature fusion module, is input into Unet; the goal of training is to minimize the difference between Unet prediction and real action, so that the model gradually learns how to generate reasonable actions under arbitrary noise levels.

9. A method for implementing grouped point cloud-driven arm manipulation based on the system described in any one of claims 1-8, characterized in that... The method includes the following steps: Step 1: VR-based teleoperation data acquisition: Step 11: The human operator puts on the VR headset and is under the same IP as the robot; After steps one and two are completed, the image from the robot's head camera will be transmitted to the VR, and at the same time, the posture recognizer will recognize the operator's head, wrist and hand postures. Step 13: The posture mapper maps the operator's pose to the robot's operating space, enabling remote operation of the robot. Step 14: After the operator completes the task demonstration, the data storage module stores the data of each joint of the robot and the point cloud collected by the head, and packages them into a compressed file for later use. Step 2, Environmental Perception and Coding: Step 2: After reading the collected data, use a point cloud grouper to group the point cloud information of each input frame to obtain a global point cloud family and a local point cloud family. Step 22: Use the global point cloud encoder and the local point cloud encoder in the point cloud encoder to process the two families of point clouds respectively, and obtain the feature information of the two families of point clouds. Steps 2 and 3: Using a state integrator, the robot state information and the number of denoised steps are concatenated with the point cloud feature information to obtain the global state features and local state features of each frame. Step 3: Two-layer feature fusion and diffusion denoising: Step 3: During the training phase, the two-layer feature fusion diffusion denoising model samples a real action sequence and adds multi-step Gaussian noise to it, degenerating it into an approximately random sequence. Step 32: Randomly select a diffusion step, and input the noisy action sequence corresponding to this step along with the two state features calculated by the two-layer feature fusion module into the diffusion denoising network. The training objective is to minimize the difference between the diffusion denoising network prediction and the actual action, so that the two-layer feature fusion diffusion denoising model can gradually learn how to generate reasonable actions under any noise level. Step 3: After sufficient training, the two-layer feature fusion diffusion denoising model can start from pure noise and gradually "de-noise" by combining environmental and state information during actual deployment, generating an executable high-quality action sequence to complete complex operation tasks.

Images