Dexterous hand control method, apparatus, device, and storage medium

Abstract

The application discloses a dexterous hand control method, device, equipment and storage medium, comprising: collecting motion data of a mechanical finger unit and real-time air pressure data of a pneumatic negative pressure circuit, and collecting three-dimensional point cloud data of a target object through a visual acquisition unit; constructing a visual representation vector corresponding to the three-dimensional point cloud data; splicing the motion data, the real-time air pressure data and the visual representation vector to obtain a joint observation state vector; based on the joint observation state vector and a preset deep reinforcement learning decision network, estimating a reward evaluation value corresponding to a multi-class grasping action mode of the dexterous hand, and determining the action with the highest reward evaluation value as a target action; outputting a control instruction corresponding to the target action, and driving the pneumatic negative pressure circuit and the mechanical finger unit to cooperatively execute the target action based on the control instruction. The application can solve the problem that the current dexterous hand control scheme has weak self-adaptive decision-making ability, resulting in insufficient grasping precision and poor reliability.

Description

Technical Field

[0001] This application relates to the field of end effector technology for industrial robots, specifically to a dexterous hand control method, device, equipment, and storage medium. Background Technology

[0002] With the continuous expansion of industrial automated sorting and collaborative robot operation scenarios, composite dexterous hands that combine pneumatic adsorption and mechanical gripping capabilities are being widely used. Existing dexterous hand grasping control solutions mostly adopt preset rule matching grasping strategies, relying on basic visual positioning to achieve object grasping, which can meet basic usage requirements in structured operation scenarios where object attributes are known.

[0003] However, existing technologies still have many limitations in complex operational scenarios involving unstructured objects and unknown object properties. First, visual perception is limited to a single dimension, only acquiring spatial location information of objects and lacking multi-granular feature representation of object spatial boundaries, hand-object proximity relationships, and geometric attributes of contact areas, making it difficult to support refined adaptive grasping decisions. Second, the coordination and adaptability between pneumatic suction cups and mechanical fingers are insufficient. In pure mechanical gripping mode, the protruding suction cup is prone to spatial interference with objects, compressing the effective working space of the envelope clamping. Furthermore, there is a lack of graded safety fallback mechanisms in scenarios of suction failure or clamping overload, which can easily lead to grasping failure or damage to devices or objects. Third, grasping modality decisions rely on manually preset rules and cannot dynamically adapt the optimal grasping method based on the real-time hand-object state. Some solutions that introduce intelligent algorithms suffer from insufficient fusion of multimodal perception information and inadequate state representation completeness, making it difficult to balance decision accuracy and scenario generalization ability.

[0004] It is evident that current dexterous hand control schemes suffer from weak adaptive decision-making capabilities, resulting in insufficient grasping accuracy and poor reliability.

[0005] The preceding description is intended to provide general background information and does not necessarily constitute prior art. Summary of the Invention

[0006] This application provides a dexterous hand control method, apparatus, device, and storage medium to solve the problem that current dexterous hand control schemes have weak adaptive decision-making capabilities, resulting in insufficient grasping accuracy and poor reliability.

[0007] In a first aspect, embodiments of this application provide a dexterous hand control method applied to a dexterous hand. The dexterous hand includes a palm base, a pneumatic negative pressure circuit, a vision acquisition unit, and a mechanical finger unit. The mechanical finger unit is hinged to the peripheral edge of the palm base and driven by a drive mechanism to perform a pinching envelope action. The pneumatic negative pressure circuit is used to generate and transmit negative pressure adsorption force. The vision acquisition unit is embedded in the palm base and is used to acquire three-dimensional point cloud data of a target object. The dexterous hand control method includes: The motion data of the mechanical finger unit and the real-time air pressure data of the pneumatic negative pressure circuit are collected, and the three-dimensional point cloud data of the target object are collected through the vision acquisition unit. Construct the visual representation vector corresponding to the 3D point cloud data; The motion data, real-time air pressure data, and visual representation vector are concatenated to obtain a joint observation state vector; Based on the joint observation state vector and the preset deep reinforcement learning decision network, the reward evaluation value corresponding to the dexterous hand performing multiple grasping action modes is estimated, and the action with the highest reward evaluation value is determined as the target action. Output the control command corresponding to the target action, and drive the pneumatic negative pressure circuit and mechanical finger unit to work together to execute the target action based on the control command.

[0008] Optionally, in some embodiments of this application, constructing the visual representation vector corresponding to the 3D point cloud data includes: A three-dimensional field of view is defined in a preset direction on the palm base, and the three-dimensional field of view is discretized into multiple voxel grids; The occupancy status of the multiple voxel grids is marked based on the three-dimensional point cloud data, and the marked voxel grids are spliced ​​together to obtain the spatial occupancy feature vector corresponding to each voxel grid. Based on the three-dimensional point cloud data and the control points corresponding to the dexterous hand, generate the surface contact feature vector and local geometric feature vector corresponding to the dexterous hand; The spatial occupancy feature vector, surface contact feature vector, and local geometric feature vector are sequentially concatenated and fused to obtain the visual representation vector.

[0009] Optionally, in some embodiments of this application, generating the surface contact feature vector and local geometric feature vector corresponding to the dexterous hand based on the three-dimensional point cloud data and the control points corresponding to the dexterous hand includes: Obtain multiple control points corresponding to the dexterous hand; Calculate the minimum spatial distance from each control point to the surface of the object corresponding to the three-dimensional point cloud data, and determine the nearest neighbor point on the surface corresponding to the minimum spatial distance; Extract the three-dimensional unit normal vector of the object surface at the nearest neighbor point of each surface; The minimum spatial distance corresponding to each control point is concatenated with the three-dimensional unit normal vector, and the concatenation result is adjusted based on a preset combination order to obtain the surface contact feature vector. Select the nearest neighbor point on the surface corresponding to the control point at the fingertip position as the reference point, and use each reference point as the spatial center to extract a local point cloud region within a preset range; High-order geometric feature mapping is performed on each local point cloud region to extract the local geometric feature descriptor corresponding to each reference point; By concatenating the local geometric feature descriptors of all reference points, the local geometric feature vector corresponding to the dexterous hand is obtained.

[0010] Optionally, in some embodiments of this application, the step of concatenating the motion data, real-time air pressure data, and visual representation vector to obtain a joint observation state vector includes: The motion data and real-time air pressure data are respectively converted into one-dimensional state vectors with the same format to obtain the motion vector corresponding to the motion data and the real-time air pressure vector corresponding to the real-time air pressure data. The motion vector, real-time air pressure vector, and visual representation vector are sequentially concatenated according to a preset modal arrangement order to obtain a joint observation state vector.

[0011] Optionally, in some embodiments of this application, the step of estimating the reward evaluation value corresponding to the multi-class grasping action modalities performed by the dexterous hand based on the joint observation state vector and a preset deep reinforcement learning decision network, and determining the action with the highest reward evaluation value as the target action, includes: The joint observation state vector is input into a preset deep reinforcement learning decision network, and the expected benefit evaluation value corresponding to each grasping action performed by the dexterous hand is output. The grabbing action modality with the highest benefit evaluation value is selected as the target action.

[0012] Optionally, in some embodiments of this application, it further includes: Construct decision-making and evaluation networks with identical structures; The joint observation state vector sample is input into the decision network, and the decision network selects the candidate action corresponding to the next observation state. The expected return evaluation value corresponding to the candidate action is calculated through the evaluation network to generate a temporal difference objective. The weight parameters of the behavior decision network are updated based on the temporal difference error until the decision network converges, thus obtaining a deep reinforcement learning decision network.

[0013] Optionally, in some embodiments of this application, the step of driving the pneumatic negative pressure circuit and the mechanical finger unit to coordinately execute the target action based on the control command includes: If the target action is the first mode, the vacuum generation circuit is activated to control the mechanical finger unit to maintain the open posture, and the flexible suction cup on the mechanical finger unit adheres to the surface of the object to establish negative pressure adsorption. If the target action is the second mode, then the vacuum generation circuit is shut down, and the mechanical finger unit is driven to close and retract. If the target action is the third mode, the vacuum generation circuit is activated to complete the adsorption and anchoring, and after adsorption and anchoring, the mechanical finger unit is driven to close and envelop the target object.

[0014] Secondly, embodiments of this application provide a dexterous hand control device applied to a dexterous hand. The dexterous hand includes a palm base, a pneumatic negative pressure circuit, a vision acquisition unit, and a mechanical finger unit. The mechanical finger unit is hinged to the peripheral edge of the palm base and driven by a drive mechanism to perform a pinching and enveloping action. The pneumatic negative pressure circuit is used to generate and transmit negative pressure adsorption force. The vision acquisition unit is embedded in the palm base and is used to acquire three-dimensional point cloud data of a target object. The dexterous hand control device includes: The acquisition module is used to acquire motion data of the mechanical finger unit and real-time air pressure data of the pneumatic negative pressure circuit, and to acquire three-dimensional point cloud data of the target object through the vision acquisition unit. The construction module is used to construct the visual representation vector corresponding to the 3D point cloud data; The splicing module is used to splice the motion data, real-time air pressure data and visual representation vector to obtain a joint observation state vector; The prediction module is used to predict the reward evaluation value corresponding to the multiple grasping action modes performed by the dexterous hand based on the joint observation state vector and the preset deep reinforcement learning decision network, and to determine the action with the highest reward evaluation value as the target action. The drive module is used to output control commands corresponding to the target action, and drive the pneumatic negative pressure circuit and the mechanical finger unit to work together to execute the target action based on the control commands.

[0015] Thirdly, embodiments of this application provide an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the dexterous hand control method as described above.

[0016] Fourthly, this application also provides a storage medium storing a computer program that can be loaded by a processor and executed as described above for the dexterous hand control method.

[0017] This application provides a dexterous hand control method, device, equipment, and storage medium, applied to a dexterous hand. The dexterous hand includes a palm base, a pneumatic negative pressure circuit, a vision acquisition unit, and a mechanical finger unit. The mechanical finger unit is hinged to the peripheral edge of the palm base and driven by a drive mechanism to perform a pinching and enveloping action. The pneumatic negative pressure circuit generates and transmits negative pressure adsorption force. The vision acquisition unit is embedded in the palm base and is used to acquire three-dimensional point cloud data of a target object. The dexterous hand control method includes: acquiring motion data of the mechanical finger unit and real-time air pressure data of the pneumatic negative pressure circuit; acquiring three-dimensional point cloud data of the target object through the vision acquisition unit; constructing a visual representation vector corresponding to the three-dimensional point cloud data; and concatenating the motion data, real-time air pressure data, and visual representation vector to obtain a combined... The dexterous hand control scheme provided in this application uses a joint observation state vector and a preset deep reinforcement learning decision network to estimate the benefit evaluation values ​​corresponding to various grasping action modes performed by the dexterous hand, and determines the action with the highest benefit evaluation value as the target action. It outputs control commands corresponding to the target action and drives the pneumatic negative pressure circuit and mechanical finger unit to collaboratively execute the target action, effectively improving grasping accuracy and operational reliability. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is an application environment diagram of the dexterous hand control method provided in the embodiments of this application; Figure 2 This is a flowchart illustrating the dexterous hand control method provided in an embodiment of this application; Figure 3 This is a schematic diagram of the structure of the dexterous hand provided in the embodiments of this application; Figure 4 This is a schematic diagram of the dexterous hand control system provided in the embodiments of this application; Figure 5This is another schematic flowchart of the dexterous hand control method provided in the embodiments of this application; Figure 6 This is another schematic flowchart of the dexterous hand control method provided in the embodiments of this application; Figure 7 This is another schematic flowchart of the dexterous hand control method provided in the embodiments of this application; Figure 8 This is a schematic diagram of the dexterous hand control device provided in the embodiments of this application; Figure 9 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0020] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of systems and methods consistent with those detailed in the appended claims or with some aspects of this application.

[0021] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover descriptions such as non-exclusive inclusion, so that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, components, features, and elements with the same names in different embodiments of this application may have the same meaning or different meanings, the specific meaning of which must be determined by its interpretation in that specific embodiment or further in conjunction with the context of that specific embodiment.

[0022] It should be understood that the specific embodiments described herein are merely illustrative of this application and are not intended to limit this application.

[0023] In the following description, the use of suffixes such as "module," "part," or "unit" to denote elements is solely for the purpose of illustrative purposes and has no specific meaning in itself. Therefore, "module," "part," or "unit" may be used interchangeably.

[0024] To address the aforementioned technical problems and overcome the shortcomings of existing technologies, embodiments of this application provide a dexterous hand control method, apparatus, device, and storage medium, which can efficiently construct and rationally inject prior information in long context scenarios, thereby improving prediction accuracy and generation quality.

[0025] Figure 1 This is a diagram illustrating the application environment of a dexterous hand control method in one embodiment. (Refer to...) Figure 1 This dexterous hand control method is applied to a dexterous hand control system. The dexterous hand control system includes a terminal 110 and a server 120. The terminal 110 and server 120 are connected via a network. The terminal 110 can be a desktop terminal or a mobile terminal, and the mobile terminal can be at least one of a mobile phone, tablet computer, or laptop computer. The server 120 can be implemented using a standalone server or a server cluster consisting of multiple servers. The server 120 is configured to execute the aforementioned dexterous hand control method, including: collecting motion data of the mechanical finger unit and real-time air pressure data of the pneumatic negative pressure circuit, and collecting three-dimensional point cloud data of the target object through the visual acquisition unit; constructing a visual representation vector corresponding to the three-dimensional point cloud data; concatenating the motion data, real-time air pressure data, and visual representation vector to obtain a joint observation state vector; based on the joint observation state vector and a preset deep reinforcement learning decision network, estimating the reward evaluation value corresponding to the dexterous hand performing multiple grasping action modes, and determining the action with the highest reward evaluation value as the target action; outputting a control command corresponding to the target action, and driving the pneumatic negative pressure circuit and mechanical finger unit to collaboratively execute the target action based on the control command.

[0026] Please see Figure 2 , Figure 2 This is a flowchart illustrating a dexterous hand control method according to an embodiment of this application. The dexterous hand control method provided in this embodiment may specifically include the following steps: S1. Collect motion data of the mechanical finger unit and real-time air pressure data of the pneumatic negative pressure circuit, and collect three-dimensional point cloud data of the target object through the vision acquisition unit.

[0027] The mechanical finger unit consists of multiple sets of biomimetic finger-like actuation structures hinged to the periphery of the palm base. Composed of a drive mechanism and multiple finger joints, it can controllably perform flexion, extension, retraction, and opening movements, serving as the core mechanical actuator for pinching and grasping actions. Motion data are quantitative parameters characterizing the current motion state and posture of the mechanical finger unit, mainly including the angular displacement, angular velocity, and drive output of each joint, reflecting the degree of opening and closing, movement trend, and force state of the mechanical finger. The pneumatic negative pressure circuit is a pneumatic system integrated inside the palm base and connected to an external air source. Composed of a vacuum generator, control valves, transmission pipelines, and a flexible suction cup, it generates and transmits negative pressure suction force to achieve suction-type grasping of objects. Real-time air pressure data is the quantified pressure value of the pneumatic negative pressure circuit at the current moment, mainly including pipeline pressure and suction cup internal cavity negative pressure value, which can intuitively reflect the working status and suction force of the suction circuit. The visual acquisition unit is a 3D imaging sensing component (such as a depth camera or structured light imaging module) embedded and fixed to the front of the palm base. Its sensing direction is towards the grasping field of view, used to acquire spatial morphological information of the target object. The target object is the object to be grasped by the dexterous hand, the target of the perception and grasping actions, and its shape, size, and material properties are diverse. The 3D point cloud data is the spatial morphological dataset output by the visual acquisition unit, composed of a large number of discrete sampling points with 3D spatial coordinate information, which can completely represent the target object's outline, size, and relative spatial position.

[0028] This application provides a dexterous hand, comprising a palm base, a pneumatic negative pressure circuit, a vision acquisition unit, and a mechanical finger unit. The mechanical finger unit is hinged to the peripheral edge of the palm base and driven by a drive mechanism to perform a pinching and enveloping action. The pneumatic negative pressure circuit generates and transmits negative pressure adsorption force. The vision acquisition unit is embedded in the palm base and is used to acquire three-dimensional point cloud data of a target object. Please refer to... Figure 3In some embodiments of this application, the mechanical finger unit 1 is the structure of the thumb, index finger, middle finger, and ring finger of the dexterous hand, hinged to the upper peripheral edge of the palm base, and equipped with a drive transmission mechanism inside, which can autonomously complete flexion, extension, and retraction movements, and can achieve enveloping and pinching mechanical gripping of objects; the hinged drive integrated component 2 is the component connecting the thumb and the palm base, which is the core part of the thumb module of the mechanical finger unit. It can drive the thumb to complete independent swinging and opening and closing movements, and also integrates the ventilation interface of the pneumatic negative pressure circuit, which can output negative pressure adsorption force to assist in handling thin and smooth objects. The gripping and fixing mechanism; the fixed mounting bracket 3 is used to lock and fix the entire dexterous hand to the end of the upper device such as the robotic arm. The air supply pipe and control cable can be inserted into the bracket to supply air source and control signal to the pneumatic negative pressure circuit and various drive structures. The vision acquisition unit 4 is embedded in the three-dimensional sensing component on the front of the palm base. The vision acquisition unit 4 can be a depth camera or a 3D imaging module. The sensing lens of the vision acquisition unit 4 faces the front area of ​​the gripping operation and can collect the three-dimensional point cloud data of the target object to be gripped, providing visual perception basis for subsequent gripping posture planning and action adaptive decision-making.

[0029] For example, specifically, using the main control module of the palm base as the system clock reference, and according to a preset sampling period, acquisition trigger commands are simultaneously sent to the sensing module of the mechanical finger unit, the pneumatic pressure sensing module of the pneumatic negative pressure circuit, and the vision acquisition unit embedded in the palm base. This unifies the acquisition time of the three types of data, eliminates timing deviations, and ensures the spatial consistency of subsequent multimodal feature fusion. The mechanical finger unit is hinged to the peripheral edge of the palm base and driven by a drive mechanism to perform a pinching envelope action. Through the sensing components built into each joint, kinematic parameters such as angular displacement, angular velocity, and drive output of each joint are collected. According to the preset joint numbering order, the kinematic parameters of all joints are summarized and organized sequentially to obtain complete mechanical finger unit motion data, which is used to characterize the current mechanical finger's pose and motion state. The pneumatic negative pressure circuit collects real-time pressure data of the circuit pipeline and suction cup cavity through pressure sensors installed in the negative pressure circuit pipeline and suction cup cavity. After basic filtering and normalization of the collected raw pressure data, real-time pressure data of the pneumatic negative pressure circuit is obtained, which is used to characterize the current working state and adsorption capacity of the adsorption circuit. The vision acquisition unit faces a preset field of view in front of the palm to perform depth imaging of the target object and obtain the three-dimensional coordinate information of each sampling point on the object surface. After basic noise reduction and coordinate system normalization, three-dimensional point cloud data of the target object in a unified spatial coordinate system is generated, which is used to characterize the spatial shape and relative position of the target object.

[0030] Finally, a unified system timestamp is added to the collected motion data, real-time air pressure data, and 3D point cloud data. After the time sequence alignment of the multi-source data is completed, the data is stored in the system cache for subsequent use in visual feature construction, multimodal state fusion, and grasping decision-making.

[0031] S2. Construct the visual representation vector corresponding to the three-dimensional point cloud data.

[0032] The visual representation vector is a one-dimensional structured numerical vector generated from the three-dimensional point cloud data of the target object after multi-granularity feature extraction and structured encoding. The visual representation vector quantifies the spatial shape, relative position, and local surface features of the target object from three levels: overall spatial occupancy, hand-object proximity relationship, and fine geometric attributes of the contact area.

[0033] For example, specifically, a three-dimensional field of view is defined in the preset perception direction of the palm base. This three-dimensional space is uniformly discretized into multiple regularly arranged voxel grids, and a unified spatial coordinate index is established. Then, based on the spatial coordinates of each point in the three-dimensional point cloud data, it is determined whether there is an object point cloud within each voxel grid, and the occupancy status of the voxel grid is marked. The occupancy status marking values ​​of all voxel grids are sequentially concatenated according to a preset spatial arrangement order to form a spatial occupancy feature vector, which is used to characterize the overall spatial boundary and macroscopic distribution pattern of the object. Next, the minimum spatial distance from each preset control point to the corresponding object surface in the three-dimensional point cloud is calculated, the nearest neighbor point on the surface corresponding to the minimum distance is determined, and the three-dimensional unit normal vector of the object surface at each nearest neighbor point is extracted. Then, the minimum spatial distance and the three-dimensional unit normal vector corresponding to each control point are concatenated one by one. Finally, the concatenation results of all control points are combined and arranged in a preset order to generate a surface contact feature vector, which is used to characterize the proximity contact relationship and contact orientation between the hand and the object.

[0034] Furthermore, the nearest neighbor point on the surface corresponding to the fingertip position control point is selected as the reference point. With each reference point as the spatial center, a local point cloud region within a preset range is extracted. High-order geometric feature mapping is performed on each local point cloud region to obtain the local geometric feature descriptor corresponding to each reference point. The local geometric feature descriptors of all reference points are concatenated in a preset order to generate a local geometric feature vector, which is used to characterize the fine geometric properties of the contact area.

[0035] Finally, according to the preset feature combination order, the spatial occupancy feature vector, surface contact feature vector and local geometric feature vector are sequentially cascaded and fused to form a unified one-dimensional vector, which is the visual representation vector corresponding to the three-dimensional point cloud data.

[0036] Optionally, in some embodiments of this application, the step "constructing the visual representation vector corresponding to the 3D point cloud data" may specifically include: A three-dimensional field of view is defined in a preset direction on the palm base, and the three-dimensional field of view is discretized into multiple voxel grids; The occupancy status of the multiple voxel grids is marked based on the three-dimensional point cloud data, and the marked voxel grids are stitched together to obtain the spatial occupancy feature vector corresponding to each voxel grid. Based on the three-dimensional point cloud data and the control points corresponding to the dexterous hand, generate the surface contact feature vector and local geometric feature vector corresponding to the dexterous hand; The spatial occupancy feature vector, surface contact feature vector, and local geometric feature vector are sequentially concatenated and fused to obtain the visual representation vector.

[0037] In some embodiments of this application, the preset direction can be the front of the work area where the palm base faces the target to be grasped, i.e., the main perception direction of the visual acquisition unit. This serves as a reference to define the perception space, ensuring that the field of view covers the area of ​​the grasping operation. The three-dimensional field of view is a three-dimensional spatial range centered on the preset direction and defined by the effective perception range of the visual acquisition unit. It is the core space for target object perception and grasping feature extraction, and all point cloud processing operations are carried out within this area. For example, the three-dimensional field of view refers to a cuboid or frustum-shaped space enclosed by the horizontal field of view (HFOV) and vertical field of view (VFOV) of the visual acquisition unit at a preset working distance (e.g., 20cm to 30cm), centered on the optical axis of the visual acquisition unit. Optionally, in some embodiments of this application, for ease of calculation, this area is simplified to a cube space with a side length of 10cm.

[0038] A voxel mesh is the smallest cubic spatial unit obtained by uniformly dividing a 3D field of view into fixed-size units. It is a discretized unit of 3D space, equivalent to a pixel in a 2D image, used to transform continuous 3D space into labelable meshable units. Occupancy status is a state attribute used to identify whether a target object point cloud exists within a single voxel mesh. It is usually presented in binary form, distinguishing between occupied and vacant states, and is a fundamental marker representing the spatial distribution of objects. The spatial occupancy feature vector is a one-dimensional numerical vector formed by concatenating the occupancy status values ​​of all voxel meshes in a predefined spatial arrangement. It macroscopically represents the overall contour, spatial boundary, and distribution position of the target object within the 3D field of view. Control points are predefined key anchor points on the dexterous hand structure, such as fingertips and joint nodes. They are reference points for calculating the relative position and contact matching relationship between the hand and the object. The surface contact feature vector is a feature vector generated by concatenating the minimum spatial distance from the control points to the object surface and the corresponding 3D normal vector of the object surface point. It is used to represent the proximity and contact orientation between key positions of the dexterous hand and the object surface, reflecting the contact matching relationship between the hand and the object. Local geometric feature vectors are feature vectors formed by extracting local surface geometric attributes from the local point cloud of the contact area and splicing them together. They represent the local geometric features of the contact area, such as the surface shape and concavity / convexity attributes, at a fine level.

[0039] For example, specifically, taking the preset perception direction of the palm base as the center and combining the effective perception range of the visual acquisition unit, the three-dimensional field of view area for target perception is determined. Then, according to a preset uniform size, this continuous three-dimensional space is uniformly divided into a large number of regularly arranged cubic pixel grids, completing the discretization processing of the three-dimensional perception space. Next, based on the three-dimensional point cloud data of the target object, it is determined whether there is an object point cloud inside each voxel grid, thereby marking the occupancy state of each voxel. Then, the occupancy state values ​​of all voxel grids are sequentially spliced ​​according to the preset arrangement order of spatial coordinates to form a spatial occupancy feature vector composed of all pixel states, used to characterize the macroscopic spatial shape of the object. Further, using the predefined control points on the dexterous hand as a reference, the minimum spatial distance from each control point to the object surface and the three-dimensional normal vector of the corresponding surface point are calculated in combination with the three-dimensional point cloud data. After orderly splicing, a surface contact feature vector is generated. At the same time, taking the object surface point corresponding to the control point in the fingertip area as the center, a local point cloud within a preset range is extracted and fine geometric attributes are extracted to generate a local geometric feature vector. Finally, according to the preset feature arrangement order, the spatial occupancy feature vector, surface contact feature vector, and local geometric feature vector are cascaded and fused one after the other to integrate three types of features of different granularities: macroscopic spatial distribution, mesoscopic contact relationship, and microscopic geometric attributes, to obtain the visual representation vector.

[0040] Optionally, in some embodiments of this application, the step "generating a surface contact feature vector and a local geometric feature vector corresponding to the dexterous hand based on the three-dimensional point cloud data and the control points corresponding to the dexterous hand" may specifically include: Obtain multiple control points corresponding to the dexterous hand; Calculate the minimum spatial distance from each control point to the surface of the object corresponding to the three-dimensional point cloud data, and determine the nearest neighbor point on the surface corresponding to the minimum spatial distance; Extract the 3D unit normal vector of the object surface at the nearest neighbor point of each surface; The minimum spatial distance corresponding to each control point is concatenated with the three-dimensional unit normal vector, and the concatenation result is adjusted based on a preset combination order to obtain the surface contact feature vector. Select the nearest neighbor point on the surface corresponding to the control point at the fingertip position as the reference point, and use each reference point as the spatial center to extract a local point cloud region within a preset range; High-order geometric feature mapping is performed on each local point cloud region to extract the local geometric feature descriptor corresponding to each reference point; By concatenating the local geometric feature descriptors of all reference points, the local geometric feature vector corresponding to the dexterous hand is obtained.

[0041] The minimum spatial distance is the minimum Euclidean distance from a single control point to all surface points in the object's 3D point cloud, quantifying the proximity of the dexterous hand's key position to the object's surface. The nearest neighbor is the object's surface point in the 3D point cloud that has the smallest distance to its corresponding control point. The 3D unit normal vector is a 3D vector at the nearest neighbor, perpendicular to the tangent plane of the object's local surface, with its magnitude normalized to 1, used to characterize the spatial orientation of the object's surface at that location. The local point cloud region is a small collection of object point clouds centered on the nearest neighbor of the fingertip, within a preset spatial range, containing only point cloud data near the contact area. The local geometric feature descriptor is a fixed-dimensional feature array obtained by quantizing and encoding the surface geometric attributes of the local point cloud region, typically containing high-order geometric information such as surface curvature, concavity / convexity attributes, and normal vector distribution characteristics.

[0042] For example, specifically, the system reads a predefined set of control points for key positions of the dexterous hand. These control points are typically placed at key grasping positions such as fingertips and joint nodes. The system obtains the spatial coordinates of all control points in the dexterous hand's body coordinate system. Then, it iterates through each control point, calculates the spatial distance between that point and all object surface points in the 3D point cloud data, and selects the result with the smallest distance value, which is the minimum spatial distance corresponding to that control point. At the same time, the object surface point corresponding to the minimum spatial distance is determined as the surface nearest neighbor of the control point.

[0043] Furthermore, for each nearest neighbor point on the surface, a local tangent plane is fitted to the point cloud within its preset neighborhood, and the normal vector of the tangent plane is calculated. The normal vector is normalized to obtain a three-dimensional unit normal vector. Then, the minimum spatial distance value corresponding to a single control point is concatenated with the three components of the corresponding three-dimensional unit normal vector to form the contact feature sub-vector corresponding to that control point. According to the preset order of control points, the contact feature sub-vectors of all control points are combined and adjusted in sequence to form a complete surface contact feature vector.

[0044] Next, control points located at the fingertip position are selected from all control points, and their corresponding nearest neighbors on the surface are used as reference points for local geometric analysis. With each reference point as the spatial center, all point clouds falling within the preset spatial radius are extracted from the overall 3D point cloud to form the local point cloud region corresponding to the fingertip contact area.

[0045] For each independent local point cloud region, high-order geometric feature mapping is performed to extract fine geometric parameters such as surface curvature, normal vector dispersion, and surface concavity / convexity properties. These parameters are then encoded and regularized according to a fixed dimension to obtain the local geometric feature descriptor corresponding to that reference point. Finally, according to the preset numbering order of the fingertip control points, the local geometric feature descriptors corresponding to all reference points are concatenated end-to-end to integrate them into a unified one-dimensional feature vector, thus obtaining the local geometric feature vector.

[0046] S3. The motion data, real-time air pressure data, and visual representation vector are concatenated to obtain a joint observation state vector.

[0047] Among them, the joint observation state vector is a unified one-dimensional feature vector that integrates the mechanical motion state of the dexterous hand, the working state of the pneumatic circuit, and the visual perception state of the target object. It is obtained by splicing three types of heterogeneous state data, namely motion data, real-time air pressure data, and visual representation vector, in a fixed order. As the input of the deep reinforcement learning decision network, it fully represents the global operation state of the hand-object interaction at the current moment.

[0048] For example, the three types of data to be spliced ​​can be pre-standardized: the motion data of the mechanical finger unit is organized into a one-dimensional ordered array and normalized according to the preset range. Specifically, the real-time air pressure data of the pneumatic negative pressure circuit is converted into a one-dimensional air pressure state array and the numerical normalization is completed according to the negative pressure range; the visual representation vector is subjected to numerical range verification and normalization calibration.

[0049] Next, the data consists of mechanical finger unit motion data, real-time air pressure data from the pneumatic negative pressure circuit, and visual representation vectors, in sequence. Simultaneously, the fixed dimensional length of each data type is locked to ensure that the total dimension after concatenation matches the input layer dimension of the deep reinforcement learning decision network. Then, following a pre-defined arrangement, the standardized motion data array, real-time air pressure data array, and visual representation vector are sequentially concatenated end-to-end to complete a one-dimensional cascade fusion, forming a continuous one-dimensional vector containing all three types of information: mechanical motion, airflow status, and visual perception.

[0050] Optionally, in some embodiments of this application, the step of "concatenating the motion data, real-time air pressure data, and visual representation vector to obtain a joint observation state vector" may specifically include: The motion data and real-time air pressure data are respectively converted into one-dimensional state vectors with the same format to obtain the motion vector corresponding to the motion data and the real-time air pressure vector corresponding to the real-time air pressure data. The motion vector, real-time air pressure vector, and visual representation vector are sequentially concatenated according to a preset modal arrangement order to obtain a joint observation state vector.

[0051] For example, specifically, a unified format processing is performed on two types of state data from different sources: the motion data of the mechanical finger unit is arranged according to the preset joint numbering order, and the kinematic parameters of each joint are arranged in sequence to form a one-dimensional ordered array, forming a motion vector; the real-time air pressure data of the pneumatic negative pressure circuit is arranged according to the preset air pressure monitoring point order, and the air pressure values ​​of each point are arranged in sequence to form a one-dimensional ordered array, forming a real-time air pressure vector. Then, the three types of one-dimensional features, namely the motion vector, the real-time air pressure vector, and the visual representation vector, are concatenated and spliced ​​in a fixed order, so that the state information of the three different modes is seamlessly integrated into a complete one-dimensional feature vector, namely the joint observation state vector.

[0052] S4. Based on the joint observation state vector and the preset deep reinforcement learning decision network, estimate the reward evaluation value corresponding to the multiple grasping action modes performed by the dexterous hand, and determine the action with the highest reward evaluation value as the target action.

[0053] The pre-set deep reinforcement learning decision network refers to a deep neural network model that has been pre-trained and deployed in the dexterous hand's main control unit. The network structure and weight parameters are preset fixed values. It takes the joint observation state vector as input and outputs the benefit evaluation value corresponding to each grasping action mode through forward inference. The grasping action mode is a set of pre-set executable grasping action patterns of the dexterous hand, such as pure mechanical finger pinching, pure pneumatic adsorption, and pneumatic coordinated clamping. Each action mode corresponds to a set of pneumatic negative pressure circuits and the coordinated control logic of the mechanical finger unit. The benefit evaluation value is a quantitative score output by the deep reinforcement learning decision network, used to characterize the expected grasping effect and overall benefit obtained by executing the corresponding grasping action mode under the current joint observation state. A higher value indicates a better grasping success rate and stability.

[0054] For example, specifically, the generated joint observation state vector is used as input and loaded into the input layer of a pre-defined deep reinforcement learning decision network to complete the global state data input for decision-making inference, providing a complete state basis for subsequent profit prediction. Then, the pre-defined deep reinforcement learning decision network performs forward inference operations on the input joint observation state vector, and combined with the decision strategy learned by the network, calculates and outputs the profit evaluation value corresponding to each of the pre-defined grasping action modalities of the dexterous hand, completing the quantitative prediction of the expected profit for each type of action. Finally, the profit evaluation values ​​corresponding to all grasping action modalities are compared, and the grasping action modal with the highest profit evaluation value is selected and determined as the target action in the current task state.

[0055] Optionally, in some embodiments of this application, the step "based on the joint observation state vector and a preset deep reinforcement learning decision network, predicting the reward evaluation value corresponding to the multi-class grasping action modalities performed by the dexterous hand, and determining the action with the highest reward evaluation value as the target action" may specifically include: The joint observation state vector is input into a preset deep reinforcement learning decision network, and the expected benefit evaluation value corresponding to each grasping action performed by the dexterous hand is output. Select the grabbing action modality with the highest benefit evaluation value as the target action.

[0056] For example, specifically, the joint observation state vector that has completed multimodal fusion is fed into a pre-defined deep reinforcement learning decision network as input data. Based on the internally pre-defined network weights and inference logic, the decision network performs forward inference operations on the current global task state of hand-object interaction, calculates the expected benefit evaluation value corresponding to each type of pre-defined grasping action performed by the dexterous hand, and then compares the numerical values ​​of the expected benefit evaluation values ​​corresponding to all grasping actions output by the decision network, and selects the grasping action modality corresponding to the largest benefit evaluation value, and determines it as the target action in the current task state.

[0057] Optionally, in some embodiments of this application, it may further include: Construct decision-making and evaluation networks with identical structures; The joint observation state vector sample is input into the decision network, and the decision network selects the candidate action corresponding to the next observation state. The expected return evaluation value corresponding to the candidate action is calculated through the evaluation network to generate a temporal difference objective. The weight parameters of the behavior decision network are updated based on the temporal difference error until the decision network converges, thus obtaining a deep reinforcement learning decision network.

[0058] The decision network, within the deep reinforcement learning framework, is a neural network responsible for action decision-making. It takes a joint observation state vector as input and outputs the selection result corresponding to the grasping action. During training, its weight parameters are continuously iteratively optimized, and upon convergence, it becomes the final deep reinforcement learning decision network used in deployment. The evaluation network is a value evaluation neural network with a structure completely identical to the decision network. Operating independently, its core function is to calculate the expected reward evaluation value corresponding to candidate actions, generating a target benchmark for training updates and avoiding training fluctuations caused by the decision network's bootstrapping updates. The temporal difference objective is the target value benchmark used to update network weights during reinforcement learning training. It consists of the immediate reward obtained from the current action, superimposed with the expected reward discount value of the next state output by the evaluation network, representing the cumulative expected total reward corresponding to the action sequence.

[0059] For example, specifically, two sets of neural networks with completely identical input and output dimensions are constructed, which are set as decision network and evaluation network respectively; the input dimension of the two networks matches the dimension of the joint observation state vector, and the output dimension corresponds to the benefit evaluation dimension of all preset grasping action modalities. Among them, the decision network is responsible for the selection of grasping actions, and the evaluation network is responsible for the estimation of action benefits and the generation of training objectives.

[0060] During training, the joint observation state vector samples are input into the decision network. The decision network performs forward inference on the input global job state based on the current weight parameters, selecting candidate actions for the next observation state from all available grasping action modalities, thus completing the action pre-selection process. Next, the candidate actions and their corresponding next observation states are input into the evaluation network, which calculates the expected reward value for each candidate action. Combining the immediate reward of the current action step with a preset discount coefficient, the expected reward value is weighted to generate a temporal difference objective for updating the network weights. Finally, the difference between the current output reward estimate of the decision network and the temporal difference objective is calculated, known as the temporal difference error. Specifically, minimizing this error is the optimization objective, and the weight parameters of the decision network are updated using the backpropagation algorithm. This process of state input, action selection, objective generation, and weight update is repeated iteratively until the output error of the decision network stabilizes within a preset range and the network converges, ultimately yielding a deep reinforcement learning decision network suitable for practical grasping decisions.

[0061] S5. Output the control command corresponding to the target action, and drive the pneumatic negative pressure circuit and mechanical finger unit to work together to execute the target action based on the control command.

[0062] Among them, the control command is a set of structured control signals generated by the main control unit based on the action mode of the target action, including the joint drive control parameters of the mechanical finger unit and the valve group switching and pressure regulation parameters of the pneumatic negative pressure circuit.

[0063] For example, specifically, the main control unit corresponding to the dexterous hand matches the preset control parameter template of the predetermined target action mode, and decomposes the action decision result into specific control quantities corresponding to two types of execution units: for the mechanical finger unit, it generates drive control parameters such as target pose and movement speed of each joint; for the pneumatic negative pressure circuit, it generates air circuit control parameters such as valve group on / off state and target negative pressure value; and encapsulates the above parameters according to the preset instruction format to generate the control instruction corresponding to the target action.

[0064] Next, the main control unit sends the generated control commands to the drive mechanism of the mechanical finger unit and the control valve group of the pneumatic negative pressure circuit. According to the motion parameters of the control commands, the drive mechanism of the mechanical finger unit drives each finger joint hinged to the palm base to complete the flexion, extension and retraction actions according to the preset trajectory, performing mechanical gripping or enveloping clamping. According to the air pressure parameters of the control commands, the pneumatic negative pressure circuit adjusts the negative pressure of the circuit, outputs the corresponding adsorption force, and performs negative pressure adsorption action.

[0065] Throughout the entire execution process, the control commands adjust the execution rhythm of the two types of units according to the preset coordinated timing of the target action, so that the mechanical clamping action and the negative pressure adsorption action cooperate with each other, such as starting synchronously or executing in sequence, and finally jointly complete the grasping operation corresponding to the target action.

[0066] Optionally, in some embodiments of this application, the step "driving the pneumatic negative pressure circuit and the mechanical finger unit to collaboratively execute the target action based on the control command" may specifically include: If the target action is the first mode, the vacuum generation circuit is activated to control the mechanical finger unit to maintain the open posture, and the flexible suction cup on the mechanical finger unit adheres to the surface of the object to establish negative pressure adsorption. If the target action is the second mode, then the vacuum generation circuit is shut down, and the mechanical finger unit is driven to close and retract. If the target action is the third mode, the vacuum generation circuit is activated to complete the adsorption and anchoring, and after adsorption and anchoring, the mechanical finger unit is driven to close and envelop the target object.

[0067] The first mode is a pure negative pressure adsorption gripping action mode, suitable for smooth, thin, fragile targets that are not suitable for mechanical clamping. It relies solely on negative pressure suction to complete the gripping, with the mechanical fingers remaining open and not participating in the clamping. The second mode is a pure mechanical clamping gripping action mode, suitable for irregularly shaped, rough-surfaced targets that are not suitable for negative pressure adsorption. It relies solely on the closing and pinching of the mechanical fingers to complete the gripping, without activating the negative pressure adsorption function. The third mode is a combined adsorption anchoring and mechanical envelope gripping action mode, combining the advantages of both negative pressure adsorption and mechanical clamping. It first adsorbs and fixes the object before performing mechanical clamping, suitable for targets that are easy to slip and have a certain weight, improving gripping stability and reliability. The vacuum generation circuit is the core functional branch of the pneumatic negative pressure circuit, consisting of a vacuum generating device, control valves, and transmission pipelines. When connected, it generates a negative pressure vacuum environment, providing the negative pressure suction required for the flexible suction cup to adsorb. Flexible suction cups are flexible adsorption actuators installed at the gripping end of mechanical finger units. They are mostly made of flexible sealing materials, which can adaptively conform to the surface of an object. Under negative pressure, they form a sealed cavity with the object's surface, generating adsorption and gripping force. Adsorption anchoring refers to first using negative pressure suction to firmly fix the target object and the dexterous hand in a stable relative position, preventing the object from shifting or deviating. This provides a stable positional reference for subsequent mechanical gripping actions and avoids the object slipping or misaligning during gripping.

[0068] For example, specifically, the main control unit reads the modal identifier of the target action, determines the action modal category to which it belongs, matches the execution control logic of the corresponding first modal, second modal or third modal, and calls the control parameters of the corresponding hardware unit.

[0069] When the target action is determined to be in the first mode, the main control unit outputs a control command to activate the vacuum generation circuit. At the same time, it controls the mechanical finger unit to maintain the open posture and not perform the gripping action. The dexterous hand is driven to move closer to the target object, so that the flexible suction cup on the mechanical finger unit adheres to the surface of the object. The vacuum generation circuit continuously generates negative pressure, and a stable negative pressure adsorption force is established in the sealed cavity between the suction cup and the surface of the object. The target object is grasped by relying on the adsorption force.

[0070] When the target action is determined to be in the second mode, the main control unit outputs a control command to shut down the vacuum generation circuit and the negative pressure adsorption function is not activated; a clamping control command is sent to the drive mechanism of the mechanical finger unit to drive the mechanical finger unit hinged to the periphery of the palm base to perform a closing and retracting action, and completes the mechanical clamping and grasping of the target object through the pinching and enveloping force of the finger joints.

[0071] When the target action is determined to be in the third mode, the main control unit first outputs a command to activate the vacuum generation circuit, and uses a flexible suction cup to adsorb the target object, completing the adsorption and anchoring of the object, and fixing the relative position of the object and the dexterous hand. After the adsorption and anchoring state is stable, the main control unit sends a clamping command to the mechanical finger unit, driving the mechanical finger unit to close and retract, forming a mechanical envelope clamping of the target object, and completing the grasping through the synergistic effect of negative pressure adsorption and mechanical clamping.

[0072] To further understand the dexterous hand control method provided in this application, the following explanation uses a dexterous hand control system as an example. Please refer to [link / reference]. Figure 4 The dexterous hand control system provided in this application includes a visual feature representation module, a double deep Q-network decision module (DDQN), and a ROS 2 (Robot Operating System 2) control node.

[0073] The peripheral connections and underlying physical control flow of the main control computing unit are as follows: Sensing signal input link: The data output end of the palm infrared depth camera (RGB-D camera) is connected to the "visual feature characterization module" via a USB 3.0 data cable.

[0074] Hand motor control link: The control signal output terminal of the ROS 2 control node is connected to the hand motor drive board; the drive board is connected to the 8 joint motors of the four-finger dexterity hand via CAN (Controller Area Network) bus.

[0075] Pneumatic circuit control link: The GPIO (General-purpose input / output) interface of the main control computing unit is connected to the relay group; the relay group controls the solenoid valve and the vacuum generator respectively. The air source is connected to the air inlet of the vacuum generator, the negative pressure output port is connected to the solenoid valve, and finally connected to the "threaded adaptive flexible suction cup" through a flexible air tube.

[0076] Negative pressure feedback link: A miniature air pressure sensor is connected in series on the trachea, and its analog signal output is connected to the main control's data acquisition interface.

[0077] To achieve seamless integration between the underlying physical hardware and the upper-level decision-making algorithm, and to overcome the dynamic bottleneck of traditional analytical methods in high-dimensional interactions, the data interaction scheme in this application is built upon the Discrete-Time Markov Decision Process (MDP) framework. The system's data flow and interaction are rigorously defined as a tuple. Where S is a finite set of states, A is a finite set of actions, P is the state transition probability matrix, R is the reward function, and γ∈[0,1] is the discount factor. The core data flow definition is as follows: (1) Definition of multimodal state observation space (Observation / State Space S) At any controlled sampling time t, the multimodal sensing data collected by the system through each physical link is strictly defined as a joint observation state vector. ∈S, its data concatenation formula is: Visual feature flow Φ(P t ): Local 3D point cloud matrix P input via USB 3.0 bus into the camera's field of view t ={p1,p2,...,p N}∈ NX3 (where N is the number of point clouds), p i The three-dimensional spatial coordinate vector (i=1,2,...,N) of the i-th sampling point in the point cloud is the point cloud P that constitutes time t. t The smallest unit, p i ∈ 3 , which are the high-order geometric feature vectors extracted by the mapping function Φ(·) of the visual feature representation module.

[0078] The real-time kinematic state of the robot reported via the CAN bus includes 8-dimensional joint angular displacement vectors q. t ∈ 8 The output torque estimation vector τ calculated from the current t ∈ 8 Pressure feedback flow P vac,t A miniature barometric pressure sensor with a real-time data acquisition interface, providing one-dimensional continuous scalar data. ∈ (Unit: kPa).

[0079] (2) Definition of Discrete-Continuous Hybrid Action Instruction Space (Action Space A) Considering the heterogeneous hardware characteristics of dexterous hands, which include both "pneumatic switches" and "continuous mechanical finger movements," the execution-end data distribution mechanism in this application embodiment does not employ traditional pure continuous trajectory fitting. Instead, it innovatively uses the output command vector a t ∈A can be decomposed into a "discrete-continuous hybrid space" of macroscopic mode selection and microscopic continuous control: a t ={m t , } Discrete mode identifier m t : Three-valued discrete logic issued by the GPIO port, where m t ∈{0,1,2} strictly maps to three system-level action branches: trigger relay pure adsorption, close gas path pure mechanical pinch, and pinch-inhalation coordinated action primitive.

[0080] Continuous angular displacement command The 8-dimensional target joint relative angular displacement increment vector sent from the ROS2 node to the CAN bus, where, ∈ 8 It is used to guide the underlying PID servoing of the motor drive board.

[0081] Through the definition of the above mathematical structure, this system successfully transforms complex electrical wiring interactions into standard high-dimensional matrices and tensor flows, laying the underlying data model foundation for subsequent multimodal feature extraction based on neural networks and DDQN collaborative decision-making.

[0082] To overcome the structural bulk and system power consumption problems caused by the use of additional motors, push rods or gear racks to actively drive the extension and retraction of the suction cup in the prior art, the embodiments of this application construct a purely physical passive adaptive yielding mechanism based on the principles of material mechanics and structural compliance.

[0083] This suction cup is integrally 3D printed from a flexible polymer material (such as thermoplastic polyurethane elastomer, TPU) with excellent elastic recovery capabilities. The outer wall of the suction cup is designed as a thin-walled corrugated tubular structure with continuous alternating concave and convex shapes. When subjected to axial compression from the target object, its mechanical behavior can be equivalently modeled as a linear / quasi-linear spring constitutive relationship, and the corresponding constitutive equation is as follows: in: When a mechanical finger closes its envelope around a target object, the object is subjected to opposing pressure, resulting in an axial external force (unit: N) on the front surface of the palm suction cup.

[0084] The equivalent structural stiffness (unit: N / mm) of this TPU threaded tubular structure in the axial direction can be controlled at an extremely low level (e.g., 0.5N / mm~1.2N / mm) by adjusting the corrugated wall thickness, thread pitch and outer diameter, thereby ensuring that no fatal local contact stress is generated on fragile objects during the yielding process.

[0085] The axial compression relief stroke of the suction cup (unit: mm), its maximum permissible physical compression stroke. The design is 25mm.

[0086] When the system is in its natural initial state without contact with any objects, 0. Under the elastic restoring force of the material itself, the suction cup is fully elongated, and its funnel-shaped front end face protrudes forward by 15mm~20mm compared to the front end face of the palm base when the mechanical finger is fully open.

[0087] When performing "Action A (Pure Adsorption Mode)," the protruding suction cups preferentially contact and tightly adhere to the flat surface of the target object. At this time, the four fingers remain spread out to avoid obstruction, and the retraction stroke continues. 0.

[0088] When performing "Action B (pure pinching mode)" or "Action C (pinch-suction combined mode)," the four mechanical fingers close to drive the target object towards the palm. When the object presses against the suction cup, >0, the threaded suction cup tube body adaptively undergoes axial folding and shrinkage ( → The suction cup actively recesses and yields. This mechanism eliminates the forward resistance generated by the suction cup, completely freeing up the envelope space of the mechanical finger, and purely physically and passively eliminating hardware space interference.

[0089] When the suction cup passively retracts to its limit or enters a cooperative gripping state, the system enters a strongly coupled physical contact phase of "pinch-suction-friction". It should be noted that if the target's action is in the third mode, the vacuum generation circuit is activated, causing the suction cup to contact the target object's surface and generate a pre-adhesion force. During the process of the mechanical finger unit closing and enveloping the target object, the suction cup is passively compressed and retracted by the target object. After the mechanical finger unit completes the envelopment, the vacuum generation circuit is controlled to reach its maximum negative pressure, completing the final adsorption and anchoring. Based on the contact mechanics principle of multimodal hybrid grippers, the system possesses a limit tangential disturbance resistance (maximum static friction force) to prevent tangential slippage when gripping unstructured, irregularly shaped, or heavily loaded target objects. The following constitutive equation for limiting friction is followed: in: It is the equivalent coefficient of friction between the end of the TPU suction cup / flexible fingertip and the surface of the target object.

[0090] The algebraic sum of the projections of the normal gripping force generated by the closure of each mechanical finger and the passive supporting force of the palm rigid surface on the contact surface normal vector (unit: N).

[0091] Active adsorption force generated by vacuum circuit activation (unit: N).

[0092] Among them, active adsorption force The magnitude of is determined by both the gas pressure state of the gas path and the physical geometric boundary, and its calculation formula is: in: The actual vacuum pressure difference between the internal cavity of the suction cup and the external environment (unit: ).

[0093] External standard atmospheric pressure (a continuous scalar constant, taken as...) ).

[0094] For the miniature barometric pressure sensor connected in series on the negative pressure link at any time Real-time acquisition of the relative pressure value inside the suction cup (unit: kPa).

[0095] The effective cross-sectional area (unit: m²) for achieving a perfect airtight edge seal between the end face of the threaded suction cup funnel and the surface of the object. 2 The calculation formula is as follows: D e De is the effective sealing diameter of the flexible sealing lip at the front end of the suction cup. 2 It is the square of the effective sealing diameter, used to calculate the effective sealing area A. e .

[0096] As can be seen from the above mechanical model, since the embodiment of this application adopts a passive folding and yielding structure, the system does not need to consume additional electrical energy and algorithm computing power to control a dedicated active push rod motor, thus eliminating physical interference in the finger bone envelope space; at the same time, at the moment of contact, the vacuum generator is instantly activated to... A sharp drop, capable of achieving this without increasing the output torque of the mechanical knuckle motor (i.e., without increasing...). Under the premise of avoiding crushing fragile objects, the maximum static friction force at the end is greatly enhanced by formula cascading. This enables multimodal collaborative operation with "low gripping force, high load capacity, and zero hardware interference" at the underlying physical mechanism level.

[0097] For further details, please refer to Figure 5 This application describes the data flow and computational architecture of the same multi-dimensional local visual interaction feature extraction algorithm. This architecture resides within the "Visual Feature Representation Module" of the main control computing unit. At the top is the "Palm Local 3D Point Cloud Input" data port; downwards, three independent and parallel cascaded computing branches extend: the "Spatial Occupation Feature Calculation Module," the "Surface Contact Feature Calculation Module," and the "Local Geometric Feature Calculation Module." The outputs of these three modules converge downwards, connecting to the "Feature Serialization and Fusion Module," ultimately outputting a visual representation vector for action decision-making. .

[0098] In each control sampling period Inside, the "visual feature representation module" receives the raw local 3D point set P acquired by the palm-mounted infrared depth camera via a USB 3.0 data cable. t ={p1,p2,...,p N}∈ NX3 ,in This represents the total number of discrete point clouds. The specific technical implementation schemes and mathematical modeling processes for the three levels are as follows: First layer: Occupancy Feature Calculation Model To provide dexterous hands with global collision avoidance warnings and coarse 3D boundary localization of target objects, the algorithm first establishes a 3D spatial cube virtual field of view directly in front of the palm. Its physical dimensions are set as .

[0099] This area Uniformly discretized along the three-dimensional axis into Voxel Grid, a set of voxel grids is represented as , where coordinate index .

[0100] Traverse the original point cloud input Each individual voxel grid The space occupancy status is defined in binary form using an indicator function: The occupancy states of all voxel meshes are unfolded in one dimension according to a fixed spatial index order to generate a... Binarized eigenvectors of dimension : T Second layer: Surface Feature Calculation Model To accurately reflect the microscopic spatial proximity and contact orientation between the dexterous hand and the target object's surface, the algorithm introduces contact degradation inference based on key control points of the hand. First, N key control points of the four-finger dexterous hand structure (including...) are defined. The set of coordinates of the mechanical fingertip center point, the geometric center points of each finger bone segment, and the geometric center point of the palm at the current moment is represented as follows: .

[0101] For any key control point in the set The algorithm calculates the point cloud distance to the target object using the Euclidean distance operator. Minimum straight-line contact distance : At the same time, find the nearest neighbor point on the object surface corresponding to this minimum distance. : Extracting the nearest neighbor point based on local surface least squares fitting or point cloud normal vector estimation methods. The three-dimensional unit normal vector of the object surface at that location .

[0102] The minimum straight-line contact distance calculated for each key control point With normal vector The data is then pieced together to form the contact representation of a single control point. (Full-hand) A key point at a moment Surface contact feature vector The expression is: The third layer: Local-Geo Feature Computation Model Because the eye-in-palm perspective (i.e., the visual acquisition unit is fixedly installed on the front of the palm base so that its perception direction is consistent with the grasping direction of the dexterous hand, so that the target object is observed from the perspective of the dexterous hand itself) can only acquire shallow point clouds of the object in close-range operations, and cannot extract complete global geometric features, the algorithm uses point cloud patches of the contact area to characterize the micro curvature and texture features of the object surface.

[0103] by The key point of the mechanical fingertip is the nearest neighbor point found. Let the center of the space sphere be the point where the physical radius of the space sphere is 1. A local region, where a subset of the point cloud (i.e., a point cloud patch) is represented as : A lightweight point cloud feature encoder based on the PointNet architecture is used to perform high-order feature mapping on point cloud patches. To ensure that feature extraction is permutation invariant to the disorder of the point cloud arrangement, the encoder employs a nonlinear shared mapping function. Combination with the channel-wise max-pooling symmetric operator. Extracted from a single point cloud patch. 3D local geometric feature descriptor The calculation formula is: in, Defined by a set of network weight parameters of a multilayer perceptron (MLP); The operator is responsible for extracting the most salient geometric features within the local region.

[0104] Will The local geometric feature descriptors corresponding to the key points of each fingertip are sequentially concatenated to generate a globally cascaded local geometric feature vector for the entire hand. : T In the feature concatenation and fusion module, the system concatenates the feature vectors of the three levels obtained independently from each branch according to a topological order from macro to micro. The final output is a complete visual interaction state representation vector that fully maps the current complex spatial perception relationship between hand and object. The cascade formula is: T Through this three-layer cascading mechanism, the system successfully integrated the original point set containing massive amounts of redundant spatial point information. The features are compressed and mapped into low-dimensional, high-information-density nonlinear feature descriptors, which can then be input into the subsequent dual-depth Q-network for high-frequency action decision-making. It should be noted that in this embodiment, local geometric features are only constructed for the four key points of the fingertip, and the calculation method for the local features of the other key points is the same.

[0105] For further details, please refer to Figure 6 This application provides a multimodal action decision and control method based on DDQN, which runs in the dual deep Q network (DDQN) decision module and ROS 2 control node of the main control computing unit.

[0106] The topmost element is the "current state feature vector". The input is a "data port"; cascaded downwards are the "feature embedding and normalization processing layer" and the "multilayer perceptron (MLP) Q-value parallel computing network"; from the output of the MLP computing network, three parallel evaluation lines are drawn downwards, outputting the expected reward values ​​corresponding to the three types of grasping actions, i.e., the expected reward value corresponding to the pure adsorption action. Pure pinching action corresponds to The corresponding action of pinching and sucking These three output terminals are all connected to the "maximum benefit action selection module" ( The output of this module is ultimately connected to the "low-level ROS 2 hardware control command issuance" port to achieve closed-loop control of the pneumatic circuit and motor drive board.

[0107] This application embodiment utilizes the hybrid action space of a dexterous hand. The decision lag problem of end-to-end fitting of continuous trajectories in unstructured scenarios is solved by decoupling and mapping the primitives into three macroscopic discrete action primitives with explicit physical meaning. Action A (pure adsorption mode) ): Set discrete mode flag bit The vacuum generator is controlled to open the pumping path, and a zero-angular displacement increment matrix is ​​simultaneously sent out through the ROS 2 node. This allows the four fingers to remain in a rigid, fully extended position. This primitive is suitable for target objects with large, smooth surfaces or sheet-like structures.

[0108] Action B (pure pinch grip mode) ): Set discrete mode flag bit The vacuum generator remains off, and the system's air path is in a state of atmospheric pressure unloading, allowing the flexible suction cup to retract based on physical elasticity. Simultaneously, by limiting the maximum output current, the eight joint motors of the mechanical fingers are controlled to perform continuous inward closing control, with the corresponding joint angular displacement increment commands satisfying... .

[0109] Action C (Pinch-Suction Coordination Mode) ): Set discrete mode flag bit The system executes phased coordinated control: In the first phase, the electromagnetic valve and vacuum generator are activated instantaneously, using the protruding flexible suction cup to adhere to the target and generate a planar adsorption force to complete the initial physical anchoring; in the second phase, while establishing negative pressure, continuous angular displacement commands are issued. This drives four-finger motors to quickly close the envelope and lock the target. This primitive is specifically designed for disturbance-resistant grasping of large, non-rigid, or heavily loaded targets.

[0110] It should be noted that the discrete pattern identifier m t∈{0,1,2}, which correspond to actions A, B, and C respectively, and the mapping relationship is m t =0 m t =1 and m t =2 .

[0111] At any control sampling time The joint observation state vector Input the decision network. The forward computation process flows in parallel according to the following mathematical model: Step 1: State Variable Normalization and Embedded Representation State vector Trigger joint operations of Batch Normalization (BN layer) and Layer Normalization (LN layer) to eliminate gradient distortion introduced by the different dimensions of multi-source sensor data (point cloud, angle, and air pressure continuous scalars): Step 2: Parallel computation of Q-values The normalized state embedding vector The input is fed into a multilayer perceptron (MLP) deep prediction network with three hidden layers (the number of neurons in each hidden layer is independently designed to be 1024, 1024, and 512, respectively). The network weight parameter set is represented as follows: The MLP network outputs in parallel the action value estimates (Q-values) for the three action primitives: in, This is a pure adsorption action primitive, corresponding to a gripping mode where only the pneumatic negative pressure circuit is activated and the mechanical fingers remain open. This is a pure pinching action primitive, corresponding to a gripping mode where the pneumatic circuit is closed and only the mechanical fingers are driven to close and hold the object. This is a primitive for the pinch-suction coordinated action, corresponding to a coordinated grasping mode that first uses negative pressure for adsorption and anchoring, and then uses mechanical fingers to close the envelope. i It is any action primitive in the action set, and i is the action category index.

[0112] Step 3: Optimal Action Primitive Selection Decision The maximum-profit action selection module adopts a deterministic greedy strategy. (Operator) online, calculates and outputs the optimal instruction vector that yields the maximum long-term cumulative return in the current state. : Solve for the optimal instruction vector Then, it is decoupled into logical identifier bits according to preset mapping rules. With angular displacement increment vector ,when hour, and ;when hour, And output the corresponding angular displacement vector; when hour, And execute the adsorption command.

[0113] ROS 2 control node acquisition Logical flag bits in and angular displacement vector It seamlessly converts to underlying GPIO level signals and CAN bus pulse matrix, enabling data interaction between software and hardware and data transmission from the underlying layer.

[0114] Mathematical Modeling of Multimodal Fusion Joint Reward Function During the offline training or online evolution phase of the network strategy, in order to enable the robot to automatically reconcile the inherent game relationship between "multi-finger pinching" and "palm adhesion" based on the object's geometry, the system constructs a joint reward function that integrates feedback from multiple physical quantities. The cascade calculation equation for the total reward value is strictly defined as follows: in, These are the preset empirical weight scalar coefficients. The underlying physical and mathematical model of each sub-reward function is defined as follows: (1) Approach Reward ) It guides the dexterous hand to efficiently approach the target object, avoiding aimless spatial searching. Its size is related to the calculated total hand size. The minimum straight-line contact distance d from each key control point to the object surface i Highly negatively correlated: (2) Effectively increase rewards (Lift Reward) ) The sparse dominant reward for the ultimate success of the task is used to determine whether a stable grasping configuration has been established. The real-time vertical height of the target object's center of mass in the robot arm's base coordinate system is defined as... The initial height is The distance is in centimeters (cm): (3) Pneumatic adsorption reward (Suction Reward) ) Explicit quantization excitation is used to determine whether a proper airtight seal is established between the suction cup and the object, and the continuous scalar value fed back by the miniature air pressure sensor is directly read. Pressure is measured in kilopascals (kPa). (4) Hardware security overload penalty item (Penalty Reward) ) Suppress rigid collisions and illegal joint overloads during execution, and read the real-time output torque estimation vector reported from the CAN bus. , This is the estimated output torque of the j-th joint motor at sampling time t (j=1,2,...,8). If the torque of any joint exceeds the preset safety limit threshold... This will trigger a high negative penalty: To completely eliminate the inherent academic bottleneck of Q-value overestimation caused by the maximization operator in traditional Deep Q-Networks (DQNs), this application strictly introduces a dual-network decoupled architecture. The system simultaneously maintains two sets of neural networks with identical structures but heterogeneous parameters: a current behavior decision network (with parameters...). ) and Target Network (parameters are) ).

[0115] At any training iteration step, the system randomly draws a physical mini-batch of samples (of size 1) from the experience replay buffer. According to the Bellman Equation of reinforcement learning, the temporal difference target (TD Target) is... The computational equation is rigorously defined as: in, This is a constant representing the future reward time-series discount factor. The underlying core logic of this formula lies in: utilizing the current behavior network. Decision in the next state The action that maximizes the evaluation benefit is the one that utilizes the target network. Evaluate the actual value of the action to achieve complete decoupling between selection and evaluation. For the next state The corresponding candidate actions are determined by maximizing the output of the target Q network in the action space A, thus eliminating ghost variables.

[0116] The policy network minimizes the mean squared time difference error (TD Error) loss function. To update the current network's weight parameters: The gradient is calculated using the backpropagation algorithm. And the Adam optimizer is used to optimize the behavioral network parameters. Perform high-frequency iterative updates. Target network parameters. The soft update mechanism is then used, with smooth tracking and synchronization performed by the behavior network at a fixed ratio α≤1: Based on the above-mentioned algorithm convergence model, the embodiments of this application enable the system to autonomously converge the optimal multimodal switching control law when facing various unknown, cross-scale object grasping tasks.

[0117] Please see Figure 7 This application provides a dual-closed-loop security protection architecture in the underlying control flow. The architecture is divided into a "pneumatic loop feedback branch" on the left and a "motor follow-up feedback branch" on the right. In the left pneumatic branch, the "Action A / C negative pressure start command" is connected to the "miniature air pressure sensor high-frequency sampling (100 Hz)"; real-time pressure data is transmitted to the main control unit at a sampling frequency of 100Hz, and the main control unit continuously samples and buffers the vacuum pressure value; when the pressure value exceeds the safe threshold range, the main control unit marks the corresponding sampling time as an abnormal state point. In the right motor branch, the "Action B / C finger close command" is cascaded to the "drive board PID position controller", and forms a dual follow-up closed loop through "encoder angle feedback" and "motor current monitoring"; if the overload lockout trigger condition is met, the position loop is cut off, and the command is directly sent to "lock the current position and switch to constant torque output mode".

[0118] In the pneumatic control loop, in order to prevent air leakage and adsorption failure caused by uneven surface, porous and breathable surface or initial position deviation of the target object, the embodiments of this application construct an active safety hard interruption interception mechanism based on timing window.

[0119] The initial time for the system to activate the vacuum generator and execute the adsorption command is set to Miniature barometric pressure sensor High sampling frequency for real-time reading of relative pressure values ​​within the suction cup cavity The main control computing unit operates within the discrete timing window. The system continuously assesses the airtightness establishment status. Its status interruption trigger indication function... The strict definition is as follows: in, The absolute negative pressure threshold constant is set (in this embodiment, it is set as...). .

[0120] When the pressure values ​​of all discrete sampling points within the evaluation window meet the threshold requirements (e.g., all are higher than the safety threshold), the global capture flag is activated. , where t k At the k-th sampling moment of the window, which contains 20 discrete sampling points, corresponding to an evaluation duration of 200ms at a system sampling frequency of 100Hz, matching the discrete sampling timing of the main control unit, it is determined that the cavity failed to form a vacuum anchor within the specified time limit. The main control computing unit immediately triggers a high-priority hardware interrupt to the underlying bus, instantly and forcibly canceling the current adsorption action flag, and seamlessly switching the mode to the action primitive in the underlying control flow. (Pure grip mode). By driving the four-finger motor to close quickly, the multi-finger rigid envelope is used to forcefully grab the target object from the bottom, thus completely eliminating the risk of slippage when relying solely on pneumatic mode.

[0121] During the gripping action performed by the mechanical fingers, in order to prevent the multi-finger rigid structure from generating excessive shear force or local stress concentration on fragile, soft or unknown hardness target objects, a position-current dual closed-loop overload follow-up protection model is introduced into the motor motion circuit.

[0122] For any articulated motor The proportional-integral-derivative (PID) controller on the underlying driver board tracks the error using joint angular displacement. Drive joint movement: in, The target joint angle displacement is determined by the upper-level planning. This provides real-time feedback of the actual joint angular displacement from the encoder. The standard bottom-level position loop control outputs torque commands. The expression is: in, Let s be the joint position tracking error, and s be the integral time dummy variable. These are the proportional, integral, and derivative control gains of the position loop, respectively.

[0123] During the pinch-close action, the current monitoring module synchronously acquires the real-time phase current of the motor at high frequency and converts it into an estimated value of the actual output torque of the joint. When the joint determination conditions are met. and If so, it is determined that the mechanical fingertip has made physical contact with the workpiece.

[0124] in, The increment of the measured output torque of the j-th joint in adjacent sampling steps is the feedback observation value of the stress torque sensor; This represents the change in position tracking error over adjacent steps. It is the contact determination threshold for the joint torque increment, which is a preset constant used to identify significant sudden changes in torque; This is the contact threshold for determining the change in joint position error, a preset constant used to identify significant deviations in position tracking. It is achieved through a dual feature mapping of torque feedback and position error. The rate of change of joint tracking error exhibits a strong nonlinear coupling with motor torque. This application defines the following constant torque switching safety judgment criteria in its embodiments: in, For low-speed joint creep threshold constants that tend to zero, The preset joint core safety overload torque threshold, The measured output torque of the j-th joint (with a vector representing the direction of motion) is covered by an absolute value operator to cover overload scenarios in both positive and negative directions, thus preventing the protection mechanism from failing during reverse mechanical overload.

[0125] Once the above conditions are met simultaneously, the system determines that the mechanical finger has achieved a tight and rigid fit with the surface of the target object, and that continued pulse transmission from the position loop would cause mechanical overload or damage to the object. The PID position controller then executes soft-touch unloading, switching the joint from position control mode to torque control mode, directly outputting a clamping torque command of fixed amplitude, no longer generating torque commands based on real-time position errors; the clamping direction is preset to the workpiece clamping direction to ensure a continuous and stable clamping force. The corrected constant torque formula is as follows: in, To achieve the preset constant clamping torque, For symbolic functions, The preset clamping direction increment (a fixed direction increment representing the joint clamping movement direction) is completely decoupled from the real-time position error. This dual closed-loop feedback protection mechanism enables the robot's dexterous hand to achieve excellent passive contact compliance and hardware burn-out protection boundaries while possessing extremely high follow-up accuracy.

[0126] This application proposes and deploys a standardized two-stage experimental verification closed-loop process from simulation to reality to evaluate the robustness of multimodal dexterous hand hardware and the DDQN decision algorithm: (1) Physics engine simulation training phase A full-scale, high-fidelity 3D digital twin model of a vision-attraction integrated robot dexterous hand is constructed using a physical simulation platform with highly parallel dynamics simulation capabilities (such as Isaac Gym). The model is then imported into the simulation environment. A physical test database of more than 100 heterogeneous 3D objects (covering everyday items, irregularly shaped industrial profiles, thin flat targets, and complex topological structural components). By deploying a domain randomization strategy, the surface friction coefficient, mass distribution, initial pose, point cloud noise, and air pressure drop of the objects are subjected to high-dynamic perturbation and confusion, and large-scale parallel reinforcement learning evolution is carried out until the Q-value function of the deep neural network and the action strategy are completely converged.

[0127] (2) Real hardware testing phase The assembled physical hardware system is deployed to the end effector interface of a six-DOF industrial robotic arm. The main control computing unit runs ROS 2 control nodes under the Ubuntu 22.04 operating system. A series of extremely small and fragile models, heavy-duty rigid metal components, and foreign objects with complex surface curvatures are prepared as real physical test samples. The initial placement and pose of the test targets are randomly changed within the experimental site, and the perception and decision network is run in a closed loop to frequently collect statistics on various physical operation indicators.

[0128] During the experimental operation phase, the system's backend log management module performs high-frequency real-time data collection and automatically calculates and outputs the following four core quantitative evaluation indicators to measure the system's overall crawling performance: (1) Grasping Success Rate ) This is used to evaluate the generalization grasping reliability of a dexterous hand multimodal configuration when facing unknown objects. The total number of grasping attempts is set to [number missing]. The indicator is defined by the following formula: Among them, M total The total number of grabbing trials, This is a conditional indicator function; the function value is 1 if the condition within the parentheses is true, and 0 otherwise. Single successful fetch. The physical boundary of the judgment is strictly constrained as follows: the dexterous hand control actuator successfully grasps the target object and the robotic arm vertically lifts the object off the table. And the hovering time in the air with arbitrarily tangential anti-disturbance capability. The internal objects showed no physical slippage or obvious configurational degradation.

[0129] (2) Multimodal action execution efficiency (Grasping Action Efficiency) ) Multimodal action execution efficiency is used to evaluate the simplification of the control sequence, and the calculation formula is: in, This represents the theoretical minimum number of steps required for a single transport task. This represents the total number of action primitives issued by the system during actual execution; the higher the value of this indicator, the lower the redundancy of the action sequence and the better the execution efficiency.

[0130] It is worth emphasizing that when the system executes "Action C (Pinch-Suction Co-operation Mode)," due to the complete spatial decoupling between its unique palm suction cup adsorption and the outer four-finger envelopment, the system can simultaneously adsorb a flat target and pinch and envelop another irregularly shaped target within the same action sequence. This metric, the task throughput ratio, is used to measure the task completion capability corresponding to a unit action sequence length. When a single compound action can simultaneously complete multiple targets, this metric value can be greater than 1 (i.e., 100%), reflecting the system's collaborative operation gain, which differs from the traditional execution efficiency metric whose theoretical upper limit is always 100%. (3) Simulation-to-Real Performance Gap ) The formula for calculating the generalization loss and robustness of deep reinforcement learning algorithms when transferring training strategies in digital twin environments to the real physical world is defined as follows: in, To improve the success rate of data capture in a simulated environment, This application aims to determine the success rate of data capture in real-world scenarios. The embodiments of this application utilize an absolute value operator to cover both high and low performance deviations, thus avoiding the vulnerability where the actual success rate is significantly higher than the logic used in simulation.

[0131] (4) Mean Orientation Error ) This is used to precisely quantify the fine-grained interactive accuracy of dexterous hand control over the pose of target objects during complex grasping or in-hand manipulation tasks. Let the total number of successfully grasped objects be... For the first For each successful sample, the final pose of the object is represented by the measured unit quaternion. The desired attitude unit quaternion of the target setting is represented as: The absolute angular deviation value of a single operation. (Unit: radians) Derived based on quaternion inner product operator: The overall average angle reversal error of the system The cascading calculation formula is: In summary, the dexterous hand control method provided in this application is applied to a dexterous hand, which includes a palm base, a pneumatic negative pressure circuit, a visual acquisition unit, and a mechanical finger unit. The mechanical finger unit is hinged to the peripheral edge of the palm base and driven by a drive mechanism to perform a pinching and enveloping action. The pneumatic negative pressure circuit is used to generate and transmit negative pressure adsorption force. The visual acquisition unit is embedded in the palm base and is used to acquire three-dimensional point cloud data of a target object. The dexterous hand control method includes: acquiring motion data of the mechanical finger unit and real-time air pressure data of the pneumatic negative pressure circuit, and acquiring three-dimensional point cloud data of the target object through the visual acquisition unit; constructing a visual representation vector corresponding to the three-dimensional point cloud data; and concatenating the motion data, real-time air pressure data, and visual representation vector to obtain a joint visual representation vector. The dexterous hand control scheme provided in this application estimates the benefit evaluation values ​​corresponding to multiple grasping action modes performed by the dexterous hand based on the joint observation state vector and a preset deep reinforcement learning decision network, and determines the action with the highest benefit evaluation value as the target action; outputs the control command corresponding to the target action, and drives the pneumatic negative pressure circuit and mechanical finger unit to coordinately execute the target action based on the control command, thereby effectively improving grasping accuracy and operational reliability.

[0132] To facilitate better implementation of the dexterous hand control method of this application, this application also provides a dexterous hand control device based on the above-described dexterous hand control method. The meanings of the terms used are the same as in the dexterous hand control method described above, and specific implementation details can be found in the descriptions within the method embodiments.

[0133] Please see Figure 8 , Figure 8 This is a schematic diagram of the dexterous hand control device provided in an embodiment of this application. The dexterous hand control device may specifically include a data acquisition module 201, a construction module 202, a splicing module 203, a prediction module 204, and a driving module 205, as detailed below: The acquisition module 201 is used to acquire the motion data of the mechanical finger unit and the real-time air pressure data of the pneumatic negative pressure circuit, and to acquire the three-dimensional point cloud data of the target object through the vision acquisition unit. Construction module 202 is used to construct the visual representation vector corresponding to the three-dimensional point cloud data; The splicing module 203 is used to splice the motion data, real-time air pressure data and visual representation vector to obtain a joint observation state vector; The prediction module 204 is used to predict the reward evaluation value corresponding to the multiple grasping action modes performed by the dexterous hand based on the joint observation state vector and the preset deep reinforcement learning decision network, and to determine the action with the highest reward evaluation value as the target action. The drive module 205 is used to output control commands corresponding to the target action, and drive the pneumatic negative pressure circuit and the mechanical finger unit to work together to execute the target action based on the control commands.

[0134] For specific limitations regarding the dexterous hand control device, please refer to the limitations of the dexterous hand control method above, which will not be repeated here. Each module in the aforementioned dexterous hand control device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in hardware or independently of the processor in a computer device, or stored in software in the memory of a computer device, so that the processor can call and execute the operations corresponding to each module.

[0135] The dexterous hand control device provided in this embodiment is applied to a dexterous hand, which includes a palm base, a pneumatic negative pressure circuit, a visual acquisition unit, and a mechanical finger unit. The mechanical finger unit is hinged to the peripheral edge of the palm base and driven by a drive mechanism to perform a pinching and enveloping action. The pneumatic negative pressure circuit is used to generate and transmit negative pressure adsorption force. The visual acquisition unit is embedded in the palm base and is used to acquire three-dimensional point cloud data of a target object. The acquisition module 201 acquires the motion data of the mechanical finger unit and the real-time air pressure data of the pneumatic negative pressure circuit, and acquires the three-dimensional point cloud data of the target object through the visual acquisition unit. The construction module 202 constructs the visual representation vector corresponding to the three-dimensional point cloud data. The stitching module 203 stitches the motion data, real-time air pressure data, and visual representation vector to obtain a joint observation state vector. The estimation module 204, based on the joint observation state vector and a preset deep reinforcement learning decision network, estimates the benefit evaluation values ​​corresponding to the various grasping action modes performed by the dexterous hand, and determines the action with the highest benefit evaluation value as the target action. The driving module 205 outputs the control command corresponding to the target action, and drives the pneumatic negative pressure circuit and the mechanical finger unit to collaboratively execute the target action based on the control command. In the dexterous hand control scheme provided in this application, the estimation of the benefit evaluation values ​​corresponding to the various grasping action modes performed by the dexterous hand, based on the joint observation state vector and a preset deep reinforcement learning decision network, and determining the action with the highest benefit evaluation value as the target action, replaces the manually preset rules to achieve adaptive intelligent decision-making; the output of the control command corresponding to the target action drives the pneumatic negative pressure circuit and the mechanical finger unit to collaboratively execute the target action, effectively improving grasping accuracy and operational reliability.

[0136] Furthermore, embodiments of this application also provide an electronic device, such as... Figure 9 As shown, it illustrates a structural schematic diagram of the electronic device involved in the embodiments of this application, specifically: The electronic device may include components such as a processor 301 with one or more processing cores, a memory 302 with one or more computer-readable storage media, a power supply 303, and an input unit 304. Those skilled in the art will understand that... Figure 9 The electronic device structure shown does not constitute a limitation on the electronic device and may include more or fewer components than shown, or combine certain components, or have different component arrangements. Wherein: The processor 301 is the control center of the electronic device. It connects various parts of the electronic device via various interfaces and lines, and performs various functions and processes data by running or executing software programs and / or modules stored in the memory 302, and by calling data stored in the memory 302, thereby providing overall monitoring of the electronic device. Optionally, the processor 301 may include one or more processing cores; preferably, the processor 301 may integrate an application processor and a modem processor, wherein the application processor mainly handles the operating system, user interface, and applications, and the modem processor mainly handles wireless communication. It is understood that the modem processor may not be integrated into the processor 301.

[0137] The memory 302 can be used to store software programs and modules. The processor 301 executes various functional applications and dexterous hand control methods by running the software programs and modules stored in the memory 302. The memory 302 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, application programs required for at least one function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created according to the use of the electronic device, etc. In addition, the memory 302 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device. Accordingly, the memory 302 may also include a memory controller to provide the processor 301 with access to the memory 302.

[0138] The electronic device also includes a power supply 303 that supplies power to various components. Preferably, the power supply 303 can be logically connected to the processor 301 through a power management system, thereby enabling functions such as charging, discharging, and power consumption management through the power management system. The power supply 303 may also include one or more DC or AC power supplies, recharging systems, power fault detection circuits, power converters or inverters, power status indicators, and other arbitrary components.

[0139] The electronic device may also include an input unit 304, which can be used to receive input digital or character information and generate keyboard, mouse, joystick, optical or trackball signal inputs related to user settings and function control.

[0140] Although not shown, the electronic device may also include a display unit, etc., which will not be described in detail here. Specifically, in this embodiment, the processor 301 in the electronic device loads the executable files corresponding to the processes of one or more applications into the memory 302 according to the following instructions, and the processor 301 runs the applications stored in the memory 302 to realize various functions, as follows: The motion data of the mechanical finger unit and the real-time air pressure data of the pneumatic negative pressure circuit are collected, and the three-dimensional point cloud data of the target object is collected through the visual acquisition unit; a visual representation vector corresponding to the three-dimensional point cloud data is constructed; the motion data, real-time air pressure data and visual representation vector are concatenated to obtain a joint observation state vector; based on the joint observation state vector and a preset deep reinforcement learning decision network, the benefit evaluation value corresponding to the dexterous hand performing multiple grasping action modes is estimated, and the action with the highest benefit evaluation value is determined as the target action; the control command corresponding to the target action is output, and the pneumatic negative pressure circuit and mechanical finger unit are driven to coordinately execute the target action based on the control command.

[0141] For details on the implementation of each of the above operations, please refer to the previous examples, which will not be repeated here.

[0142] This application embodiment is based on a joint observation state vector and a preset deep reinforcement learning decision network to estimate the benefit evaluation value corresponding to multiple grasping action modes performed by the dexterous hand, and determines the action with the highest benefit evaluation value as the target action, replacing the manually preset rules to achieve adaptive intelligent decision-making; outputs the control command corresponding to the target action to drive the pneumatic negative pressure circuit and mechanical finger unit to perform the target action in a coordinated manner, effectively improving grasping accuracy and operational reliability.

[0143] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be performed by instructions, or by instructions controlling related hardware. These instructions can be stored in a computer-readable storage medium and loaded and executed by a processor.

[0144] Therefore, embodiments of this application provide a storage medium storing a plurality of instructions that can be loaded by a processor to execute steps in any of the dexterous hand control methods provided in embodiments of this application. For example, the instructions can execute the following steps: The motion data of the mechanical finger unit and the real-time air pressure data of the pneumatic negative pressure circuit are collected, and the three-dimensional point cloud data of the target object is collected through the visual acquisition unit; a visual representation vector corresponding to the three-dimensional point cloud data is constructed; the motion data, real-time air pressure data and visual representation vector are concatenated to obtain a joint observation state vector; based on the joint observation state vector and a preset deep reinforcement learning decision network, the benefit evaluation value corresponding to the dexterous hand performing multiple grasping action modes is estimated, and the action with the highest benefit evaluation value is determined as the target action; the control command corresponding to the target action is output, and the pneumatic negative pressure circuit and mechanical finger unit are driven to coordinately execute the target action based on the control command.

[0145] For details on the implementation of each of the above operations, please refer to the previous examples, which will not be repeated here.

[0146] The storage medium may include: read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0147] Since the instructions stored in the storage medium can execute the steps of any of the dexterous hand control methods provided in the embodiments of this application, the beneficial effects that any of the dexterous hand control methods provided in the embodiments of this application can achieve can be realized. For details, please refer to the previous embodiments, which will not be repeated here.

[0148] The above provides a detailed description of a dexterous hand control method, apparatus, device, and storage medium provided in the embodiments of this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A method for controlling a dexterous hand, characterized in that, An application to a dexterous hand includes a palm base, a pneumatic negative pressure circuit, a vision acquisition unit, and a mechanical finger unit. The mechanical finger unit is hinged to the peripheral edge of the palm base and driven by a drive mechanism to perform a pinching and enveloping action. The pneumatic negative pressure circuit generates and transmits negative pressure adsorption force. The vision acquisition unit is embedded in the palm base and is used to acquire three-dimensional point cloud data of a target object. The dexterous hand control method includes: The motion data of the mechanical finger unit and the real-time air pressure data of the pneumatic negative pressure circuit are collected, and the three-dimensional point cloud data of the target object are collected through the vision acquisition unit. Construct the visual representation vector corresponding to the 3D point cloud data; The motion data, real-time air pressure data, and visual representation vector are concatenated to obtain a joint observation state vector; Based on the joint observation state vector and the preset deep reinforcement learning decision network, the reward evaluation value corresponding to the dexterous hand performing multiple grasping action modes is estimated, and the action with the highest reward evaluation value is determined as the target action. Output the control command corresponding to the target action, and drive the pneumatic negative pressure circuit and mechanical finger unit to work together to execute the target action based on the control command.

2. The dexterous hand control method according to claim 1, characterized in that, The construction of the visual representation vector corresponding to the 3D point cloud data includes: A three-dimensional field of view is defined in a preset direction on the palm base, and the three-dimensional field of view is discretized into multiple voxel grids; The occupancy status of the multiple voxel grids is marked based on the three-dimensional point cloud data, and the marked voxel grids are spliced ​​together to obtain the spatial occupancy feature vector corresponding to each voxel grid. Based on the three-dimensional point cloud data and the control points corresponding to the dexterous hand, generate the surface contact feature vector and local geometric feature vector corresponding to the dexterous hand; The spatial occupancy feature vector, surface contact feature vector, and local geometric feature vector are sequentially concatenated and fused to obtain the visual representation vector.

3. The dexterous hand control method according to claim 2, characterized in that, The step of generating surface contact feature vectors and local geometric feature vectors corresponding to the dexterous hand based on the three-dimensional point cloud data and the control points corresponding to the dexterous hand includes: Obtain multiple control points corresponding to the dexterous hand; Calculate the minimum spatial distance from each control point to the surface of the object corresponding to the three-dimensional point cloud data, and determine the nearest neighbor point on the surface corresponding to the minimum spatial distance; Extract the three-dimensional unit normal vector of the object surface at the nearest neighbor point of each surface; The minimum spatial distance corresponding to each control point is concatenated with the three-dimensional unit normal vector, and the concatenation result is adjusted based on a preset combination order to obtain the surface contact feature vector. Select the nearest neighbor point on the surface corresponding to the control point at the fingertip position as the reference point, and use each reference point as the spatial center to extract a local point cloud region within a preset range; High-order geometric feature mapping is performed on each local point cloud region to extract the local geometric feature descriptor corresponding to each reference point; By concatenating the local geometric feature descriptors of all reference points, the local geometric feature vector corresponding to the dexterous hand is obtained.

4. The dexterous hand control method according to claim 1, characterized in that, The step of concatenating the motion data, real-time air pressure data, and visual representation vector to obtain a joint observation state vector includes: The motion data and real-time air pressure data are respectively converted into one-dimensional state vectors with the same format to obtain the motion vector corresponding to the motion data and the real-time air pressure vector corresponding to the real-time air pressure data. The motion vector, real-time air pressure vector, and visual representation vector are sequentially concatenated according to a preset modal arrangement order to obtain a joint observation state vector.

5. The dexterous hand control method according to claim 1, characterized in that, The method, based on the joint observation state vector and a preset deep reinforcement learning decision network, estimates the reward evaluation values ​​corresponding to the various grasping action modalities performed by the dexterous hand, and determines the action with the highest reward evaluation value as the target action, including: The joint observation state vector is input into a preset deep reinforcement learning decision network, and the expected benefit evaluation value corresponding to each grasping action performed by the dexterous hand is output. Select the grabbing action modality with the highest benefit evaluation value as the target action.

6. The dexterous hand control method according to claim 5, characterized in that, Also includes: Construct decision-making and evaluation networks with identical structures; The joint observation state vector sample is input into the decision network, and the decision network selects the candidate action corresponding to the next observation state. The expected return evaluation value corresponding to the candidate action is calculated through the evaluation network to generate a temporal difference objective. The weight parameters of the behavior decision network are updated based on the temporal difference error until the decision network converges, thus obtaining a deep reinforcement learning decision network.

7. The dexterous hand control method according to claim 1, characterized in that, The process of driving the pneumatic negative pressure circuit and the mechanical finger unit to coordinately execute the target action based on the control command includes: If the target action is the first mode, the vacuum generation circuit is activated to control the mechanical finger unit to maintain the open posture, and the flexible suction cup on the mechanical finger unit adheres to the surface of the object to establish negative pressure adsorption. If the target action is the second mode, then the vacuum generation circuit is shut down, and the mechanical finger unit is driven to close and retract. If the target action is the third mode, the vacuum generation circuit is activated to complete the adsorption and anchoring, and after adsorption and anchoring, the mechanical finger unit is driven to close and envelop the target object.

8. A dexterous hand control device, characterized in that, An application to a dexterous hand includes a palm base, a pneumatic negative pressure circuit, a vision acquisition unit, and a mechanical finger unit. The mechanical finger unit is hinged to the peripheral edge of the palm base and driven by a drive mechanism to perform a pinching and enveloping action. The pneumatic negative pressure circuit generates and transmits negative pressure adsorption force. The vision acquisition unit is embedded in the palm base and is used to acquire three-dimensional point cloud data of a target object. The dexterous hand control device includes: The acquisition module is used to acquire motion data of the mechanical finger unit and real-time air pressure data of the pneumatic negative pressure circuit, and to acquire three-dimensional point cloud data of the target object through the vision acquisition unit. The construction module is used to construct the visual representation vector corresponding to the 3D point cloud data; The splicing module is used to splice the motion data, real-time air pressure data and visual representation vector to obtain a joint observation state vector; The prediction module is used to predict the reward evaluation value corresponding to the multiple grasping action modes performed by the dexterous hand based on the joint observation state vector and the preset deep reinforcement learning decision network, and to determine the action with the highest reward evaluation value as the target action. The drive module is used to output control commands corresponding to the target action, and drive the pneumatic negative pressure circuit and the mechanical finger unit to work together to execute the target action based on the control commands.

9. An electronic device, characterized in that, include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements the steps of the dexterous hand control method as described in any one of claims 1-7.

10. A storage medium, characterized in that, The computer program is stored and can be loaded by a processor and executed as described in any one of claims 1-7.

Images