Humanoid robot occlusion grasping method and apparatus

Abstract

The application provides a human-shaped robot shielding grabbing method and device, the method comprising: obtaining a target material, selecting a target sub-array set and an array excitation parameter from a sub-array mapping library according to the material and the end orientation, and controlling a distributed RFID antenna array arranged on a mechanical arm end effector base and / or a finger bone extension area; determining a sampling pose according to a pose uncertainty reduction amount and a cost of moving from a candidate sampling pose to a preset grabbing approach pose, and scanning to obtain a micro scanning frame; monitoring a tactile array, generating a plurality of contact absolute reference events in a grabbing approach or a micro scanning trajectory and forming a continuous tactile contact point sequence, jointly calibrating an RF phase offset and an end external parameter error online based on the continuous tactile contact point sequence, a corresponding mechanical arm pose and contact point geometric information, and grabbing a pose. The application can solve the problems of low pose accuracy, disconnection between scanning and grabbing actions in robot grabbing in a shielding environment.

Description

Technical Field

[0001] This application relates to the fields of robot occlusion grasping and RFID near-field positioning technology, and in particular to a method and device for humanoid robot occlusion grasping. Background Technology

[0002] In scenarios such as smart warehousing, manufacturing sorting, and densely packed picking, humanoid robots need to locate and grasp target objects with RFID tags in occluded, stacked, low-texture, and high-reflectivity environments. Existing pure vision or vision-fusion depth sensor methods are prone to detection loss, pose jumps, and misgrabbing when the target is occluded, making it difficult to simultaneously achieve the three objectives of finding the target, confirming its identity, and completing stable grasping.

[0003] Currently, existing active exploration methods integrating radio frequency and vision at the end of the robotic arm include RFusion and RF-Grasp. These solutions integrate antennas and cameras at the end of the robotic arm, supplemented by active exploration or reinforcement learning, which can improve grasping capabilities in completely occluded environments.

[0004] However, existing technologies suffer from the following core shortcomings: First, they fail to dynamically adjust RF transmission and reception parameters for different object materials (such as metal, liquid, and coated cardboard boxes), resulting in large signal fluctuations and unstable observation quality. Second, the active exploration strategy only aims to reduce pose uncertainty, lacking an online calibration mechanism for RF positioning errors caused by tactile contact events, making it difficult for positioning errors to converge stably before grasping when the end-effector approaches the target. Third, scanning trajectories are mostly fixed, with the scanning endpoint and the final grasping approach pose being independent of each other, causing repetitive movements and jitter during posture switching of the robotic arm. Fourth, pose estimation is mostly limited to coarse positioning or single-tag positioning, lacking the ability to solve six-degree-of-freedom pose (6DoF) problems by combining object geometric priors and multi-tag rigid body constraints, resulting in insufficient robustness in environments with dense occlusion and multipath propagation. In other words, existing technologies are still focused on finding obstructions through RF sensing, and have not yet established a unified closed-loop control chain for end-effector distributed RFID antenna array observation, tactile contact events, and grasping pre-pose. Summary of the Invention

[0005] In view of this, embodiments of this application provide a humanoid robot occlusion grasping method and device to eliminate or improve one or more defects existing in the prior art.

[0006] One aspect of this application provides a method for grasping occlusions in a humanoid robot, comprising: The material category of the target object output by the vision system of the humanoid robot is obtained, and according to the material category and the current end-effector orientation of the humanoid robot's robotic arm, the corresponding target subarray set and array excitation parameters are selected from the pre-built subarray mapping library to control the corresponding antenna unit in the distributed RFID antenna array arranged in the end-effector base and / or finger bone extension area of ​​the robotic arm to transmit and receive RF signals according to the array excitation parameters. The next sampling pose of the robotic arm is dynamically planned, wherein the next sampling pose is jointly determined based on the reduction of pose uncertainty and the cost of moving from the candidate sampling pose to the preset grasping approach pose; the distributed RFID antenna array is controlled to perform scanning sampling on the target object at the next sampling pose and acquire micro-scan frames containing RFID observation data. The output signal of the tactile array arranged on the end effector base is monitored in real time to see if it meets the preset contact triggering conditions. If so, a contact absolute reference event is generated, and a continuous tactile contact point sequence formed by multiple contact absolute reference events is accumulated in the grasping approach or micro-scanning trajectory. Based on the continuous tactile contact point sequence, the corresponding current and historical robot poses and contact point geometric information, the RF phase offset and end effector extrinsic parameter error are jointly calibrated online to obtain online calibration result data containing tactile contact constraints. Based on the RFID observation data and the online calibration results, the visual perception information output by the vision system, the tactile contact constraints, and the preset geometric prior information of the target object are fused to determine the six-degree-of-freedom pose for controlling the robotic arm to perform the operation of grasping the target object.

[0007] In some embodiments of this application, the step of acquiring the material category of the target object output by the humanoid robot's vision system, and selecting the corresponding target subarray set and array excitation parameters from a pre-built subarray mapping library based on the material category and the current end-effector orientation of the humanoid robot's robotic arm, to control the corresponding antenna units in the distributed RFID antenna array arranged in the end-effector base and / or finger bone extension area of ​​the robotic arm to transmit and receive RF signals according to the array excitation parameters, includes: Obtain the material category of the target object output by the humanoid robot's vision system, and obtain the current end-effector orientation of the humanoid robot's robotic arm; Based on the material type and the end face orientation, a pre-built subarray mapping library jointly driven by material and orientation is queried to determine the target subarray set and array excitation parameters corresponding to the humanoid robot for maximizing the signal-to-noise ratio and phase availability of the scanning sampling; wherein, the subarray mapping library contains the target subarray set and array excitation parameters pre-calibrated for the reflection and absorption characteristics of radio frequency signals of different materials under different combinations of material types and different end face orientations; The system controls the antenna units in the distributed RFID antenna array located in the end effector base and / or finger bone extension area of ​​the robotic arm, which correspond to the target subarray set. Based on the array excitation parameters, it forms a near-field radiation mode that is adapted to the target material and end-effector orientation to transmit and receive RF signals.

[0008] In some embodiments of this application, the dynamic planning of the next sampling pose of the robotic arm, controlling the distributed RFID antenna array to perform scanning sampling on the target object at the next sampling pose and acquire micro-scan frames containing RFID observation data, includes: Based on preset optimization criteria, the objective function value corresponding to each candidate sampling pose in the preset candidate sampling pose set is calculated; wherein, the optimization criteria include: a measure of the reduction of pose uncertainty and a cost measure for measuring the cost required to move from the candidate sampling pose to the preset grasping approach pose. Based on the objective function value, select one from the candidate sampling pose set as the next sampling pose of the robotic arm; The robotic arm is controlled to move to the next sampling pose, and the distributed RFID antenna array is controlled to perform scanning sampling on the target object at the next sampling pose; Acquire the micro-scan frame containing RFID phase observations and received signal strength indications obtained after performing a scan sampling.

[0009] In some embodiments of this application, the optimization criterion is characterized by the objective function shown in Formula 1: (Formula 1) Where J(q) represents the objective function value; IG(q) represents the information gain about the target object's pose that can be obtained by performing scan sampling at the candidate sampling pose q; and C pregrasp (q)(q) represents the cost required to move from the candidate sampling pose q to the preset grasping approach pose, and λ is the balance coefficient; the IG(q) and C pregrasp (q)(q) is normalized before weighted summation; IG(q) is calculated based on the reduction in pose uncertainty based on information entropy, the change in trace based on the covariance matrix, or the improvement based on confidence score. C pregrasp (q) Based on the candidate sampling pose q The Euclidean distance to the preset grasping approach pose, the joint space motion cost, or the motion time cost are calculated.

[0010] In some embodiments of this application, when the output signal of the tactile array arranged on the end effector base in real time is monitored to ensure that it meets the preset contact triggering conditions, if so, then based on the continuous tactile contact point sequence formed in the grasping approach or micro-scanning trajectory, a joint online calibration is performed on the error terms related to RF positioning to obtain online calibration result data containing tactile contact constraints, including: The output signal of the tactile array arranged on the end effector base is acquired in real time; Determine whether the output signal meets the preset contact triggering conditions; wherein, the contact triggering conditions include at least one of: detecting a light touch event, detecting an edge contact event, and detecting a pressure change reaching a preset threshold. When the output signal meets the preset contact triggering condition, a contact absolute reference event is generated, and the corresponding contact point is added to the continuous tactile contact point sequence; In response to each of the aforementioned contact absolute reference events, the corresponding current robotic arm pose information and contact point geometry information are obtained, wherein the contact point geometry information includes the contact position and the contact normal; the continuous tactile contact point sequence includes at least multiple contact points obtained under different postures; Based on the continuous tactile contact point sequence, the corresponding current and historical robotic arm pose information, and the contact point geometry information, joint online calibration is performed on the RF positioning-related error terms to calibrate the RF phase offset and end-effector extrinsic error to a unified coordinate system, generating online calibration result data containing tactile contact constraints. The RF positioning-related error terms include: RF phase offset and end-effector extrinsic error; the tactile contact constraints include: contact point position coordinates, contact point normal vector, corresponding robotic arm pose, and timestamp. The term "end-effector extrinsic error" refers to the error in the extrinsic parameter matrix of the end effector relative to the base coordinate system of the humanoid robot.

[0011] In some embodiments of this application, the step of jointly calibrating the RF positioning-related error terms based on the continuous tactile contact point sequence, the corresponding current and historical robotic arm pose information, and the contact point geometry information, so as to calibrate the RF phase offset and the end effector extrinsic error to a unified coordinate system, includes: The continuous tactile contact point sequence, the robotic arm pose information corresponding to each contact point, and the contact point geometric information constitute a calibration constraint set; wherein, the continuous tactile contact point sequence includes at least three non-collinear contact poses; Based on the calibration constraint set, a nonlinear optimization objective function is constructed, which includes contact position residual, contact normal residual, and RF phase residual. The joint estimate of the RF phase offset and the end extrinsic error is obtained by iterative solution. Correction values ​​corresponding to the RF phase offset and the end extrinsic error are generated according to the joint estimate. The online calibration result data also includes the correction values.

[0012] In some embodiments of this application, the step of determining the six-degree-of-freedom pose for controlling the robotic arm to perform the grasping operation based on the RFID observation data and the online calibration result data, by fusing the visual perception information output by the vision system, the tactile contact constraints, and the preset geometric prior information of the target object, includes: Construct a joint optimization objective function; wherein the joint optimization objective function is used to minimize the weighted sum of at least two terms in each residual term: The joint optimization objective function is solved by employing at least one optimization method among factor graph optimization, extended Kalman filtering, or graph neural network constraint inference to obtain a six-degree-of-freedom pose for controlling the robotic arm to perform the operation of grasping the target object. Each of the aforementioned residual terms includes: The RF phase residual and RSSI residual are constructed based on the phase observation values ​​in the RFID observation data and the received signal strength indication; Visual pose residuals are constructed based on the initial six-degree-of-freedom poses in the visual perception information output by the vision system. The tactile contact constraint residual is constructed based on the contact point position coordinates, contact point normal vector and current robot arm pose in the tactile contact constraint. Geometric prior constraint residuals are constructed based on the geometric dimensions or shape features of the target object in the preset geometric prior information; And, a multi-tag rigid body distance constraint residual constructed based on a fixed distance between multiple RFID tags on the target object.

[0013] In some embodiments of this application, the humanoid robot occlusion grasping method further includes: Temporal filtering is performed on the six-DOF poses obtained from solving at multiple consecutive time points to obtain smoothed six-DOF poses; The last sampled pose corresponding to the micro-scan frame is used as the preset grasping approach pose, and the robotic arm is controlled to perform a grasping operation on the target object based on the smoothed six-degree-of-freedom pose. After controlling the robotic arm to perform the operation of grasping the target object, the RFID identification information of the target object is read again and compared with the preset task target identification. If they match, a grasping result confirming successful grasping is generated. The confidence score is determined based on the weights corresponding to each residual term in the process of solving the joint optimization objective function; and a micro-scan frame summary is generated based on the micro-scan frame to generate a capture evidence package; wherein, the capture evidence package includes the RFID identification information, the smoothed six-degree-of-freedom pose, the confidence score, the micro-scan frame summary, the capture result, and a preset signature; Based on the occlusion rate output by the vision system, the signal-to-noise ratio of the RFID observation data, the contact intensity in the tactile contact constraint, and the historical state estimation results, the information matrix weights corresponding to each residual term in the joint optimization objective function are dynamically adjusted.

[0014] Another aspect of this application provides a humanoid robot occlusion grasping system, comprising: The material-aware array element selection module is used to obtain the material category of the target object output by the vision system of the humanoid robot, and select the corresponding target subarray set and array excitation parameters from the pre-built subarray mapping library according to the material category and the current end-effector orientation of the humanoid robot's robotic arm, so as to control the corresponding antenna unit in the distributed RFID antenna array arranged in the end-effector base and / or finger bone extension area of ​​the robotic arm to transmit and receive RF signals according to the array excitation parameters. The information gain micro-scan planning module is used to dynamically plan the next sampling pose of the robotic arm, control the distributed RFID antenna array to perform scanning sampling on the target object at the next sampling pose, and acquire micro-scan frames containing RFID observation data. The tactile absolute reference calibration module is used to monitor in real time whether the output signal of the tactile array arranged on the end effector base meets the preset contact triggering conditions. If so, a continuous tactile contact point sequence is formed in the grasping approach or micro-scanning trajectory. Based on the continuous tactile contact point sequence, the corresponding robotic arm pose and contact point geometric information, the error terms related to RF positioning are jointly calibrated online to obtain online calibration result data containing tactile contact constraints. The geometric constraint pose solving module is used to determine the six-degree-of-freedom pose for controlling the robotic arm to perform the grasping operation based on the RFID observation data and the online calibration result data, the visual perception information output by the vision system, the tactile contact constraints, and the preset geometric prior information of the target object.

[0015] A third aspect of this application provides an electronic device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the humanoid robot occlusion grasping method described above.

[0016] A fourth aspect of this application provides a computer-readable storage medium having a computer program stored thereon that, when executed by a processor, implements the humanoid robot occlusion grasping method described above.

[0017] The fifth aspect of this application provides a computer program product comprising a computer program that, when executed by a processor, implements the humanoid robot occlusion grasping method described above.

[0018] The humanoid robot occlusion grasping method provided in this application can adaptively select the target subarray set and array excitation parameters according to different object materials and the orientation of the robotic arm end effector, thereby suppressing the reflection, absorption, and multipath interference of complex materials such as metals, liquids, and coated cardboard boxes on radio frequency signals, and improving the signal-to-noise ratio and phase availability of near-field observations. By using the reduction of pose uncertainty and the cost of approaching the grasping target pose as optimization criteria to dynamically plan the sampling pose, the scanning trajectory can naturally converge to the grasping approach pose while obtaining information gain, reducing repetitive movements and pose switching jitter of the robotic arm, and improving the continuity of scanning and grasping actions. Utilizing fingertips... The tactile array forms a continuous sequence of multi-pose contact points during the grasping approach or micro-scanning trajectory. Based on this continuous tactile contact point sequence, the radio frequency phase offset and end-effector extrinsic error are jointly calibrated online. This improves the positioning accuracy of the end-effector during the target approach process and enables it to stably converge to a preset grasping accuracy range before grasping. Finally, by fusing radio frequency observation data, visual perception information, tactile contact constraints, and geometric prior information of the target object to jointly optimize and solve the six-degree-of-freedom pose, more stable and accurate pose estimation can be obtained in dense occlusion and multipath environments. This improves the overall grasping success rate and operational efficiency of humanoid robots in occluded scenarios.

[0019] Additional advantages, objectives, and features of this application will be set forth in part in the description which follows, and will in part become apparent to those skilled in the art upon review of the following description, or may be learned by practice of the application. The objectives and other advantages of this application can be realized and obtained by means of the structures specifically pointed out in the specification and drawings.

[0020] Those skilled in the art will understand that the purposes and advantages that can be achieved with this application are not limited to those specifically described above, and that the above and other purposes that this application can achieve will be more clearly understood from the following detailed description. Attached Figure Description

[0021] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, do not constitute a limitation thereof. The components in the drawings are not drawn to scale but are merely for illustrating the principles of this application. For ease of illustration and description of certain parts of this application, corresponding portions in the drawings may be enlarged, i.e., may appear larger relative to other components in an exemplary device actually manufactured according to this application. In the drawings: Figure 1 This is a schematic diagram of the first process of a humanoid robot occlusion grasping method in one embodiment of this application.

[0022] Figure 2 This is a schematic diagram of the second process of the humanoid robot occlusion grasping method in one embodiment of this application.

[0023] Figure 3 This is a schematic diagram of the structure of a humanoid robot occlusion grasping system in one embodiment of this application.

[0024] Figure 4 This is a schematic diagram of the architecture and data flow of a humanoid robot occlusion grasping system in an application example of this application.

[0025] Figure 5 This is a flowchart illustrating the humanoid robot occlusion grasping method in an application example of this application.

[0026] Figure 6 This is a schematic diagram of the integrated contact reference calibration and scanning-grabbing mechanism in an application example of this application. Detailed Implementation

[0027] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the embodiments and accompanying drawings. Here, the illustrative embodiments and their descriptions are used to explain this application, but are not intended to limit it.

[0028] It should also be noted that, in order to avoid obscuring this application with unnecessary details, only the structures and / or processing steps closely related to the solution according to this application are shown in the accompanying drawings, while other details that are not closely related to this application are omitted.

[0029] It should be emphasized that the term "including / comprises" as used herein refers to the presence of a feature, element, step, or component, but does not exclude the presence or addition of one or more other features, elements, steps, or components.

[0030] It should also be noted that, unless otherwise specified, the term "connection" in this article can refer not only to a direct connection, but also to an indirect connection involving an intermediary.

[0031] In the following description, embodiments of the present application will be illustrated with reference to the accompanying drawings. In the drawings, the same reference numerals represent the same or similar parts, or the same or similar steps.

[0032] First, it should be noted that the closest solutions to this application are end-effector integrated RF / vision systems such as RFusion and RF-Grasp. These solutions have demonstrated that integrating antennas and cameras into the end effector of a robotic arm, supplemented by active exploration or reinforcement learning, can improve grasping capabilities in completely occluded environments. However, these solutions focus on "finding occluders through RF sensing" and have not yet established a unified closed loop integrating the end-effector distributed RFID antenna array, tactile contact events, and grasping pre-pose.

[0033] In practical industrial grasping, the problem goes beyond simply "sensing" the target. Different materials exhibit significant differences in the reflection, absorption, and multipath effects of RF signals; during the end-effector's approach to the target, positional errors and phase deviations in the end-effector stage can be amplified into grasping failure; if the scanning path becomes disconnected from the subsequent grasping path, it will cause unnecessary swaying of the robotic arm and increase latency. Therefore, simply combining "RF + vision + touch" is insufficient to form a stable and innovative technical solution.

[0034] Specifically, the aforementioned prior art has the following problems: (1) Existing end-to-end RF solutions typically use single antennas or non-finger-tip antenna layouts, making it difficult to obtain sufficiently stable near-field observations at the grasping end, especially under complex materials such as metal, liquid containers, and coated cardboard boxes, where signal fluctuations are obvious.

[0035] (2) Existing active exploration methods mainly reduce position uncertainty by “moving to obtain more perspectives”, but do not provide a direct link between tactile contact events and RF phase calibration, which makes it impossible to converge the system error in the end stage in real time.

[0036] (3) Traditional scanning trajectories are mostly fixed circular, spiral or grid forms. There is no unified planning between the scanning endpoint and the final grasping approach pose, resulting in repeated motion and unstable posture switching before grasping.

[0037] (4) Existing pose estimation methods mostly rely on coarse localization or single-label localization, lacking a 6DoF solution mechanism that combines object geometric priors with multi-label rigid body constraints, resulting in insufficient robustness under dense occlusion and multipath conditions.

[0038] Based on this, in order to solve the problems of low pose accuracy and disconnect between scanning and grasping actions in robot grasping under occlusion environments in existing methods, the embodiments of this application provide a humanoid robot occlusion grasping method, a humanoid robot occlusion grasping system for executing the humanoid robot occlusion grasping method, a physical device, a computer-readable storage medium, and a computing layer program product. By using material perception-driven array element selection, information gain-driven micro-scanning, tactile absolute reference calibration, geometric constraint pose solving, and scanning-grasping collaborative optimization, the pose accuracy, grasping success rate, and action continuity in occlusion grasping scenarios are improved.

[0039] This application further aims to address the following issues: First, to enable RF observation to truly achieve fingertip-level fine control, rather than merely providing coarse positioning assistance; second, to transform tactile contact events into online RF error calibration criteria when the end-effector approaches the target; third, to allow scanning and grasping actions to share the same optimization target, reducing redundant paths; and fourth, to improve robustness through multimodal confidence adaptive fusion and multi-label geometric constraints.

[0040] The following examples will provide a detailed description.

[0041] Based on this, embodiments of this application provide a method for humanoid robot occlusion grasping that can be implemented by a humanoid robot occlusion grasping system, see [link to relevant documentation]. Figure 1 The humanoid robot occlusion grasping method specifically includes the following: Step 100: Obtain the material category of the target object output by the humanoid robot's vision system, and select the corresponding target subarray set and array excitation parameters from the pre-built subarray mapping library according to the material category and the current end-effector orientation of the humanoid robot's robotic arm, so as to control the corresponding antenna unit in the distributed RFID antenna array arranged in the end-effector base and / or finger bone extension area of ​​the robotic arm to transmit and receive RF signals according to the array excitation parameters.

[0042] It should be noted that the material category is the type of surface material of the object identified by the vision system, used to predict the reflection and absorption characteristics of radio frequency signals. The subarray mapping library is a database that pre-stores the optimal subarray set and array excitation parameters under different combinations of material categories and robotic arm end-effector orientations, which can be obtained through experimental calibration or simulation. The target subarray set is a combination of one or more antenna elements selected from the distributed RFID antenna array, used for current sampling. The array excitation parameters are a set of parameters used to control the amplitude and / or phase weights of each antenna element to form the target's near-field radiation pattern. The distributed RFID antenna array is arranged at the base of the robotic arm end-effector and / or the finger bone extension area to acquire RF observations near the end-effector. The signal-to-noise ratio (SNR) refers to the ratio of the received signal strength to the noise strength; phase availability refers to the stability and repeatability of phase observations, used for subsequent positioning calculations.

[0043] Specifically, in one example, the humanoid robot's vision system (e.g., an RGB-D camera) acquires images of the area where the target object is located. The system then outputs the material category of the target object using target detection and material recognition algorithms (e.g., a deep learning-based classifier), such as "metal," "liquid container," or "coated cardboard box." Simultaneously, the robot controller reads the six-degree-of-freedom pose of the current robotic arm's end effector in the base coordinate system to obtain the end effector orientation. Based on the material category and end effector orientation, the system queries a pre-built "material-orientation-subarray" mapping library. This mapping library is established through offline calibration: for different materials (e.g., the strong reflection characteristics of metal to radio frequency signals, and the strong absorption characteristics of liquids to radio frequency signals), the signal-to-noise ratio (SNR) and phase availability of each subarray combination in the distributed RFID antenna array are tested under different end effector orientations, and the subarray set and array excitation parameters that maximize the SNR and phase availability are recorded. The system then queries to obtain the target subarray set (e.g., selecting antenna elements 2, 3, and 5 in the array) and array excitation parameters (e.g., amplitude weighting coefficients and phase delay values). Subsequently, the controller controls the corresponding target subarray in the distributed RFID antenna array arranged in the end effector base and / or finger bone extension area to transmit and receive radio frequency (RF) signals according to the array excitation parameters.

[0044] Step 200: Dynamically plan the next sampling pose of the robotic arm, control the distributed RFID antenna array to perform scanning sampling on the target object at the next sampling pose, and acquire micro-scan frames containing RFID observation data.

[0045] The next sampling pose is the position and orientation that the robotic arm's end effector will move to, used to perform an RF scan. Scan sampling is the process of acquiring observation data by transmitting signals through a distributed RFID antenna array and receiving reflected or responding signals from RFID tags on a target object while maintaining a fixed pose. A micro-scan frame is the data structure generated from a single scan sampling, containing scan identifiers, sampling poses, subarray identifiers, array excitation parameters, RSSI vector, phase vector, signal-to-noise ratio, and contact markers. The measure of pose uncertainty reduction is an indicator of how much ambiguity a single scan can reduce in estimating the position and orientation of the target object; it is typically implemented as information gain. The grasping approach pose (Pre-Grasp) is the pose the robotic arm needs to reach before grasping the target object, usually located directly above or to the side of the object, a few centimeters away. The cost measure is the resource consumption required to move from the current candidate pose to the grasping approach pose, such as the end effector's linear distance, joint angle changes, or movement time.

[0046] Specifically, in one example, after obtaining the target subarray set and array excitation parameters, the system enters the micro-scanning phase. A set of candidate sampling poses is predefined (e.g., multiple points on a sphere centered on the estimated position of the current target object). For each candidate sampling pose, an objective function is calculated. The candidate sampling pose that maximizes the objective function is selected as the next sampling pose. The controller drives the robotic arm to move to this pose, and then controls the target subarray in the distributed RFID antenna array to transmit and receive radio frequency signals, recording the Received Signal Strength Indicator (RSSI) and phase value of each tag. This data, along with the current sampling pose, subarray number, etc., is packaged into a micro-scan frame. The above process is repeated until a stopping condition is met (e.g., the information gain is below a threshold or the grasping approach pose is approached).

[0047] Step 300: If the output signal of the tactile array arranged on the end effector base meets the preset contact triggering conditions in real time, then the error terms related to RF positioning are calibrated online using the current pose of the robotic arm and the geometric information of the contact point to obtain online calibration result data containing tactile contact constraints.

[0048] Understandably, a haptic array is a sensor array composed of multiple haptic sensitive units, capable of detecting contact position, contact force, pressure distribution, and slippage tendency. Preset contact trigger conditions are pre-defined haptic signal criteria used to identify the moment when the fingertip makes effective contact with a target object or environment. A contact absolute reference event (ContactRef) is a signal that triggers haptic contact, indicating that the robot arm pose and contact geometry at that moment can serve as a local contact geometry reference; multiple contact absolute reference events can further constitute a continuous sequence of haptic contact points for online calibration. Contact point geometry information includes the three-dimensional position coordinates of the contact point in the fingertip coordinate system and the normal vector of the contact surface. RF phase offset is the deviation between the received signal phase and the phase corresponding to the actual distance, caused by changes in the signal propagation path, hardware delay, or multipath effects. End-effector extrinsic error is the deviation between the actual and nominal poses of the gripper, typically caused by robot arm kinematic calibration errors or load deformation. A unified coordinate system is the global reference system of the robot system, such as a base coordinate system or a world coordinate system. Tactile contact constraints are sets of geometric and mechanical information generated by contact events, used as rigid constraints in subsequent pose solving.

[0049] Specifically, in one example, during the micro-scanning process and the grasping approach phase, the system reads the output signal of the tactile array (e.g., capacitive or piezoresistive tactile sensors) arranged on the end effector base in real time. Preset contact triggering conditions include: detecting a light touch (pressure value exceeding a resting threshold and short duration), detecting edge contact (only edge units respond among multiple tactile units), or detecting a pressure change reaching a preset threshold (e.g., pressure change rate greater than 5 kPa / ms). When any condition is met, the system generates a contact absolute reference event (ContactRef) and adds the corresponding contact point position, contact normal, and end effector pose to a continuous tactile contact point sequence. As the robotic arm continues to approach the target in multiple sampling poses, the system continuously accumulates multiple contact absolute reference events. Based on this continuous tactile contact point sequence and the corresponding historical robotic arm pose information, the system performs joint online calibration of the phase offset and end effector extrinsic parameter errors present in the RF positioning, and stores the calibration results as online calibration result data. This data also includes tactile contact constraints, namely the coordinates of each contact point position, contact normal, corresponding robotic arm pose, and timestamp.

[0050] Step 400: Based on the RFID observation data and the online calibration result data, the visual perception information output by the vision system, the tactile contact constraints, and the preset geometric prior information of the target object are fused to determine the six-degree-of-freedom pose for controlling the robotic arm to perform the operation of grasping the target object.

[0051] The visual perception information consists of the detection results of the target object output by the vision system, including initial pose, category, and occlusion area. Pre-defined geometric prior information refers to the known geometric attributes of the target object, such as length, width, height, shape features, and relative positional relationships between multiple tags. The joint optimization objective function is a cost function formed by weighted summation of multiple residual terms; the optimal pose estimate is obtained by minimizing this function. The RF phase residual is the difference between the actual observed phase and the theoretical phase calculated based on the current estimated pose and the signal propagation model. The RSSI residual is the difference between the actual received signal strength and the theoretical strength. The visual pose residual is the deviation between the current estimated pose and the initial pose directly output by the vision system. The tactile contact constraint residual is an error term constructed based on the geometric constraint that the tactile contact point should be located on the object's surface. The geometric prior constraint residual is a regularization term that forces the estimated object size to be consistent with the pre-defined geometric prior. The multi-tag rigid body distance constraint residual is a rigid constraint that the distance between tags should remain constant when multiple RFID tags are attached to the target object. Factor Graph Optimization (FGO) is a nonlinear optimization method based on probabilistic graphical models, suitable for fusing data from different sensor sources. Extended Kalman Filter (EKF) is a filtering algorithm for recursive state estimation of nonlinear systems. Graph Neural Network Constraint Inference is a method that uses graph neural networks to learn constraint relationships and perform inference. Six Degrees of Freedom (6DoF) pose describes an object's position and orientation in three-dimensional space, consisting of three translational degrees of freedom and three rotational degrees of freedom.

[0052] Specifically, in one example, the system acquires multiple micro-scan frames (containing RFID observation data: phase and RSSI) and online calibration result data (containing tactile contact constraints and calibrated error parameters). Simultaneously, the vision system outputs visual perception information of the target object (e.g., initial six-DOF pose, target detection box, occlusion rate). Furthermore, the system pre-stores geometric prior information of the target object (e.g., 3D model, dimensions, fixed distances between multiple RFID tags). A joint optimization objective function is constructed, which includes at least two of the following residual terms: RF phase residual (the difference between the phase observation and the desired phase calculated based on the current estimated pose), RSSI residual, visual pose residual (the difference between the current estimated pose and the initial visual pose), tactile contact constraint residual (rigid constraints constructed using contact point positions and normals), geometric prior constraint residual (the difference between the estimated object size and the preset size), and multi-tag rigid body distance constraint residual (the difference between the estimated distance between multiple tags and the preset fixed distance). Factor graph optimization (FGO), extended Kalman filter (EKF), or graph neural network constraint inference are employed to minimize the weighted sum of residual terms, thereby obtaining the six-DOF pose (position x, y, z and attitude roll, pitch, yaw) of the target object. This pose can be directly used to control the robotic arm to perform grasping operations.

[0053] As described above, the humanoid robot occlusion grasping method provided in this application can adaptively select the target subarray set and array excitation parameters according to different object materials and the orientation of the robotic arm end effector, thereby suppressing the reflection, absorption, and multipath interference of complex materials such as metals, liquids, and coated cardboard boxes on radio frequency signals, and improving the signal-to-noise ratio and phase availability of near-field observations. By using the reduction of pose uncertainty and the cost of approaching the grasping target pose as optimization criteria to dynamically plan the sampling pose, the scanning trajectory can naturally converge to the grasping approach pose while obtaining information gain, reducing repetitive movements and posture switching jitter of the robotic arm, and improving the continuity of scanning and grasping actions. The system utilizes a fingertip tactile array to form a continuous sequence of multi-pose contact points during grasping approaches or micro-scanning trajectories. Based on this continuous tactile contact point sequence, it jointly calibrates the radio frequency phase offset and end-effector extrinsic error online, which can improve the positioning accuracy of the end-effector during target approach and ensure stable convergence to a preset grasping accuracy range before grasping. Finally, by fusing radio frequency observation data, visual perception information, tactile contact constraints, and geometric prior information of the target object for joint optimization to solve the six-degree-of-freedom pose, it can obtain more stable and accurate pose estimation in dense occlusion and multipath environments, thereby improving the overall grasping success rate and operational efficiency of humanoid robots in occluded scenarios.

[0054] To further address the problem that existing technologies fail to dynamically adjust RF transmission parameters for different object materials and end-effector orientations, leading to large signal quality fluctuations and unstable phase observations in complex material environments, this application provides a humanoid robot occlusion grasping method in an embodiment. (See [link to previous document]). Figure 2 Step 100 of the humanoid robot occlusion grasping method specifically includes the following: Step 110: Obtain the material category of the target object output by the humanoid robot's vision system, and obtain the current end-effector orientation of the humanoid robot's robotic arm.

[0055] Step 120: Based on the material type and the end face orientation, query the pre-built subarray mapping library jointly driven by material and orientation to determine the target subarray set and array excitation parameters corresponding to the humanoid robot for maximizing the signal-to-noise ratio and phase availability of the scanning sampling; wherein, the subarray mapping library contains the target subarray set and array excitation parameters pre-calibrated for the reflection and absorption characteristics of different materials to radio frequency signals under different combinations of material types and different end face orientations.

[0056] Step 130: Control the antenna units in the distributed RFID antenna array arranged in the end effector base and / or finger bone extension area of ​​the robotic arm that correspond to the target subarray set, and form a near-field radiation mode adapted to the target material and end face orientation according to the array excitation parameters, and transmit and receive RF signals.

[0057] It should be noted that the material and orientation jointly driven subarray mapping library is a mapping table or function that uses material type and robotic arm end-effector orientation as joint inputs and outputs corresponding optimal subarray sets and array excitation parameters. The joint driving means that the output is determined by both material and orientation conditions simultaneously, rather than a single condition. Reflection and absorption characteristics refer to the different materials' ability to reflect radio frequency signals (e.g., strong reflection from metal surfaces) and absorb signals (e.g., strong absorption from liquids and coated cardboard boxes), affecting the received signal strength and phase stability. Pre-calibration involves experimentally measuring the signal-to-noise ratio (SNR) and phase availability of different subarrays for various materials under multiple orientations, and pre-recording the optimal combination. Pre-calibration also refers to, in the offline stage, controlling the transmission signals of each subarray of the distributed RFID antenna array for different material objects and different robotic arm end-effector orientations, collecting the signal-to-noise ratio (SNR) and phase variance of the echoes, selecting the subarray set and array excitation parameters that maximize SNR and minimize phase variance, and storing them in the mapping library.

[0058] Specifically, in one example, suppose a humanoid robot encounters a metal gear during a sorting task. The vision system identifies the material type as "metal" and simultaneously reads that the current end effector of the robotic arm is pointing vertically downwards (approaching the object). The system queries the "subarray mapping library jointly driven by material and orientation." Pre-calibration data in this library for the "metal + vertically downwards" combination indicates that metal strongly reflects radio frequency signals, and using a full array would introduce severe multipath interference. However, using three centrally symmetrically distributed antenna elements (subarray number S3), configured with equal amplitude and a phase lag of 120° in the near-field excitation parameters, can improve the signal-to-noise ratio by 12dB and reduce the phase availability (phase observation standard deviation) to within 5°. Based on this, the system selects the target subarray set as {S3}, with the aforementioned weights for the array excitation parameters. Subsequently, the controller controls the three antenna elements corresponding to S3 in the distributed RFID antenna array of the end effector base and / or finger bone extension area to transmit and receive radio frequency signals according to the set array excitation parameters.

[0059] As can be seen from the above description, the humanoid robot occlusion grasping method provided in this application embodiment can adaptively select the optimal subarray and array excitation parameters, significantly improve the signal-to-noise ratio and phase availability of near-field observation, and suppress reflection, absorption and multipath interference caused by materials.

[0060] To further address the problem that traditional scanning trajectories do not consider the connection with subsequent grasping actions, resulting in the scanning endpoint being far from the grasping position and increasing redundant motion and time, a humanoid robot occlusion grasping method is provided in this application embodiment, see [link to relevant documentation]. Figure 2 Step 200 of the humanoid robot occlusion grasping method specifically includes the following: Step 210: Based on the preset optimization criteria, calculate the objective function value corresponding to each candidate sampling pose in the preset candidate sampling pose set; wherein, the optimization criteria include: a measure of the reduction of pose uncertainty and a cost measure for measuring the cost required to move from the candidate sampling pose to the preset grasping approach pose.

[0061] Step 220: Select one from the candidate sampling pose set as the next sampling pose of the robotic arm based on the objective function value.

[0062] Step 230: Control the robotic arm to move to the next sampling pose, and control the distributed RFID antenna array to perform scanning sampling on the target object at the next sampling pose.

[0063] Step 240: Obtain the micro-scan frame containing RFID phase observations and received signal strength indications obtained after performing the scan sampling.

[0064] The candidate sampling pose set is a pre-determined set of several possible robot arm end-effector poses (positions and orientations) for scanning. These poses are typically distributed in the space surrounding the target object, such as points on a sphere or grid centered on the currently estimated target position, used to select the best pose for the next scan. The optimization criterion is a rule or function used to evaluate the quality of candidate poses, specifically including two sub-criterions: reducing pose uncertainty (i.e., information gain) and reducing the cost of moving to the grasping approximation pose. The objective function value is a numerical value calculated for each candidate sampling pose according to the optimization criterion, used for comparison and selection. A larger value (or smaller, depending on the definition) indicates a better pose. The cost metric is the resource consumption required to move from the current candidate sampling pose to the preset grasping approximation pose, such as end-effector linear distance, joint movement time, or energy loss. The metric used to measure the cost required to move from the candidate sampling pose to the preset grasping approximation pose is the cost metric.

[0065] Specifically, in one example, assume the target object is a metal gear in a stacked basket, and the current end effector of the robotic arm is located approximately 20cm diagonally above the gear. The system's preset candidate sampling pose set contains five poses: 10cm directly above (q1), 10cm to the left front (q2), 10cm to the right front (q3), 10cm to the left rear (q4), and 10cm to the right rear (q5). The preset grasping approach pose (Pre-Grasp) is located 3cm directly above the gear. For each candidate pose, the information gain term and the cost term for moving from that pose to the grasping approach pose are first normalized to the [0,1] interval, and then the objective function value is calculated. The calculated values ​​are: q1 has a normalized information gain of 0.90, a normalized cost of 0.20, and an objective value of 0.80; q2 has a normalized information gain of 0.60, a normalized cost of 0.75, and an objective value of 0.225; q3 is similar; q4 and q5 are even lower. The maximum target value corresponds to q1, so q1 is selected as the next sampling pose. The controller drives the robotic arm to move to q1, then controls the distributed RFID antenna array to transmit signals and collect echoes, recording phase observations and RSSI to form a micro-scan frame.

[0066] As can be seen from the above description, the humanoid robot occlusion grasping method provided in this application embodiment can enable the scanning pose sequence to naturally converge to the grasping approach pose, reduce the repeated swinging and switching jitter of the robotic arm, and improve the continuity and efficiency of the grasping action.

[0067] To further address the lack of a unified objective function in existing technologies to quantify the balance between information acquisition and action costs, which makes it difficult for scanning strategies to simultaneously achieve accuracy and efficiency, this application provides a humanoid robot occlusion grasping method. The optimization criterion in this method is represented by the objective function shown in Formula 1: (Formula 1) Where J(q) represents the objective function value; IG(q) represents the information gain about the target object's pose that can be obtained by performing scan sampling at the candidate sampling pose q; and C pregrasp (q) represents the cost required to move from the candidate sampling pose q to the preset grasping approach pose, and λ is the balance coefficient; the IG(q) and C pregrasp (q) Normalize the result before weighted summation; IG(q) is calculated based on the reduction in pose uncertainty based on information entropy, the change in trace based on the covariance matrix, or the improvement based on confidence score. C pregrasp (q) Based on the candidate sampling pose q The Euclidean distance to the preset grasping approach pose, the joint space motion cost, or the motion time cost are calculated.

[0068] Information entropy is a metric for measuring the uncertainty of a random variable. In robot localization, the higher the entropy of the probability distribution of the target object's pose, the higher the uncertainty. The amount by which entropy is reduced through scanning and sampling is the reduction in pose uncertainty based on information entropy. The trace of the covariance matrix is ​​the sum of the diagonal elements of the covariance matrix, representing the sum of the variances of the pose estimation across all dimensions. A decrease in the trace of the covariance matrix indicates a decrease in overall uncertainty, and its change can be used as a measure of information gain. The confidence score is a rating of the reliability of the current pose estimation (e.g., between 0 and 1). The increase in the score represents the increase in confidence after scanning and can be used as a substitute measure of information gain. Euclidean distance is the straight-line distance between two points in three-dimensional space. It is the straight-line distance between the candidate sampled pose position and the grasping approach pose position. Joint space motion cost is the weighted sum of all joint angle changes required for the robotic arm to move from the candidate sampled pose to the grasping approach pose, reflecting the mechanical energy consumption or wear of the motion. Motion time cost is the total time required for the robotic arm to move from the candidate sampling pose to the grasping approach pose, including the acceleration, constant speed, and deceleration phases.

[0069] Specifically, in one example, the target object is a metal gear, and the current estimated pose uncertainty is high. Candidate sampling poses q1 (10cm directly above) and q2 (10cm to the left front) are used, with the preset approximation pose being 3cm directly above the gear. When calculating the objective function value using Formula 1, the information gain and cost terms are first normalized, with a balance coefficient λ=0.8.

[0070] For q1: normalized information gain IG(q1) = 0.80, normalized cost C pregrasp If (q1) = 0.20, then: J(q1) = 0.80 - 0.8 × 0.20 = 0.64.

[0071] For q2: normalized information gain IG(q2) = 0.55, normalized cost C pregrasp If (q2) = 0.70, then: J(q2) = 0.55 - 0.8 × 0.70 = -0.01.

[0072] Since 0.64 > -0.01, q1 is chosen as the next sampling pose. In practical applications, the information gain can be calculated based on information entropy, covariance matrix trace, or confidence boost, and the Euclidean distance, joint space motion cost, or motion time cost can be normalized and used as C. pregrasp (q).

[0073] As can be seen from the above description, the humanoid robot occlusion grasping method provided in this application embodiment can flexibly balance the reduction of pose uncertainty and motion cost with adjustable parameter λ, thereby optimizing the time and energy consumption of the overall grasping process.

[0074] To further address the problem in existing technologies where haptic feedback is only used for collision avoidance and does not utilize multiple haptic contact events for joint online calibration of RF positioning errors, leading to difficulties in stable convergence of phase shift and extrinsic parameter errors when the end-effector approaches the target, this application provides a humanoid robot occlusion grasping method, see [link to relevant documentation]. Figure 2 Step 300 of the humanoid robot occlusion grasping method specifically includes the following: Step 310: Acquire the output signal of the tactile array arranged on the end effector base in real time; Step 320: Determine whether the output signal meets the preset contact triggering conditions; wherein the contact triggering conditions include at least one of: detecting a light touch event, detecting an edge contact event, and detecting a pressure change reaching a preset threshold. Step 330: When the output signal meets the preset contact triggering condition, a contact absolute reference event is generated, and the corresponding contact point is added to the continuous tactile contact point sequence; Step 340: In response to each of the contact absolute reference events, obtain the corresponding current robot arm pose information and contact point geometry information, wherein the contact point geometry information includes the contact position and the contact normal; Step 350: Based on the continuous tactile contact point sequence, the corresponding current and historical robotic arm pose information, and the contact point geometry information, perform joint online calibration on the RF positioning-related error terms to calibrate the RF phase offset and end-effector extrinsic error to a unified coordinate system, generating online calibration result data containing tactile contact constraints; wherein, the RF positioning-related error terms include: RF phase offset and end-effector extrinsic error; the tactile contact constraints include: contact point position coordinates, contact point normal vector, corresponding robotic arm pose, and timestamp; the end-effector extrinsic error is the error in the extrinsic matrix of the end effector relative to the humanoid robot's base coordinate system.

[0075] The extrinsic parameter matrix is ​​a homogeneous transformation matrix describing the pose (position and orientation) of the gripper (end-effector) in the robot's base coordinate system. The end-effector extrinsic parameter error is the deviation between the true and nominal values ​​of this matrix. A pressure change reaching a preset threshold occurs when the rate or absolute value of the contact pressure change detected by the tactile sensor exceeds a pre-defined limit, such as a pressure increase of 5 kPa within 10 milliseconds, used to determine a valid contact event. The online calibration result data is a data packet output from the calibration process, containing the RF phase offset obtained iteratively based on a continuous sequence of tactile contact points, the end-effector extrinsic parameter error correction, and tactile contact constraints (position coordinates of each contact point, contact normal vector, corresponding robot arm pose, and timestamp).

[0076] Specifically, in one example, suppose a humanoid robot is grasping a laminated cardboard box. The fingertip tactile array consists of 16 piezoresistive sensors (4×4 grid). The system reads the pressure value of each unit in real time at a frequency of 100Hz. As the robotic arm gradually approaches the target during the grasping and micro-scanning process, it detects light touch or edge contact events in multiple different end-effector postures and generates contact absolute reference events (ContactRef) for each event.

[0077] In response to each of the aforementioned contact absolute reference events, the controller records the corresponding end-effector pose, as well as the coordinates and contact normal of the contact point in the fingertip coordinate system. As the robotic arm continues to move, the system accumulates a continuous tactile contact point sequence S={P1,P2,P3,P4}, where at least three contact poses are non-collinear, thus providing over-constraints for subsequent joint calibration.

[0078] Based on the continuous tactile contact point sequence, the pose information corresponding to each contact point, and the synchronously acquired RF phase observation, the system constructs a nonlinear optimization objective function that includes contact position residual, contact normal residual, and RF phase residual, and performs joint estimation of RF phase offset and end-effector error in a unified coordinate system.

[0079] For example, during a calibration process, the system can optimize the objective function using nonlinear least squares or factor graph iterative solution. When the iteration converges, the corresponding RF phase offset correction and end-parameter error correction are output, and the continuous tactile contact point sequence, the pose of each contact point, the timestamp, and the correction are written into the online calibration result data.

[0080] As can be seen from the above description, the humanoid robot occlusion grasping method provided in this application embodiment can use a multi-pose continuous contact point sequence to perform joint online calibration of RF phase offset and end-effector error, improve near-field positioning accuracy, and make the end-effector error stably converge to a preset grasping accuracy range before grasping.

[0081] To further address the issue that existing technologies can only detect errors but lack a quantitative calibration model based on multi-pose tactile contact sequences, a humanoid robot occlusion grasping method provided in this application embodiment includes the following specific steps in step 350: Step 351: The continuous tactile contact point sequence, the robotic arm pose information corresponding to each contact point, and the contact point geometric information are used to form a calibration constraint set.

[0082] Step 352: Construct a nonlinear optimization objective function based on the calibration constraint set, including contact position residual, contact normal residual, and RF phase residual, and obtain a joint estimate of the RF phase offset and the end-effector error through iterative solution; generate correction values ​​corresponding to the RF phase offset and the end-effector error according to the joint estimate, so as to calibrate the RF phase offset and the end-effector error to a unified coordinate system; the online calibration result data also includes: the correction values ​​corresponding to the RF phase offset and the end-effector error.

[0083] The calibration constraint set is an overdetermined set composed of the robot arm pose, contact point position, and contact normal corresponding to multiple tactile contact events. Each contact event provides a local contact geometric reference. After accumulating multiple contact events under different end-effector poses, they can be used to jointly solve for the phase offset and end-effector extrinsic parameter error in RF localization. The correction value is a compensation value generated based on the joint estimate and is used to update subsequent RF localization and end-effector pose transformations.

[0084] Specifically, in one example, after multiple consecutive tactile contact events are triggered, the robot arm end-effector pose corresponding to each event is recorded as T_base_effector(i), the coordinates of the contact point in the fingertip coordinate system are recorded as p_tip(i), and the contact normal is recorded as n(i), and the corresponding RF phase observations are recorded simultaneously. The system organizes the above observations into a calibration constraint set.

[0085] Under the aforementioned set of calibration constraints, it is assumed that the end-point extrinsic error is... RF phase offset is Then, contact position constraints, contact normal constraints, and phase observation constraints can be established simultaneously based on multiple contact events, thereby forming constraints regarding... and The indeterminate nonlinear equation system.

[0086] Contact position constraints can be expressed as follows: the spatial position of each contact point after correction by nominal pose and extrinsic parameters is geometrically consistent with the target surface. The RF phase constraint can be expressed as: the residual between the observed phase at each moment and the theoretical phase calculated based on the current estimated pose is minimized; The system can use Levenberg-Marquardt optimization, factorial optimization, or other iterative solution methods to minimize the weighted sum of the above residual terms and obtain the final extrinsic parameter error. Phase offset from RF The joint estimation results.

[0087] The joint estimation results are applied to all subsequent RF positioning calculations: new RF phase observations are compensated according to the phase correction amount, and the nominal pose of the robotic arm is transformed using the updated extrinsic parameter matrix so that all data are calibrated to a unified coordinate system.

[0088] In addition to including tactile contact constraints and calibrated error values, the online calibration results data can also explicitly record the continuous tactile contact point sequence, the residual convergence results at each time point, the phase correction amount, and the translation and rotation correction amounts of the external parameters.

[0089] As can be seen from the above description, the humanoid robot occlusion grasping method provided in this application embodiment can convert multi-pose tactile contact sequences into quantifiable RF phase and extrinsic parameter joint correction quantities, realize implementable online error compensation, and improve the long-term stability of the positioning system.

[0090] To further address the issues that existing pose estimation methods are mostly limited to single-modal or single-label localization, failing to fully utilize tactile contact, geometric priors, and multi-label rigid body constraints, and exhibiting poor robustness in occlusion and multipath environments, this application provides a humanoid robot occlusion grasping method in its embodiments. (See [link to relevant documentation]). Figure 2 Step 400 in the humanoid robot occlusion grasping method specifically includes the following: Step 410: Construct a joint optimization objective function; wherein the joint optimization objective function is used to minimize the weighted sum of at least two terms in each residual term: Step 420: Solve the joint optimization objective function using at least one of factor graph optimization, extended Kalman filtering, or graph neural network constraint inference to obtain the six-degree-of-freedom pose for controlling the robotic arm to perform the grasping operation of the target object.

[0091] Each of the aforementioned residual terms includes: (1) The RF phase residual and RSSI residual constructed based on the phase observation values ​​in the RFID observation data and the received signal strength indication; (2) Visual pose residuals constructed based on the initial six-degree-of-freedom poses in the visual perception information output by the vision system; (3) The tactile contact constraint residual is constructed based on the contact point position coordinates, contact point normal vector and current robot arm pose in the tactile contact constraint; (4) Geometric prior constraint residuals constructed based on the geometric dimensions or shape features of the target object in the preset geometric prior information; (5) Multi-tag rigid body distance constraint residuals constructed based on the fixed distance between multiple RFID tags on the target object.

[0092] The residual term represents the difference between the actual observed value and the theoretical value calculated based on the current estimated pose. The optimal pose is solved by minimizing the weighted sum of the residuals during optimization. The weighted sum is obtained by multiplying each residual term by its corresponding weight coefficient and summing the results, balancing the importance of information from different sensors. The RF phase residual is the difference between the phase observation of the RFID tag and the expected phase calculated based on the current estimated pose and signal propagation model, reflecting distance and angle estimation errors. The RSSI residual is the difference between the observed received signal strength indication and the theoretical strength value, which can be used to assist in distance constraints. The visual pose residual is the deviation between the currently estimated target pose and the initial six-DOF pose directly output by the vision system, ensuring that the optimization result does not deviate from the visual observation. The tactile contact constraint residual is an error term constructed based on the rigid constraint that the tactile contact point should be located on the object surface. Specifically, it involves transforming the contact point coordinates to the world coordinate system using the currently estimated object pose and calculating the deviation between it and the actual pose of the robotic arm. Geometric prior constraint residuals are regularization terms constructed based on the pre-defined geometric dimensions or shape features (such as length, width, height, and symmetry) of the target object, forcing the estimated object model to be consistent with the prior. Multi-tag rigid body distance constraint residuals are those where the physical distance between any two RFID tags should remain constant when multiple tags are attached to the target object (rigid body constraint). This residual is the difference between the observed distance between tags and the pre-defined fixed distance. Factor Graph Optimization (FGO) is a nonlinear optimization method based on a probabilistic graphical model. It constructs a bipartite graph with variable nodes (such as pose) and factor nodes (such as residual constraints), and solves for the optimal state by iteratively minimizing the sum of squares of all factor residuals. Extended Kalman Filter (EKF) is a recursive Bayesian estimation algorithm suitable for nonlinear systems, fusing multi-source sensor data through prediction and update steps. Graph Neural Network Constraint Inference is a method that uses graph neural networks to learn constraint relationships and perform inference, capable of handling complex high-order constraints.

[0093] Specifically, in one example, assume the target object is a metal gear with two RFID tags. Pre-defined geometric prior information: the gear diameter is 10cm, thickness is 2cm, and the two tags are located symmetrically on the edge of the gear, with a spacing of 8cm. The vision system outputs the initial six-DOF pose as: position (0.1, 0.2, 0.5)m, orientation (0°, 0°, 30°). Tactile contact constraints: a single contact event records the contact point position (0.02, 0.01, 0)m (fingerprint coordinates), contact normal (0, 0, 1), and the current robotic arm pose T. RFID observation data: the phase observations of the two tags are 120° and 240°, respectively, with RSSI values ​​of -45dBm and -48dBm, respectively.

[0094] Then, a joint optimization objective function is constructed: using a weighted sum form, the following residual terms are selected: (1) RF phase residual: Calculate the difference between the theoretical phase and the actual phase of each tag based on the current estimated pose. For example, the phase residual of tag 1 is 2°.

[0095] (2) RSSI residual: The difference between theoretical RSSI and actual RSSI, for example, the RSSI residual of tag 1 is 1dB.

[0096] (3) Visual pose residuals: translation difference (0.01, 0.02, 0) m and rotation difference 1° between the current estimated pose and the initial visual pose.

[0097] (4) Tactile contact constraint residual: After transforming the contact point from the fingertip coordinate system to the world coordinate system, it should fall on the gear surface. Calculate the distance between the transformed point and the nearest point on the gear surface, for example, 0.5 mm.

[0098] (5) Geometric prior constraint residual: the difference between the currently estimated gear diameter and the preset 10cm, for example, 0.2cm.

[0099] (6) Multi-label rigid body distance constraint residual: Calculate the distance between two labels based on the current estimated pose, and the difference between the distance and the preset 8cm, for example, 0.3cm.

[0100] The weights for each residual term are set as follows: RF phase residual weight 0.5, RSSI residual weight 0.2, visual pose residual weight 0.8, tactile contact constraint residual weight 1.0, geometric prior constraint residual weight 0.6, and multi-label rigid body distance constraint residual weight 0.9. The joint optimization objective function is: 0.5 × (sum of squared phase residuals) + 0.2 × (sum of squared RSSI residuals) + 0.8 × (sum of squared visual pose residuals) + 1.0 × (sum of squared tactile residuals) + 0.6 × (sum of squared geometric prior residuals) + 0.9 × (sum of squared multi-label distance residuals).

[0101] Then, Factor Graph Optimization (FGO) was employed: the six-DOF pose of the target object was used as variable nodes, and each residual term was used as a factor node to construct a factor graph. Iterative optimization was performed using libraries such as GTSAM or Ceres Solver to minimize the total cost. After 10 iterations, the objective function converged, yielding the optimal six-DOF pose: position (0.101, 0.202, 0.501)m, attitude (0.5°, 0.3°, 30.2°). This pose was directly used to control the robotic arm to grasp the gear.

[0102] As can be seen from the above description, the humanoid robot occlusion grasping method provided in this application embodiment can integrate multimodal information (RF, vision, tactile, geometric prior, multi-label distance) for joint optimization, which significantly improves the accuracy and stability of six-degree-of-freedom pose estimation in dense occlusion and multipath environments.

[0103] To further address the shortcomings of existing methods, such as lack of smoothing of pose sequences, identification of grasping individuals, and dynamic weight adjustment based on real-time sensor quality, which leads to poor grasping stability, difficulty in traceability, and inability to adapt to environmental changes, this application provides a humanoid robot occlusion grasping method in an embodiment. (See also...) Figure 2 The humanoid robot occlusion grasping method further includes the following content after step 400: Step 500: Perform temporal filtering on the six-degree-of-freedom poses obtained from solving at multiple consecutive time points to obtain smoothed six-degree-of-freedom poses.

[0104] Step 600: Use the last sampled pose corresponding to the micro-scan frame as the preset grasping approach pose, and control the robotic arm to perform a grasping operation on the target object based on the smoothed six-degree-of-freedom pose.

[0105] Step 700: After controlling the robotic arm to perform the operation of grasping the target object, the RFID identification information of the target object is read again and compared with the preset task target identification. If they match, a grasping result confirming successful grasping is generated.

[0106] Step 800: Determine the confidence score based on the weights corresponding to each residual term in the solution process of the joint optimization objective function; and generate a micro-scan frame summary based on the micro-scan frame to generate a capture evidence package; wherein the capture evidence package includes the RFID identification information, the smoothed six-degree-of-freedom pose, the confidence score, the micro-scan frame summary, the capture result, and a preset signature.

[0107] Step 900: Based on the occlusion rate output by the vision system, the signal-to-noise ratio of the RFID observation data, the contact intensity in the tactile contact constraint, and the historical state estimation results, dynamically adjust the information matrix weights corresponding to each residual term in the joint optimization objective function.

[0108] Temporal filtering smooths the six-DOF pose sequence obtained from multiple consecutive time points (e.g., solving once every 0.1 seconds) to eliminate noise and jitter. Common methods include Kalman filtering, moving average, or low-pass filtering. The smoothed six-DOF pose is a more continuous and stable pose estimate after temporal filtering, used for actual grasping control. The last sampled pose refers to the end-effector pose corresponding to the last scan sample executed during the micro-scanning process. Since the scan trajectory has converged to the vicinity of the grasping approach pose through optimization criteria, this pose can be directly used as the grasping approach pose. RFID identification information is the unique identification code (e.g., EPC code) stored on the RFID tag on the target object, used to confirm the object's identity. The task target identifier is the RFID identification code corresponding to the target object to be grasped, preset in the system, such as the target ID in the task list. The grasping result is the result status of the grasping operation, such as "success," "failure," or "partial success." The confidence score is a quantitative score of the reliability of the final six-DOF pose estimate, typically between 0 and 1. The score can be calculated based on the convergence degree and weight allocation of each residual term during the joint optimization solution process. The smaller the residual and the more reasonable the weight, the higher the score. The micro-scan frame summary is a compilation of key information extracted from all micro-scan frames, such as the number of scans, average signal-to-noise ratio, final sampling pose, and phase variance, used for quickly tracing the scanning process. The capture evidence package is a structured information package recording the entire process data of a complete capture task, used for subsequent auditing, fault analysis, or quality traceability. The preset signature is a pre-generated digital signature or hash check value used to verify the integrity and authenticity of the capture evidence package and prevent data tampering. The occlusion rate is the proportion (0~1) of the target object output by the vision system that is occluded by other objects, for example, the ratio of the visible area to the total area of ​​an object calculated through image segmentation. The signal-to-noise ratio (SNR) (for RFID observation data) is the ratio of the received signal strength to the noise intensity, reflecting the reliability of RFID measurements. The contact strength is the magnitude or pressure value of the contact force detected by the tactile array, reflecting the confidence level of the tactile constraint. Historical state estimation results are a sequence of estimated pose values ​​for the target object over several past time points, used to predict the uncertainty at the current moment. Information matrix weights are elements of the inverse covariance matrix corresponding to each residual term in the joint optimization objective function, used to adjust the influence of that residual term on the overall optimization. Dynamic adjustment means changing these weights based on real-time sensor quality.

[0109] Specifically, in one example, a humanoid robot is grasping a metal gear with an RFID tag. The system has already solved for the six-DOF pose at five consecutive time points (t0~t4) using weight 7. The original pose has slight jitter (e.g., position fluctuates within 0.5mm, and attitude fluctuates within 0.1°). A Kalman filter is used to process the pose sequence. The state variables are set as position and velocity, and the observed value is the pose output by weight 7. After filtering, a smoothed six-DOF pose is obtained: position (0.101, 0.202, 0.501)m, attitude (0.5°, 0.3°, 30.2°), which is more stable than the original values ​​(standard deviation reduced to 0.05mm). During the micro-scan, the last sampled pose is 3.5cm directly above the gear (the next sampled pose). The system automatically sets this pose as the preset grasping approach pose (Pre-Grasp). Then, the robotic arm is controlled to move to the grasping approximation pose based on the smoothed six-DOF pose, and performs a gripper closing operation to grasp the gear. After grasping, the RFID tag on the target object is read again to obtain the identification information "EPC:1234.5678.ABCD". The system compares this information with the preset task target identification "EPC:1234.5678.ABCD", and the two match, so the grasping result is "successful". Based on the final weights of each residual term in the joint optimization objective function solution process (e.g., RF phase residual weight 0.5, final residual 2°; RSSI residual weight 0.2, final residual 1dB; visual pose residual weight 0.8, final translation difference 0.5mm; tactile constraint residual weight 1.0, final residual 0.1mm; geometric prior residual weight 0.6, final residual 0.2cm; multi-tag distance residual weight 0.9, final residual 0.1cm), the confidence score is calculated as the reciprocal of the average normalized residual = 0.92 (out of 1.0). Simultaneously, a summary is generated based on all micro-scan frames: total number of scans = 8, average signal-to-noise ratio = 28dB, final sampling pose = (0.100, 0.200, 0.535)m, phase variance = 3°².

[0110] The system packages the above information into a capture evidence package, which includes: RFID identification information "EPC:1234.5678.ABCD", smoothed six-DOF pose (0.101, 0.202, 0.501)m / (0.5°, 0.3°, 30.2°), confidence score of 0.92, micro-scan frame summary, capture result "successful", and a preset signature. This evidence package is stored in the database for subsequent traceability. Before the next capture task begins (or in subsequent scans of the same task), the system dynamically adjusts the weights in the optimization function based on real-time sensor quality. (1) The occlusion rate of the visual system output is currently 30% (the target part is occluded), so the information matrix weight of the visual pose residual is reduced (from 0.8 to 0.5).

[0111] (2) The signal-to-noise ratio of the RFID observation data is currently 25dB (good), and the RF residual weight is kept at 0.5.

[0112] (3) The contact intensity in the tactile contact constraint is 2N (stronger), and the weight of the tactile residual is increased from 1.0 to 1.2.

[0113] (4) The historical state estimation results (variance of the past 10 poses) show that the pose drift is small, so other weights are kept unchanged.

[0114] Through this adaptive adjustment, the optimization function can more effectively integrate information from multiple sources to address visual degradation caused by occlusion.

[0115] As can be seen from the above description, the humanoid robot occlusion grasping method provided in this application embodiment can achieve smooth and continuous grasping, automatic identity verification and full-process traceability. At the same time, it can adaptively fuse multimodal data based on real-time information such as occlusion and signal-to-noise ratio to improve the robustness and reliability of the system in dynamic environments.

[0116] From a software perspective, this application also provides a humanoid robot occlusion grasping system for performing all or part of the aforementioned humanoid robot occlusion grasping method, see [link to relevant documentation]. Figure 3 The humanoid robot occlusion grasping system specifically includes the following components: The material-aware array element selection module 10 is used to obtain the material category of the target object output by the vision system of the humanoid robot, and select the corresponding target subarray set and array excitation parameters from the pre-built subarray mapping library according to the material category and the current end-effector orientation of the humanoid robot's robotic arm, so as to control the corresponding antenna unit in the distributed RFID antenna array arranged in the end-effector base and / or finger bone extension area of ​​the robotic arm to transmit and receive RF signals according to the array excitation parameters. The information gain micro-scanning planning module 20 is used to dynamically plan the next sampling pose of the robotic arm, control the distributed RFID antenna array to perform scanning sampling on the target object at the next sampling pose, and acquire a micro-scanning frame containing RFID observation data. The tactile absolute reference calibration module 30 is used to monitor in real time whether the output signal of the tactile array arranged on the end effector base meets the preset contact triggering conditions. If so, a continuous tactile contact point sequence is formed in the grasping approach or micro-scanning trajectory. Based on the continuous tactile contact point sequence, the corresponding robotic arm pose and contact point geometric information, the error terms related to RF positioning are jointly calibrated online to obtain online calibration result data containing tactile contact constraints. The geometric constraint pose solving module 40 is used to determine the six-degree-of-freedom pose for controlling the robotic arm to perform the grasping operation of the target object by fusing the visual perception information output by the vision system, the tactile contact constraints, and the preset geometric prior information of the target object, based on the RFID observation data and the online calibration result data.

[0117] The embodiments of the humanoid robot occlusion grasping system provided in this application can be used to execute the processing flow of the embodiments of the humanoid robot occlusion grasping method described above. Its functions will not be repeated here, but can be referred to the detailed description of the embodiments of the humanoid robot occlusion grasping method described above.

[0118] All steps of the humanoid robot occlusion grasping method described in this application are executed in real time on the humanoid robot itself. Specifically, the onboard controller of the humanoid robot (including a processor and memory) is used to execute all processing logic such as material recognition, subarray mapping, scan planning, tactile calibration, pose fusion, and grasping control. The onboard controller can be an embedded computer, an industrial control computer, or a real-time computing unit.

[0119] In an optional implementation, the humanoid robot can interact with a remote server via its communication module. The server can be a server on the task scheduling center side or a server on an intermediate platform (e.g., a third-party server platform with a communication link to the task scheduling center server). The server can be used to: issue grasping task targets, receive grasping evidence packages uploaded by the robot, update subarray mapping library parameters, or perform offline training and model updates. The server can include a single computer device or a server cluster consisting of multiple servers.

[0120] The server and the humanoid robot can communicate using any suitable network protocol, including those not yet developed as of the date of this application. Such network protocols may include, for example, TCP / IP, UDP / IP, HTTP, HTTPS, etc. Furthermore, the network protocols may also include RPC (Remote Procedure Call Protocol), REST (Representational State Transfer Protocol), etc., used on top of the aforementioned protocols.

[0121] As can be seen from the above description, the humanoid robot occlusion grasping system provided in this application embodiment can solve the problems of low pose accuracy and disconnect between scanning and grasping actions in robot grasping under occlusion. By deeply integrating fingertip RFID with tactile sensation, it improves the positioning accuracy of the end stage and meets the grasping accuracy requirements, improves the stability of pose estimation and grasping success rate in occlusion grasping scenarios, and naturally transitions the scanning endpoint to the grasping approach pose, reducing redundant actions and improving grasping continuity and operation efficiency.

[0122] To further illustrate the above embodiments, this application also provides a humanoid robot occlusion grasping method executed by a humanoid robot occlusion grasping system. A distributed RFID antenna array is arranged on the end effector base and / or finger bone extension area, and multiple contact events from the fingertip tactile array are used to form a continuous tactile contact point sequence as an online calibration constraint for RF positioning. During the scanning phase, a fixed trajectory is not used; instead, the next sampling point is selected based on real-time information gain and grasping approach cost. During the pose solving phase, material perception, geometric priors, multi-label rigid body constraints, and contact constraints are incorporated into the same optimization model. This technological chain differs from the broad descriptions of active exploration and RF-assisted grasping in existing technologies.

[0123] This application follows a closed loop: first, identify materials and occlusions; then, select the optimal subarray; next, approximate the target position through information gain scanning; then, complete joint calibration using a sequence of continuous tactile contact points; and finally, output the grasping approximation pose. In other words, scanning is not an independent stage, but directly serves grasping; and tactile feedback is not simply a safety switch, but a source of calibration constraints for fine-tuning RF positioning.

[0124] In this design, the distributed RFID antenna array and the tactile array are not simply placed side by side, but rather share a limited space through partitioned layout, shielding layers, and decoupling design. The vision module outputs the material category and visible area, and the material-aware array element selection module (formerly "array element selection module") selects the optimal subarray or array excitation weights based on the material category and the current orientation of the robotic arm's end effector (formerly "current orientation"). Subsequently, the scanning planner selects the next step from the candidate sampling poses that is "the one with the largest information gain and closest to the preset grasping approach pose (Pre-Grasp)". When the tactile unit detects a contact edge or a light touch event in multiple end effector poses, the tactile absolute reference calibration module (formerly "calibration module") uses the formed continuous tactile contact point sequence to perform joint iterative calibration of the RF phase offset (formerly "RF phase error") and the end effector extrinsic parameter error to reduce the end effector error.

[0125] The humanoid robot occlusion grasping system can be composed of the core functional modules shown in Table 1 below, and the modules are interconnected according to... Figure 4 The data flow and control flow shown work together: Table 1

[0126] The method and procedure of this application are as follows: Figure 5 The steps are executed sequentially as shown, but they are not mechanically parallel; instead, the output of the previous step serves as the valid input for the next step, forming a closed loop. S1: Initial Visual Perception and Material Recognition Acquire an image of the target region, and calculate the target detection bounding box, occlusion rate, initial six-DOF pose, and material category. If the occlusion rate exceeds a threshold or the visual pose covariance is too large, proceed to the RFID micro-scanning process.

[0127] S2: Material-driven subarray selection and near-field excitation configuration The target subarray set and array excitation parameters are selected from a pre-built subarray mapping library jointly driven by material and orientation (formerly the "material-orientation-optimal element combination" mapping library) to maximize the signal-to-noise ratio (SNR) and phase availability in the current sampling phase. Different subarray configurations can be used for objects such as metal surfaces, liquid bottles, and cardboard boxes.

[0128] S3: Information gain-driven adaptive microscanning From the set of candidate sampling poses, the next sampling point is selected according to the objective function.

[0129] S4: Joint online calibration based on continuous tactile contact point sequences When the tactile array detects a light touch, edge contact, or pressure change reaching a preset threshold, it triggers a contact absolute reference event (ContactRef). The system accumulates multiple contact absolute reference events during the grasping approach or micro-scanning trajectory to form a continuous tactile contact point sequence. Using this continuous tactile contact point sequence, the corresponding robotic arm pose, and contact geometry information, the system performs joint online calibration of the RF phase offset and end-effector extrinsic error to achieve online error compensation.

[0130] S5: Geometric Priors and Six-DOF Pose Solving under Multi-Label Constraints A joint optimization objective function is constructed to minimize the weighted sum of at least two of the following residual terms: RF phase residual, RSSI residual, visual pose residual, tactile contact constraint residual, and geometric prior constraint residual. If the target object has multiple RFID tags, a multi-tag rigid body distance constraint residual (i.e., rigid body distance constraint between tags) is also introduced to obtain a more stable six-degree-of-freedom pose solution.

[0131] S6: Confidence-Adaptive Fusion and Temporal Smoothing Based on the occlusion rate output by the vision system, the signal-to-noise ratio of RFID observation data, the contact intensity in the tactile contact constraint, and the historical state estimation results, the weights of the information matrix corresponding to each residual term in the joint optimization objective function are dynamically adjusted, and the pose continuity is maintained through temporal filtering.

[0132] S7: Scanning—Capturing Co-convergence and Identity Verification The last sampling pose (i.e. the final micro-scan endpoint) corresponding to the micro-scan frame is directly used as the preset grasping approach pose (Pre-Grasp) to reduce replanning; after the gripping is completed, the RFID identification information of the target object is read again and compared with the preset task target identification. If they match, a closed-loop grasping is formed.

[0133] in, Figure 6 Corresponding to S4 (joint online calibration based on continuous tactile contact point sequences) and S7 (scan-grasp collaborative convergence), area A is also related to S2 and S3. Area A is for micro-scanning and material subarray selection, area B is for joint calibration after contact events are triggered and a continuous tactile contact point sequence is formed, and area C is for direct conversion of the scan endpoint to Pre-Grasp. Detailed explanations are as follows: (1) Area A (micro-scanning and material subarray selection): corresponds to the material-driven subarray selection in S2 and the information gain-driven adaptive micro-scanning process in S3.

[0134] (2) Area B (Joint online calibration after contact event triggering): Corresponds to the joint online calibration process based on the continuous tactile contact point sequence of S4.

[0135] (3) C region (scan endpoint directly converted to Pre-Grasp): corresponds to the scan-grab co-convergence process of S7.

[0136] The subarray selection can be achieved through a rule base, a learning model, or online search; as long as its output is used to dynamically select the array element set or array excitation weights and participate in subsequent pose solving, it falls under the alternative schemes of this application. The multimodal fusion framework can employ factor graph optimization (FGO), extended Kalman filtering (EKF), or graph neural network constraint inference; as long as it utilizes the joint constraints of vision, RF, and tactile feedback to output the grasping pose, it falls within the scope of this application. Contact events can be output by capacitive, piezoresistive, or optical tactile units; as long as they can form a continuous sequence of tactile contact points and be used for joint online calibration of RF errors and end-effector errors, they do not affect the scope of protection of this application.

[0137] To ensure that technical personnel in this field can directly understand and reproduce this solution, Table 2 below lists the key data structures involved in this application example and their functions.

[0138] Table 2

[0139] Based on this, the system and method provided in this application application example design a distributed RFID antenna array and signal decoupling structure arranged in the end effector base and / or finger bone extension area; a mechanism for selecting RFID subarrays and array excitation parameters based on object material and robotic arm end-effector orientation (i.e., a subarray mapping library jointly driven by material and orientation); a micro-scanning planning mechanism jointly driven by information gain and grasping approach cost; a mechanism for joint online calibration of RF phase offset and end-effector extrinsic error using multi-pose continuous tactile contact point sequences; and a six-degree-of-freedom pose solving mechanism combining geometric priors, multi-label rigid body constraints, and confidence adaptive fusion.

[0140] Therefore, compared with the prior art, the system and method provided in this application example have the following specific beneficial effects: (1) Compared with existing solutions such as RFusion and RF-Grasp, this application arranges the RF sampling position on the end effector base and / or the finger bone extension area, and introduces a material-driven array element selection mechanism, which can better adapt to the RF fluctuation problem caused by complex materials.

[0141] (2) This application uses multiple tactile contact events to form a continuous tactile contact point sequence as a constraint for RF positioning and online calibration. This is different from the conventional scheme that only uses tactile sensation as a collision detection or safety switch, and can improve the error convergence in the end-point near-contact stage.

[0142] (3) This application enables scanning and grasping to share a unified goal through joint planning of information gain and grasping approach cost, thereby reducing repeated paths and improving action continuity.

[0143] (4) This application enhances pose robustness in occlusion, multipath and single-label distortion scenarios by using geometric prior, multi-label rigid body constraints and confidence adaptive fusion.

[0144] This application also provides an electronic device, which may include a processor, a memory, a receiver, and a transmitter. The processor is used to execute the humanoid robot occlusion grasping method mentioned in the above embodiments. The processor and memory can be connected via a bus or other means, taking a bus connection as an example. The receiver can be connected to the processor and memory via wired or wireless means.

[0145] The processor can be a central processing unit (CPU). The processor can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or combinations of the above types of chips.

[0146] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs, non-transitory computer-executable programs, and modules, such as the program instructions / modules corresponding to the humanoid robot occlusion grasping method in the embodiments of this application. The processor executes various functional applications and data processing by running the non-transitory software programs, instructions, and modules stored in the memory, thereby implementing the humanoid robot occlusion grasping method in the above method embodiments.

[0147] The memory may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function; the data storage area may store data created by the processor, etc. Furthermore, the memory may include high-speed random access memory and non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, the memory may optionally include memory remotely located relative to the processor, which can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0148] The one or more modules are stored in the memory, and when executed by the processor, the humanoid robot occlusion grasping method in the embodiment is executed.

[0149] In some embodiments of this application, the user equipment may include a processor, a memory, and a transceiver unit. The transceiver unit may include a receiver and a transmitter. The processor, memory, receiver, and transmitter may be connected via a bus system. The memory is used to store computer instructions, and the processor is used to execute the computer instructions stored in the memory to control the transceiver unit to send and receive signals.

[0150] As one implementation method, the functions of the receiver and transmitter in this application can be implemented by transceiver circuits or dedicated transceiver chips, and the processor can be implemented by dedicated processing chips, processing circuits or general-purpose chips.

[0151] As another implementation approach, the server provided in this application embodiment can be implemented using a general-purpose computer. That is, the program code implementing the processor, receiver, and transmitter functions is stored in memory, and the general-purpose processor implements the processor, receiver, and transmitter functions by executing the code in memory.

[0152] This application also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the aforementioned humanoid robot occlusion grasping method. The computer-readable storage medium can be a tangible storage medium, such as random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, floppy disks, hard disks, removable storage disks, CD-ROMs, or any other form of storage medium known in the art.

[0153] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the aforementioned humanoid robot occlusion grasping method.

[0154] Those skilled in the art will understand that the exemplary components, systems, and methods described in conjunction with the embodiments disclosed herein can be implemented in hardware, software, or a combination of both. Whether implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application. When implemented in hardware, it can be, for example, electronic circuits, application-specific integrated circuits (ASICs), appropriate firmware, plug-ins, function cards, etc. When implemented in software, the elements of this application are programs or code segments used to perform the required tasks. The programs or code segments can be stored on a machine-readable medium or transmitted over a transmission medium or communication link via data signals carried on a carrier wave.

[0155] It should be clarified that this application is not limited to the specific configurations and processes described above and shown in the figures. For the sake of brevity, detailed descriptions of known methods are omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of this application is not limited to the specific steps described and shown. Those skilled in the art can make various changes, modifications, and additions, or change the order of steps, after understanding the spirit of this application.

[0156] In this application, features described and / or illustrated for one embodiment may be used in the same or similar manner in one or more other embodiments, and / or combined with or in place of features of other embodiments.

[0157] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Various modifications and variations can be made to the embodiments of this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A method for grasping occlusions using a humanoid robot, characterized in that, include: The material category of the target object output by the vision system of the humanoid robot is obtained, and according to the material category and the current end-effector orientation of the humanoid robot's robotic arm, the corresponding target subarray set and array excitation parameters are selected from the pre-built subarray mapping library to control the corresponding antenna unit in the distributed RFID antenna array arranged in the end-effector base and / or finger bone extension area of ​​the robotic arm to transmit and receive RF signals according to the array excitation parameters. The next sampling pose of the robotic arm is dynamically planned, wherein the next sampling pose is jointly determined based on the reduction of pose uncertainty and the cost of moving from the candidate sampling pose to the preset grasping approach pose; the distributed RFID antenna array is controlled to perform scanning sampling on the target object at the next sampling pose and acquire micro-scan frames containing RFID observation data. The output signal of the tactile array arranged on the end effector base is monitored in real time to see if it meets the preset contact triggering conditions. If so, a contact absolute reference event is generated, and a continuous tactile contact point sequence formed by multiple contact absolute reference events is accumulated in the grasping approach or micro-scanning trajectory. Based on the continuous tactile contact point sequence, the corresponding current and historical robot poses and contact point geometric information, the RF phase offset and end effector extrinsic parameter error are jointly calibrated online to obtain online calibration result data containing tactile contact constraints. Based on the RFID observation data and the online calibration results, the visual perception information output by the vision system, the tactile contact constraints, and the preset geometric prior information of the target object are fused to determine the six-degree-of-freedom pose for controlling the robotic arm to perform the operation of grasping the target object.

2. The humanoid robot occlusion grasping method according to claim 1, characterized in that, The process involves acquiring the material category of the target object output by the humanoid robot's vision system, and selecting a corresponding target subarray set and array excitation parameters from a pre-built subarray mapping library based on the material category and the current end-effector orientation of the humanoid robot's robotic arm. This is used to control the corresponding antenna units in the distributed RFID antenna array arranged in the end-effector base and / or finger bone extension area of ​​the robotic arm to transmit and receive RF signals according to the array excitation parameters, including: Obtain the material category of the target object output by the humanoid robot's vision system, and obtain the current end-effector orientation of the humanoid robot's robotic arm; Based on the material type and the end face orientation, a pre-built subarray mapping library jointly driven by material and orientation is queried to determine the target subarray set and array excitation parameters corresponding to the humanoid robot for maximizing the signal-to-noise ratio and phase availability of the scanning sampling; wherein, the subarray mapping library contains the target subarray set and array excitation parameters pre-calibrated for the reflection and absorption characteristics of radio frequency signals of different materials under different combinations of material types and different end face orientations; The system controls the antenna units in the distributed RFID antenna array located in the end effector base and / or finger bone extension area of ​​the robotic arm, which correspond to the target subarray set. Based on the array excitation parameters, it forms a near-field radiation mode that is adapted to the target material and end-effector orientation to transmit and receive RF signals.

3. The humanoid robot occlusion grasping method according to claim 1, characterized in that, The dynamic planning of the next sampling pose of the robotic arm, and the control of the distributed RFID antenna array to perform scanning sampling on the target object at the next sampling pose and acquire micro-scan frames containing RFID observation data, includes: Based on preset optimization criteria, the objective function value corresponding to each candidate sampling pose in the preset candidate sampling pose set is calculated; wherein, the optimization criteria include: a measure of the reduction of pose uncertainty and a cost measure for measuring the cost required to move from the candidate sampling pose to the preset grasping approach pose. Based on the objective function value, select one from the candidate sampling pose set as the next sampling pose of the robotic arm; The robotic arm is controlled to move to the next sampling pose, and the distributed RFID antenna array is controlled to perform scanning sampling on the target object at the next sampling pose; Acquire the micro-scan frame containing RFID phase observations and received signal strength indications obtained after performing a scan sampling.

4. The humanoid robot occlusion grasping method according to claim 3, characterized in that, The optimization criterion is characterized by the objective function shown in Formula 1: (Formula 1) Where J(q) represents the objective function value; IG(q) represents the information gain about the target object's pose that can be obtained by performing scan sampling at the candidate sampling pose q; and C pregrasp (q) represents the cost required to move from the candidate sampling pose q to the preset grasping approach pose, and λ is the balance coefficient; the IG(q) and C pregrasp (q) Normalize the result before weighted summation; IG(q) is calculated based on the reduction in pose uncertainty based on information entropy, the change in trace based on the covariance matrix, or the improvement based on confidence score. C pregrasp (q) Based on the candidate sampling pose q The Euclidean distance to the preset grasping approach pose, the cost of joint space motion, or the cost of motion time are calculated.

5. The humanoid robot occlusion grasping method according to claim 1, characterized in that, When the real-time monitoring of the output signal of the tactile array arranged on the end effector base meets the preset contact triggering conditions, if so, then based on the continuous tactile contact point sequence formed in the grasping approach or micro-scanning trajectory, the error terms related to RF positioning are jointly calibrated online to obtain online calibration result data containing tactile contact constraints, including: The output signal of the tactile array arranged on the end effector base is acquired in real time; Determine whether the output signal meets the preset contact triggering conditions; wherein, the contact triggering conditions include at least one of: detecting a light touch event, detecting an edge contact event, and detecting a pressure change reaching a preset threshold. When the output signal meets the preset contact triggering condition, a contact absolute reference event is generated, and the corresponding contact point is added to the continuous tactile contact point sequence; In response to each of the aforementioned contact absolute reference events, the corresponding current robotic arm pose information and contact point geometry information are obtained, wherein the contact point geometry information includes the contact position and the contact normal; the continuous tactile contact point sequence includes at least multiple contact points obtained under different postures; Based on the continuous tactile contact point sequence, the corresponding current and historical robotic arm pose information, and the contact point geometry information, joint online calibration is performed on the RF positioning-related error terms to calibrate the RF phase offset and end-effector extrinsic error to a unified coordinate system, generating online calibration result data containing tactile contact constraints. The RF positioning-related error terms include: RF phase offset and end-effector extrinsic error; the tactile contact constraints include: contact point position coordinates, contact point normal vector, corresponding robotic arm pose, and timestamp. The term "end-effector extrinsic error" refers to the error in the extrinsic parameter matrix of the end effector relative to the base coordinate system of the humanoid robot.

6. The humanoid robot occlusion grasping method according to claim 5, characterized in that, The step of performing joint online calibration of RF positioning-related error terms based on the continuous tactile contact point sequence, the corresponding current and historical robotic arm pose information, and contact point geometric information, to calibrate the RF phase offset and the end effector extrinsic parameter error to a unified coordinate system, includes: The continuous tactile contact point sequence, the robotic arm pose information corresponding to each contact point, and the contact point geometric information constitute a calibration constraint set; wherein, the continuous tactile contact point sequence includes at least three non-collinear contact poses; Based on the calibration constraint set, a nonlinear optimization objective function is constructed, which includes contact position residual, contact normal residual, and RF phase residual. The joint estimate of the RF phase offset and the end extrinsic error is obtained by iterative solution. Correction values ​​corresponding to the RF phase offset and the end extrinsic error are generated according to the joint estimate. The online calibration result data also includes the correction values.

7. The humanoid robot occlusion grasping method according to claim 3, characterized in that, The step of determining the six-degree-of-freedom pose for controlling the robotic arm to perform the grasping operation based on the RFID observation data and the online calibration result data, by fusing the visual perception information output by the vision system, the tactile contact constraints, and the preset geometric prior information of the target object, includes: Construct a joint optimization objective function; wherein the joint optimization objective function is used to minimize the weighted sum of at least two terms in each residual term: The joint optimization objective function is solved by employing at least one optimization method among factor graph optimization, extended Kalman filtering, or graph neural network constraint inference to obtain a six-degree-of-freedom pose for controlling the robotic arm to perform the operation of grasping the target object. Each of the aforementioned residual terms includes: The RF phase residual and RSSI residual are constructed based on the phase observation values ​​in the RFID observation data and the received signal strength indication; Visual pose residuals are constructed based on the initial six-degree-of-freedom poses in the visual perception information output by the vision system. The tactile contact constraint residual is constructed based on the contact point position coordinates, contact point normal vector and current robot arm pose in the tactile contact constraint. Geometric prior constraint residuals are constructed based on the geometric dimensions or shape features of the target object in the preset geometric prior information; And, a multi-tag rigid body distance constraint residual constructed based on a fixed distance between multiple RFID tags on the target object.

8. The humanoid robot occlusion grasping method according to claim 7, characterized in that, Also includes: Temporal filtering is performed on the six-DOF poses obtained from solving at multiple consecutive time points to obtain smoothed six-DOF poses; The last sampled pose corresponding to the micro-scan frame is used as the preset grasping approach pose, and the robotic arm is controlled to perform a grasping operation on the target object based on the smoothed six-degree-of-freedom pose. After controlling the robotic arm to perform the operation of grasping the target object, the RFID identification information of the target object is read again and compared with the preset task target identification. If they match, a grasping result confirming successful grasping is generated. The confidence score is determined based on the weights corresponding to each residual term in the process of solving the joint optimization objective function; and a micro-scan frame summary is generated based on the micro-scan frame to generate a capture evidence package; wherein, the capture evidence package includes the RFID identification information, the smoothed six-degree-of-freedom pose, the confidence score, the micro-scan frame summary, the capture result, and a preset signature; Based on the occlusion rate output by the vision system, the signal-to-noise ratio of the RFID observation data, the contact intensity in the tactile contact constraint, and the historical state estimation results, the information matrix weights corresponding to each residual term in the joint optimization objective function are dynamically adjusted.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the humanoid robot occlusion grasping method as described in any one of claims 1 to 8.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the humanoid robot occlusion grasping method as described in any one of claims 1 to 8.

Images