Adaptive grasping method and system incorporating physical property reasoning and haptic feedback

Abstract

The application discloses an adaptive grasping method and system combining physical property reasoning and tactile feedback, which comprises the following steps: acquiring a language grasping instruction and a visual image of a target object, obtaining a candidate grasping pose of the target object based on the visual image, and reasoning coarse-grained physical properties of the target object by using a visual-language multi-modal model; the coarse-grained physical properties are refined by a serial neural network to obtain fine-grained physical properties and calculate an initial grasping force; an initial grasping is performed based on the initial grasping force and the candidate grasping pose, and a tactile image is collected, a real-time tactile feedback force is calculated based on the generalized Hooke's law; the initial grasping force and the real-time tactile feedback force are fused according to a predetermined rule to obtain an adjusted grasping force according to the stiffness category of the target object; and the grasping action is continuously performed according to the adjusted grasping force until the language grasping instruction is completed.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and more specifically, to an adaptive grasping method and system that combines physical attribute reasoning and tactile feedback. Background Technology

[0002] With the rapid development of artificial intelligence and robotics, robot adaptive grasping technology is gradually shifting from traditional geometric and mechanical driven models to intelligent control methods based on multimodal perception and deep learning. Under this trend, the introduction of the Visual-Language Model (VLM) provides robots with stronger semantic understanding and task generalization capabilities, greatly expanding their application scope in unstructured environments.

[0003] In recent years, pre-trained VLM models such as CLIP (Contrastive Language–Image Pre-training), BLIP (Bootstrapping Language–Image Pre-training), and MiniGPT (Mini Generative Pre-trained Transformer) have been widely used in robot grasping tasks to achieve target recognition, task instruction understanding, and semantic reasoning of physical attributes.

[0004] However, most existing methods rely on pre-trained Virtual Models (VLMs) for semantic recognition and coarse inference of physical properties (such as stiffness and material) of target objects. Since VLMs are not optimized for physical interactions, their outputs typically lack fine-grained physical modeling capabilities, especially when dealing with objects that look similar but have significantly different physical properties, often leading to misjudgments and causing grasping strategies to fail. Furthermore, because VLMs rely on images and natural language descriptions for understanding physical information, lacking a connection to actual contact behavior, their outputs are sensitive to factors such as input perspective, lighting, and background interference. Even for the same target object, inference results may be inconsistent across different scenarios or times, lacking stability and reliability. Simultaneously, the large parameter scale of current mainstream VLM models places high demands on computational resources, making direct deployment on edge devices or mobile robot platforms difficult. This becomes a significant obstacle to practical deployment in resource-constrained, power-sensitive, or real-time-critical application scenarios.

[0005] In addition, many grasping methods based on vision-language models are open-loop systems that fail to fully utilize tactile feedback information during actual contact and are difficult to dynamically adjust grasping strategies, resulting in problems such as improper force control, grasping failure, or damage to objects, especially when dealing with flexible and easily deformable objects. Summary of the Invention

[0006] To address at least one deficiency or improvement need in existing technologies, this invention provides an adaptive grasping method and system that combines physical attribute reasoning and tactile feedback. By introducing a Virtual Property Model (VLM) to perform physical attribute reasoning on the target object, and employing an end-to-end serial physical attribute reasoning method to refine the coarse estimate of the VLM through post-processing, as well as an end-to-end parallel architecture knowledge transfer method, this invention solves the problem of difficulty in deployment on resource-constrained edge devices or real-time systems, significantly improves the accuracy of physical attribute reasoning, and utilizes tactile feedback to adjust the grasping force in real time, thereby improving grasping stability and security.

[0007] To achieve the above objectives, according to a first aspect of the present invention, an adaptive grasping method combining physical property reasoning and tactile feedback is provided. The method includes: acquiring a verbal grasping command and a visual image of a target object; acquiring candidate grasping poses of the target object based on the visual image; and inferring coarse-grained physical properties of the target object using a visual-verbal multimodal model; refining the coarse-grained physical properties through a serial neural network to obtain fine-grained physical properties and calculating an initial grasping force; performing an initial grasping based on the initial grasping force and the candidate grasping poses and acquiring a tactile image; calculating a real-time tactile feedback force based on the generalized Hooke's law; fusing the initial grasping force and the real-time tactile feedback force according to a predetermined rule based on the stiffness category of the target object to obtain an adjusted grasping force; and continuing to perform the grasping action based on the adjusted grasping force until the verbal grasping command is completed.

[0008] In one exemplary embodiment, the method further includes: freezing the pre-trained visual-language multimodal model; The same training sample is received in parallel and outputs physical attribute predictions through a frozen visual-language multimodal model and a physical attribute inference subnetwork, respectively. The output of the frozen visual-language multimodal model is used as a soft label, and the physical attribute inference subnetwork is trained by knowledge distillation through regression loss. During the inference phase, only the physical attribute inference subnetwork is deployed to predict the physical attributes of the target object.

[0009] In an exemplary embodiment, obtaining candidate grasping poses of a target object based on the visual image includes: inputting the visual image into a pre-trained deep neural network to extract image visual features; performing deconvolution and upsampling on the image visual features using a grasping projector to obtain the grasping point, grasping angle, and the width of the gripper opening of the robot's end effector on the target object in the scene; and performing non-maximum suppression and coordinate transformation on the grasping point, grasping angle, and gripper opening width to obtain candidate grasping poses of the target object.

[0010] In an exemplary embodiment, the calculation of real-time tactile feedback force based on the generalized Hooke's law includes: acquiring multiple frames of tactile images in a non-contact state before grasping, calculating and storing an average depth map to eliminate sensor inherent noise; during the grasping process, using a fully connected neural network to predict surface normals and calculate gradient fields pixel-by-pixel on the real-time acquired tactile images; reconstructing a relative depth map from the gradient field by solving the Poisson equation, and then multiplying the relative depth map by a pre-calibrated scaling factor and subtracting the average depth map to obtain an absolute depth map.

[0011] In one exemplary embodiment, after obtaining the absolute depth map, the method further includes: converting the absolute depth map into local compressive strain; calculating the normal force according to the generalized Hooke's law using the Young's modulus estimated in advance by the visual-language multimodal model and the local compressive strain; and using the normal force as a tactile feedback force.

[0012] In an exemplary embodiment, the step of fusing the initial grasping force and the real-time tactile feedback force according to a predetermined rule to obtain an adjusted grasping force based on the stiffness category of the target object includes: classifying the target object into a rigid object or a flexible object based on the object stiffness category estimated in advance by the visual-language multimodal model; when the object is determined to be a rigid object, selecting the larger value between the initial grasping force and the tactile feedback force as the final grasping force; when the object is determined to be a flexible object, selecting the smaller value between the initial grasping force and the tactile feedback force as the final grasping force.

[0013] In an exemplary embodiment, classifying a target object into a rigid object or a flexible object based on the object stiffness category estimated in advance by the visual-linguistic multimodal model includes: invoking the visual-linguistic multimodal model, using a visual image of the target object as input and outputting a Young's modulus estimate; converting the Young's modulus estimate into a calibrated Young's modulus according to an offline calibrated mapping function; classifying the target object as a rigid object if the Young's modulus is greater than or equal to a first fixed threshold; and classifying the target object as a flexible object if the Young's modulus is less than or equal to a second fixed threshold; wherein the first fixed threshold is greater than the second fixed threshold.

[0014] According to a second aspect of the present invention, an adaptive grasping system combining physical property reasoning and tactile feedback is also provided, executing the above-described adaptive grasping method combining physical property reasoning and tactile feedback, comprising: an acquisition module configured to acquire a language grasping command and a visual image of a target object; a grasping prediction module configured to acquire candidate grasping poses of the target object based on visual input; a physical property reasoning module configured to obtain an initial physical property estimate of the target object through a pre-trained visual-language multimodal model; a tactile feedback module configured to acquire a real-time tactile image of the target object and calculate a tactile feedback force during the grasping process; and a force type analysis module configured to fuse the initial grasping force corresponding to the initial physical property estimate with the tactile feedback force based on the stiffness category of the target object to generate a final grasping force command; wherein the force type analysis module takes the larger value between the initial grasping force and the tactile feedback force for rigid objects and the smaller value for flexible objects, performing closed-loop adaptive grasping.

[0015] According to a third aspect of the invention, a computer-readable storage medium is also provided, wherein a computer program is stored therein, wherein the computer program is configured to execute the above-described adaptive grasping method combining physical attribute reasoning and haptic feedback at runtime.

[0016] According to a fourth aspect of the invention, an electronic device is also provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the aforementioned adaptive grasping method combining physical attribute reasoning and tactile feedback through the computer program.

[0017] In summary, compared with the prior art, the above-described technical solutions conceived by this invention can achieve the following beneficial effects: (1) This invention provides an adaptive grasping method that combines physical property reasoning and tactile feedback. It designs two modules: tactile feedback and force type analysis. By combining vision- and language-based multimodal methods with tactile feedback-based methods, it utilizes the physical prior knowledge of VLM to ensure generalization ability while providing real-time feedback and adjusting the grasping force. This avoids excessive force or insufficient gripping by the gripper, improves grasping stability and safety, and achieves closed-loop control.

[0018] (2) By using a two-stage inference architecture, the coarse estimate of VLM is refined through post-processing, which makes up for its deficiency in physical detail modeling and significantly improves the inference accuracy. The regression network introduces consistency constraints and physical prior supervision during training, which enables the same object to output consistent and stable physical properties under multiple observations, effectively solving the problem of unstable original inference results.

[0019] (3) Through the knowledge distillation transfer mechanism, the prior capabilities of the pre-trained model are preserved while avoiding the loading of a large VLM during the inference stage, which effectively reduces the demand for video memory, computing resources and power consumption, and is suitable for resource-constrained environments such as mobile robots and embedded systems. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 A flowchart illustrating an optional adaptive grasping method combining physical property reasoning and tactile feedback, provided for an embodiment of this application; Figure 2 A flowchart illustrating another optional adaptive grasping method combining physical property reasoning and tactile feedback provided for an embodiment of this application; Figure 3 A schematic diagram of the architecture of an optional adaptive grasping system combining physical property reasoning and tactile feedback, provided for an embodiment of this application; Figure 4 This is a schematic diagram of an optional electronic device provided in an embodiment of this application. Detailed Implementation

[0022] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.

[0023] The terms "first," "second," "third," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish different objects, not to describe a specific order. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or apparatuses.

[0024] According to one aspect of the embodiments of this application, an adaptive grasping method combining physical property reasoning and tactile feedback is provided. The following is in conjunction with... Figure 1 This application describes an adaptive grasping method that combines physical property reasoning and tactile feedback, as provided in its embodiments.

[0025] Figure 1 This is a flowchart illustrating an optional adaptive grasping method combining physical property reasoning and tactile feedback, as provided in an embodiment of this application. Figure 1 As shown, the process of this method may include the following steps: S102, acquire the language grasping command and the visual image of the target object, acquire the candidate grasping pose of the target object based on the visual image, and use the vision-language multimodal model to infer the coarse-grained physical properties of the target object. S104, The coarse-grained physical properties are refined by regression through a serial neural network to obtain fine-grained physical properties and calculate the initial gripping force; S106, perform initial grasping and acquire tactile images based on the initial grasping force and the candidate grasping pose, and calculate real-time tactile feedback force based on the generalized Hooke's law; S108, according to the stiffness category of the target object, the initial gripping force and the real-time tactile feedback force are fused according to a predetermined rule to obtain an adjusted gripping force; S110, based on the adjusted gripping force, continues to perform the gripping action until the language gripping command is completed.

[0026] The embodiments of this application provide an adaptive grasping method that combines physical property reasoning and tactile feedback, which can be applied to scenarios where mobile robots adaptively grasp target objects.

[0027] Figure 2 A flowchart illustrating another optional adaptive grasping method combining physical property reasoning and haptic feedback, provided for embodiments of this application, is shown below. Figure 1 and Figure 2As shown, exemplarily, when a robot grasps a target object, the operator can give the robot verbal grasping commands via voice or text, such as "gently place the bag of soft bread on the left onto the plate." The grasping robot simultaneously acquires color visual images and depth images of the current scene of the target object using an RGB-D camera. The acquired visual images are input into a pre-trained ResNet-101 (Residual Network with 101 layers, a deep neural network composed of 101 residual blocks) backbone to extract visual feature maps. The visual feature maps are then processed by a grasping projector (convolution and upsampling) to output pixel-level grasping confidence, grasping angle, and gripper opening. Post-processing can retain the top 5 candidate grasping poses, or other numbers of candidate grasping poses; here, 5 are used as an example. The visual images and verbal commands are fed together into a contrastive learning-based language-visual pre-trained model with frozen parameters. Text-image alignment is performed, and then the MLP head outputs coarse-grained physical properties.

[0028] Coarse-grained physical properties are input into a two-layer serial neural network (e.g., a 256-unit ReLU MLP), which outputs fine-grained physical properties. The initial gripping force is calculated using the calibrated Young's modulus and gripper geometry parameters, and the initial gripping is performed. The robot approaches the object with the highest confidence pose from the candidate gripping pose set, and the gripper closes with the initial expected gripping force (i.e., the initial gripping force) calculated based on the fine-grained physical properties.

[0029] The GelSight visual-tactile sensor continuously outputs a sequence of grayscale images after obtaining a series of tactile feedback images. Normal estimation, Poisson reconstruction, and absolute depth mapping are performed on the grayscale image sequence to calculate the average compressive strain and the real-time tactile feedback force. Stiffness category determination and force fusion are then performed to determine the adjusted gripping force. The process is executed in a closed loop until the command is completed. For example, after stabilization, the robot moves the object above the "plate" according to a verbal command, releases the gripper, and the task is completed.

[0030] Through steps S102 to S110, by acquiring a language grasping command and a visual image of the target object, candidate grasping poses of the target object are obtained based on the visual image, and coarse-grained physical properties of the target object are inferred using a visual-language multimodal model; the coarse-grained physical properties are refined by regression using a serial neural network to obtain fine-grained physical properties and calculate the initial grasping force; the initial grasping is performed based on the initial grasping force and the candidate grasping poses, and tactile images are acquired; the real-time tactile feedback force is calculated based on the generalized Hooke's law; according to the stiffness category of the target object, the initial grasping force and the real-time tactile feedback force are fused according to a predetermined rule to obtain an adjusted grasping force; the grasping action continues to be executed based on the adjusted grasping force until the language grasping command is completed. This solves the problem of difficulty in deployment on resource-constrained edge devices or real-time systems, significantly improves the accuracy of physical property inference, and improves grasping stability and security by adjusting the grasping force in real time using tactile feedback.

[0031] Figure 3 This is a schematic diagram of an optional adaptive grasping system combining physical property reasoning and tactile feedback, provided in an embodiment of this application. Figure 3 As shown, the system includes: The acquisition module is configured to acquire language capture commands and visual images of the target object; The grasping prediction module is configured to obtain candidate grasping poses of the target object based on visual input; The physical property reasoning module is configured to obtain initial physical property estimates of the target object through a pre-trained visual-linguistic multimodal model; The tactile feedback module is configured to acquire real-time tactile images of the target object and calculate tactile feedback force during the grasping process; The force type parsing module is configured to fuse the initial grasping force corresponding to the initial physical property estimate and the tactile feedback force based on the stiffness category of the target object to generate the final grasping force command. Specifically, the force type analysis module takes the larger value between the initial grasping force and the tactile feedback force for rigid objects and the smaller value for flexible objects, and performs closed-loop adaptive grasping.

[0032] Optionally, the capture prediction module (i.e., Figure 3The grasping module in the image is as follows: RGB images are input into a pre-trained ResNet-101 (Residual Network with 101 layers, a deep neural network composed of 101 residual blocks) model to extract visual features. Then, the image passes through a grasping projector, a neural network consisting of a series of convolutional layers, upsampling layers, and non-linear activation functions, to obtain the grasping point, grasping angle, and the width of the end effector gripper's opening on the target object in the scene. Finally, post-processing is performed to obtain the grasping pose of the target object, thus executing the grasping action.

[0033] The physical property reasoning module integrating VLM (Vision-Language Model) is used to obtain the physical properties of the target object, such as mass and friction, by introducing prior knowledge of VLM and using physical property reasoning methods, and then calculate the initial gripping force.

[0034] Haptic feedback module: By grasping the target object, the end gripper equipped with the GelSight visual tactile sensor obtains a series of tactile feedback images.

[0035] Force type analysis module: By integrating the VLM inference module and haptic feedback module, the initial grasping force and haptic feedback force are obtained respectively. This module needs to calculate a more accurate grasping force based on objects with different rigidities.

[0036] By combining vision- and language-based multimodal methods with tactile feedback-based methods, and utilizing the physical prior knowledge of VLM to ensure generalization ability, the system can provide real-time feedback and adjust the gripping force to avoid excessive force or insufficient gripping, thereby improving gripping stability and safety and achieving closed-loop control.

[0037] In one exemplary embodiment, the method further includes: S11, freeze the pre-trained visual-language multimodal model; S12 receives the same training sample in parallel and outputs physical attribute predictions through the frozen visual-language multimodal model and the physical attribute reasoning sub-network, respectively. S13, using the output of the frozen visual-language multimodal model as a soft label, the physical attribute reasoning subnetwork is trained by knowledge distillation through regression loss; S14, During the inference phase, only the physical attribute inference subnetwork is deployed to predict the physical attributes of the target object.

[0038] In this embodiment, the inference module integrating VLM is further described. Optionally, the physical properties of the target object, such as mass and friction, are predicted using the following three different physical property inference methods: 1) Directly using VLM for reasoning: mainly relies on the physical prior knowledge of the pre-trained VLM to reason about the physical properties of the target object, such as mass and friction.

[0039] 2) End-to-end serial neural network architecture: First, VLM is used to obtain a rough estimate of physical properties from images and natural language prompts. Then, a neural network trained based on physical prior knowledge is used to regress and refine the estimation results of VLM to predict more accurate physical properties, and finally used to calculate the initial gripping force.

[0040] 3) End-to-end parallel neural network architecture: Physical property estimates are obtained in parallel from a parameter-frozen VLM and a lightweight network, respectively. The two results are then averaged to enhance the robustness of physical property prediction, and finally used for the calculation of the initial grasping force. This method is similar to knowledge distillation, which is equivalent to using the output of the VLM as a soft label or knowledge distillation source.

[0041] In this embodiment, a knowledge transfer method based on an end-to-end parallel architecture is employed, using a knowledge distillation-like mechanism, to transfer the implicit physical property knowledge from a frozen pre-trained Virtual Model (VLM) to a lightweight physical property inference sub-network. This sub-network is trained in parallel with the VLM and continuously learns the physical properties output by the VLM during the training phase using a specific loss function (such as regression loss). During the inference phase, there is no need to call the original large model; physical property prediction can be completed solely relying on this lightweight quantum network, thereby supporting subsequent grasping force estimation.

[0042] In one exemplary embodiment, obtaining the candidate grasping pose of the target object based on the visual image includes: S21, The visual image is input into a pre-trained deep neural network to extract the visual features of the image; S22, the image visual features are deconvolved and upsampled by the grasping projector to obtain the grasping point, grasping angle and the width of the gripper opening of the end effector of the grasping robot in the scene. S23, perform non-maximum suppression and coordinate transformation on the grasping point, grasping angle and gripper opening width to obtain the candidate grasping pose of the target object.

[0043] In this embodiment of the application, the input visual image is encoded into high-dimensional features by a pre-trained deep network, and the grasping projector performs deconvolution and upsampling on the features to directly regress the three parameters of the grasping point pixel coordinates, grasping angle, and gripper opening width in the image space.

[0044] Furthermore, non-maximum suppression is applied to the obtained discrete pixel-level results to remove redundant predictions. Then, the results are converted into three-dimensional poses in the robot's base coordinate system through in-camera hand-eye calibration, thus forming a candidate grasping pose list for subsequent force control and execution stages.

[0045] This embodiment achieves complete pose (grasp point, angle, and opening) in a single network forward pass, eliminating the need for the multi-stage error accumulation of traditional methods that involve detection followed by fitting. High spatial resolution is maintained through a deconvolution-upsampling structure, and NMS uses only a distance threshold for filtering, reducing overall latency and meeting real-time closed-loop requirements. It is versatile, easy to deploy, and can be seamlessly embedded into edge computing platforms.

[0046] In one exemplary embodiment, the calculation of real-time tactile feedback force based on the generalized Hooke's law includes: S31: Acquire multiple frames of tactile images in a non-contact state before grasping, calculate and store the average depth map to eliminate sensor inherent noise; S32, during the grasping process, uses a fully connected neural network to predict the surface normal and calculate the gradient field pixel by pixel on the real-time acquired tactile images; S33. The relative depth map is reconstructed from the gradient field by solving the Poisson equation. Then, the relative depth map is multiplied by a pre-calibrated scaling factor and the average depth map is subtracted to obtain the absolute depth map.

[0047] In this embodiment, for example, a visual-tactile sensor acquires multiple frames of tactile images in a non-contact state before grasping. Here, the deformation depth of the gel on the visual-tactile sensor surface is characterized by a depth map. However, since this sensor cannot acquire depth information, a series of methods are needed to obtain the deformation depth map. Specifically, a fully connected neural network is first used to predict the surface normal, then the gradient field is calculated using the normal, and finally the Poisson equation is used to solve for the relative depth. The sensor's inherent noise is eliminated by averaging the depth of several frames in a non-contact state, and the absolute depth map is obtained by multiplying the relative depth map by a scaling factor.

[0048] In one exemplary embodiment, after obtaining the absolute depth map, the method further includes: S41, convert the absolute depth map into local compressive strain; S42, using the Young's modulus estimated in advance by the visual-language multimodal model and the local compressive strain, calculate the normal force according to the generalized Hooke's law, and use the normal force as the tactile feedback force.

[0049] In this embodiment, for parallel grasping, the deformation of the gel on the surface of the visual-tactile sensor can be approximated as unidirectional compression. Simultaneously, the gel on the sensor surface is considered an isotropic material, and the strain can be calculated using an absolute depth map. Given Young's modulus and strain, the normal force causing the sensor surface deformation is estimated according to the generalized Hooke's law, and this force is used as the tactile feedback force to adjust the grasping force.

[0050] In an exemplary embodiment, the step of fusing the initial gripping force and the real-time haptic feedback force according to a predetermined rule based on the stiffness category of the target object to obtain an adjusted gripping force includes: S51, based on the object stiffness category estimated in advance by the vision-language multimodal model, classify the target object into a rigid object or a flexible object; S52, when the object is determined to be a rigid object, the larger value between the initial grasping force and the tactile feedback force is selected as the final grasping force; S53, when the object is determined to be a flexible object, the smaller value between the initial grasping force and the tactile feedback force is selected as the final grasping force.

[0051] It is understandable that for rigid objects, due to their high stiffness, the local deformation is small when in contact, and the applied clamping force is mainly concentrated near the contact point. The tactile feedback force will quickly approach or exceed the initial gripping force. At this time, the larger value between the initial gripping force and the tactile feedback force is taken (because rigid objects are not easily deformed, there is no situation where the clamping force is too large and damages the object. Taking the larger value can provide greater friction to prevent slippage).

[0052] For flexible objects, their low stiffness leads to significant deformation of the contact surface, dispersing the gripping force over a larger contact area. The tactile feedback force may be lower than the initial grasping force. Furthermore, the normal force is estimated using a deformation depth map of the object being grasped, applying the same force to grasp objects of the same shape but different stiffness. The lower the stiffness of the object, the smaller the value in the depth map, and thus the smaller the estimated normal force. In this case, the smaller of the initial grasping force and the tactile feedback force should be used to avoid excessive force that could damage the object.

[0053] In one exemplary embodiment, classifying the target object into a rigid object or a flexible object based on the object stiffness category estimated in advance by the visual-language multimodal model includes: S61, invoke the vision-language multimodal model, using the visual image of the target object as input and output Young's modulus estimate; S62, convert the estimated Young's modulus value into the calibrated Young's modulus based on the offline calibrated mapping function; S63 If ​​the Young's modulus is greater than or equal to the first fixed threshold, then the target object is classified as a rigid object; S64, if the Young's modulus is less than or equal to the second fixed threshold, then the target object is classified as a flexible object; Wherein, the first fixed threshold is greater than the second fixed threshold.

[0054] Specifically, during the offline calibration phase, M samples of known materials (metal, hard plastic, soft rubber, foam, etc.) are collected, and their true Young's modulus is measured using a standard mechanical testing machine. Simultaneously, the VLM inference module of this system is used to estimate the RGB images of the samples to obtain the estimated Young's modulus value of the model output.

[0055] Using the true Young's modulus as the horizontal axis and the estimated Young's modulus as the vertical axis, a linear or piecewise fitting is performed to obtain the mapping function f(·), such that E_calib = f(E_vlm), where E_calib is the calibrated Young's modulus and E_vlm is the estimated Young's modulus. Stiffness thresholds are set, including a second fixed threshold T_low and a first fixed threshold T_high (which can be determined by experience or engineering requirements). If E_calib ≥ T_high, it is marked as a "rigid object"; if E_calib ≤ T_low, it is marked as a "flexible object"; and objects between these two values ​​are marked as "medium stiffness," with the system defaulting to a conservative flexible strategy.

[0056] This embodiment demonstrates how to match corresponding grasping strategies to target objects with different stiffness, thereby improving the success rate of grasping actions, avoiding damage to objects, and enhancing the adaptability and stability of robot grasping in real and complex environments.

[0057] According to another aspect of the embodiments of this application, a storage medium is also provided. Optionally, in this embodiment, the storage medium can be used to execute the program code of any of the adaptive grasping methods combining physical attribute reasoning and haptic feedback described in the embodiments of this application.

[0058] Optionally, in this embodiment, the storage medium is configured to store program code for performing the following steps: S1, acquire language grasping instructions and visual images of the target object, acquire candidate grasping poses of the target object based on the visual images, and use a visual-language multimodal model to infer the coarse-grained physical properties of the target object. S2, the coarse-grained physical properties are refined by regression through a serial neural network to obtain fine-grained physical properties and calculate the initial gripping force; S3, perform initial grasping and acquire tactile images based on the initial grasping force and the candidate grasping pose, and calculate real-time tactile feedback force based on the generalized Hooke's law; S4, according to the stiffness category of the target object, the initial gripping force and the real-time tactile feedback force are fused according to a predetermined rule to obtain the adjusted gripping force; S5, based on the adjusted gripping force, continues to execute the gripping action until the language gripping command is completed.

[0059] Optionally, specific examples in this embodiment can refer to the examples described in the above embodiments, and will not be repeated in this embodiment.

[0060] The computer-readable storage medium may include, but is not limited to, any type of disk, including floppy disks, optical disks, DVDs, CD-ROMs, microdrives, as well as magneto-optical disks, ROMs, RAMs, EPROMs, EEPROMs, DRAMs, VRAMs, flash memory devices, magnetic cards or optical cards, nanosystems (including molecular memory ICs), or any type of medium or device suitable for storing instructions and / or data.

[0061] According to another aspect of the embodiments of this application, an electronic device is also provided for implementing the above-described adaptive grasping method combining physical attribute reasoning and tactile feedback, the electronic device being a server, a terminal, or a combination thereof.

[0062] Figure 4 This is a schematic diagram of the structure of an optional electronic device according to an embodiment of this application, such as... Figure 4 As shown, it includes a processor 402, a communication interface 404, a memory 406, and a communication bus 408. The processor 402, communication interface 404, and memory 406 communicate with each other via the communication bus 408. Memory 406 is used to store computer programs; When processor 402 executes a computer program stored in memory 406, it performs the following steps: S1, acquire language grasping instructions and visual images of the target object, acquire candidate grasping poses of the target object based on the visual images, and use a visual-language multimodal model to infer the coarse-grained physical properties of the target object. S2, the coarse-grained physical properties are refined by regression through a serial neural network to obtain fine-grained physical properties and calculate the initial gripping force; S3, perform initial grasping and acquire tactile images based on the initial grasping force and the candidate grasping pose, and calculate real-time tactile feedback force based on the generalized Hooke's law; S4, according to the stiffness category of the target object, the initial gripping force and the real-time tactile feedback force are fused according to a predetermined rule to obtain the adjusted gripping force; S5, based on the adjusted gripping force, continues to execute the gripping action until the language gripping command is completed.

[0063] Optionally, the communication bus can be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. This communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 4 The symbol is represented by a single thick line, but this does not indicate that there is only one bus or one type of bus. The communication interface is used for communication between the aforementioned electronic device and other devices.

[0064] The memory may include RAM, or non-volatile memory, such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.

[0065] The processors mentioned above can be general-purpose processors, including but not limited to: CPU (Central Processing Unit), NP (Network Processor), etc.; they can also be DSP (Digital Signal Processor), ASIC (Application Specific Integrated Circuit), FPGA (Field-Programmable Gate Array), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0066] Optionally, specific examples in this embodiment can refer to the examples described in the above embodiments, and will not be repeated here.

[0067] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to this application.

[0068] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0069] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, which may include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, etc.

[0070] The foregoing description is merely an exemplary embodiment of this disclosure and should not be construed as limiting the scope of this disclosure. Any equivalent changes and modifications made in accordance with the teachings of this disclosure shall still fall within the scope of this disclosure. Those skilled in the art will readily conceive of embodiments of this disclosure upon considering the specification and practicing the disclosure herein. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not described herein. The specification and embodiments are to be considered exemplary only, and the scope and spirit of this disclosure are defined by the claims.

[0071] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0072] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. An adaptive grasping method combining physical attribute reasoning and tactile feedback, characterized in that, include: The system acquires language grasping instructions and visual images of the target object, obtains candidate grasping poses of the target object based on the visual images, and uses a vision-language multimodal model to infer the coarse-grained physical properties of the target object. The coarse-grained physical properties are refined by regression using a serial neural network to obtain fine-grained physical properties and calculate the initial gripping force. Based on the initial grasping force and the candidate grasping pose, perform initial grasping and acquire tactile images, and calculate real-time tactile feedback force based on the generalized Hooke's law; Based on the stiffness category of the target object, the initial gripping force and the real-time tactile feedback force are fused according to a predetermined rule to obtain an adjusted gripping force; Continue the grasping action based on the adjusted grasping force until the language grasping command is completed; The step of fusing the initial gripping force and the real-time tactile feedback force according to a predetermined rule based on the stiffness category of the target object to obtain the adjusted gripping force includes: Based on the object stiffness category estimated in advance by the vision-language multimodal model, the target object is classified into a rigid object or a flexible object; When the object is determined to be rigid, the larger of the initial grasping force and the tactile feedback force is selected as the final grasping force. When the object is determined to be flexible, the smaller value between the initial grasping force and the tactile feedback force is selected as the final grasping force.

2. The adaptive grasping method combining physical attribute reasoning and tactile feedback as described in claim 1, characterized in that, The method further includes: Freeze the pre-trained visual-language multimodal model; The same training samples are received in parallel and output physical attribute predictions through a frozen visual-language multimodal model and a physical attribute reasoning subnetwork, respectively. The output of the frozen visual-language multimodal model is used as a soft label, and the physical attribute reasoning subnetwork is trained by knowledge distillation through regression loss; During the inference phase, only the physical attribute inference subnetwork is deployed to predict the physical attributes of the target object.

3. The adaptive grasping method combining physical attribute reasoning and tactile feedback as described in claim 1, characterized in that, The process of obtaining candidate grasping poses of the target object based on the visual image includes: The visual image is input into a pre-trained deep neural network to extract the image's visual features; By deconvolving and upsampling the visual features of the image using a grasping projector, the grasping point, grasping angle, and the width of the gripper opening at the end of the grasping robot are obtained for the target object in the scene. Non-maximum suppression and coordinate transformation are performed on the gripping point, gripping angle, and gripper opening width to obtain the candidate gripping pose of the target object.

4. The adaptive grasping method combining physical attribute reasoning and tactile feedback as described in claim 1, characterized in that, The calculation of real-time tactile feedback force based on the generalized Hooke's law includes: Before grasping, acquire multiple frames of tactile images in an uncontacted state, calculate and store the average depth map to eliminate sensor inherent noise; During the grasping process, the surface normals and gradient fields are predicted pixel by pixel from the real-time acquired tactile images using a fully connected neural network. The relative depth map is reconstructed from the gradient field by solving the Poisson equation, and then the relative depth map is multiplied by a pre-calibrated scaling factor and subtracted from the average depth map to obtain the absolute depth map.

5. The adaptive grasping method combining physical attribute reasoning and tactile feedback as described in claim 4, characterized in that, After obtaining the absolute depth map, the method further includes: The absolute depth map is converted into local compressive strain. Using the Young's modulus estimated in advance by the visual-language multimodal model and the local compressive strain, the normal force is calculated according to the generalized Hooke's law, and the normal force is used as the tactile feedback force.

6. The adaptive grasping method combining physical attribute reasoning and tactile feedback as described in claim 1, characterized in that, The step of classifying a target object into a rigid object or a flexible object based on the object stiffness category estimated in advance by the visual-language multimodal model includes: The visual-language multimodal model is invoked, using the visual image of the target object as input and outputting Young's modulus estimates. The estimated Young's modulus is converted into a calibrated Young's modulus based on the offline calibrated mapping function; If the Young's modulus is greater than or equal to the first fixed threshold, the target object is classified as a rigid object. If the Young's modulus is less than or equal to the second fixed threshold, the target object is classified as a flexible object. Wherein, the first fixed threshold is greater than the second fixed threshold.

7. An adaptive grasping system combining physical attribute reasoning and tactile feedback, executing the adaptive grasping method combining physical attribute reasoning and tactile feedback as described in any one of claims 1 to 6, characterized in that, include: The acquisition module is configured to acquire language capture commands and visual images of the target object; The grasping prediction module is configured to obtain candidate grasping poses of the target object based on visual input; The physical property reasoning module is configured to obtain initial physical property estimates of the target object through a pre-trained visual-linguistic multimodal model; The tactile feedback module is configured to acquire real-time tactile images of the target object and calculate tactile feedback force during the grasping process; The force type parsing module is configured to fuse the initial grasping force corresponding to the initial physical property estimate and the tactile feedback force based on the stiffness category of the target object to generate the final grasping force command. Specifically, the force type analysis module takes the larger value between the initial grasping force and the tactile feedback force for rigid objects and the smaller value for flexible objects, and performs closed-loop adaptive grasping.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored program, wherein the program, when executed, performs the method of any one of claims 1 to 6.

9. An electronic device comprising a memory and a processor, characterized in that, The memory stores a computer program, and the processor is configured to execute the method of any one of claims 1 to 6 through the computer program.

Images