Object grasping method, object grasping device, electronic device, and storage medium

Abstract

Embodiments of the present application provide an object grasping method, an object grasping device, an electronic device and a storage medium, and belong to the technical field of artificial intelligence. The method comprises: performing three-dimensional reconstruction according to a plurality of image frames to obtain an object three-dimensional model, performing material marking on the object three-dimensional model according to the visible component material types of all image frames to obtain a target material type, performing object recognition according to the image frames to obtain an unknown part material type and an unknown part proportion of a target object, respectively querying a material database according to the target material type and the unknown part material type to obtain first and second material physical properties, calculating a target grasping force according to the object three-dimensional model, the first and second material physical properties, the unknown part proportion and an object contact area of a mechanical arm, calculating a grasping position according to the object three-dimensional model and the first and second material physical properties, and grasping the target object according to the target grasping force and the grasping position, which can improve the accuracy of object grasping.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to an object grasping method, an object grasping device, an electronic device, and a storage medium. Background Technology

[0002] With the development of artificial intelligence and machine vision technologies, robotic arms can be used to perform object grasping tasks. In these technologies, the robotic arm grasps objects based on a fixed grasping force and position. However, this grasping process can be hampered by insufficient or excessive force, or poor grasping position, leading to inaccurate object grasping. Therefore, improving the accuracy of object grasping has become a pressing issue. Summary of the Invention

[0003] The main objective of this application is to provide an object grasping method, object grasping device, electronic device, and storage medium, which aim to improve the accuracy of object grasping.

[0004] To achieve the above objectives, a first aspect of this application proposes an object grasping method applied to a robotic arm, the method comprising:

[0005] Acquire images of the target object from multiple shooting angles to obtain multiple image frames;

[0006] For each image frame, a first object recognition is performed on the image frame to obtain the material type of the visible components of the target object;

[0007] A 3D model of the object is obtained by performing 3D reconstruction based on multiple image frames.

[0008] The object's 3D model is material-marked based on the material types of the visible components in all image frames to obtain the target material type;

[0009] Based on the image frame, a second object is identified to obtain the material type and proportion of the unknown part of the target object;

[0010] The first material physical property is obtained by querying the preset material database according to the target material type, and the second material physical property is obtained by querying the material database according to the unknown material type.

[0011] The target grasping force is calculated based on the object's 3D model, the physical properties of the first material, the physical properties of the second material, the proportion of the unknown part, and the object contact area of ​​the robotic arm.

[0012] Calculate the grab position based on the object's 3D model, the physical properties of the first material, and the physical properties of the second material;

[0013] Grasp the target object based on the target grasping force and the grasping position.

[0014] In some embodiments, calculating the target grasping force based on the object's three-dimensional model, the physical properties of the first material, the physical properties of the second material, the proportion of the unknown portion, and the object contact area of ​​the robotic arm includes:

[0015] The first gripping force is calculated based on the object's 3D model, the physical properties of the first material, the physical properties of the second material, and the proportion of the unknown portion.

[0016] The second gripping force is calculated based on the physical properties of the first material, the physical properties of the second material, and the object contact area of ​​the robotic arm.

[0017] The target gripping force is determined based on the first gripping force and the second gripping force.

[0018] In some embodiments, determining the target gripping force based on the first gripping force and the second gripping force includes:

[0019] By comparing the first gripping force and the second gripping force, a gripping force relationship is obtained; the gripping force relationship is used to indicate that the first gripping force is less than the second gripping force or that the first gripping force is greater than or equal to the second gripping force.

[0020] A gripping force limit is determined based on the gripping force relationship, the first gripping force, and the second gripping force; the gripping force limit includes a lower gripping force limit and an upper gripping force limit;

[0021] The target gripping force is determined based on the lower limit gripping force and the upper limit gripping force.

[0022] In some embodiments, determining the gripping force limit based on the gripping force relationship, the first gripping force, and the second gripping force includes:

[0023] If the gripping force relationship indicates that the first gripping force is greater than or equal to the second gripping force, then the second gripping force is taken as the lower limit gripping force, and the first gripping force is taken as the upper limit gripping force;

[0024] If the gripping force relationship indicates that the first gripping force is less than the second gripping force, then the difference between the first gripping force and the second gripping force is calculated to obtain a gripping difference value; the lower limit gripping force is calculated based on the first gripping force, the gripping difference value, and a preset weighting factor; the upper limit gripping force is calculated based on the first gripping force, the gripping difference value, and the preset weighting factor.

[0025] In some embodiments, calculating the gripping position based on the object's three-dimensional model, the physical properties of the first material, and the physical properties of the second material includes:

[0026] Obtain the first cross-sectional area of ​​the three-dimensional model of the object in the first direction, and calculate the first coordinate based on the first cross-sectional area, the first material physical properties, and the second material physical properties;

[0027] Obtain the second cross-sectional area of ​​the object's 3D model in the second direction, and calculate the second coordinates based on the second cross-sectional area, the physical properties of the first material, and the physical properties of the second material;

[0028] Obtain the third cross-sectional area of ​​the object's 3D model in a third direction, and calculate the third coordinate based on the third cross-sectional area, the physical properties of the first material, and the physical properties of the second material; any two of the first direction, the second direction, and the third direction are mutually perpendicular.

[0029] The grab position is determined based on the first coordinate, the second coordinate, and the third coordinate.

[0030] In some embodiments, performing first object recognition on the image frame to obtain the material type of the visible components of the target object includes:

[0031] Object detection is performed on the image frame to obtain the two-dimensional coordinates of the object;

[0032] The image frame is initially segmented based on the two-dimensional coordinates of the object to obtain an object instance segmentation image;

[0033] The object instance segmentation image is further segmented based on the preset cue point density to obtain the component instance segmentation image;

[0034] Material identification is performed based on the segmented images of the object instance and the segmented images of the component instance to obtain the material type of the visible component.

[0035] In some embodiments, the step of material labeling the 3D model of the object based on the material types of the visible parts in all image frames to obtain the target material type includes:

[0036] The component region is obtained by performing a two-dimensional projection on the three-dimensional model of the object;

[0037] For each image frame, the original material type of the object's 3D model is obtained based on the component region and the material type of the visible component;

[0038] The target material type is determined based on the original material type obtained from each image frame.

[0039] To achieve the above objectives, a second aspect of this application provides an object grasping device applied to a robotic arm, the device comprising:

[0040] The acquisition module is used to acquire images of the target object from multiple shooting angles, resulting in multiple image frames.

[0041] The first recognition module is used to perform a first object recognition on each image frame to obtain the material type of the visible parts of the target object;

[0042] The 3D reconstruction module is used to perform 3D reconstruction based on multiple image frames to obtain a 3D model of the object.

[0043] The marking module is used to mark the material of the three-dimensional model of the object according to the material type of the visible parts in all image frames, so as to obtain the target material type;

[0044] The second recognition module is used to perform second object recognition based on the image frame to obtain the material type and proportion of the unknown part of the target object.

[0045] The query module is used to query a preset material database according to the target material type to obtain a first material physical property, and to query the material database according to the unknown material type to obtain a second material physical property;

[0046] The first calculation module is used to calculate the target grasping force based on the three-dimensional model of the object, the physical properties of the first material, the physical properties of the second material, the proportion of the unknown part, and the object contact area of ​​the robotic arm.

[0047] The second calculation module is used to calculate the grab position based on the three-dimensional model of the object, the physical properties of the first material, and the physical properties of the second material.

[0048] The grasping module is used to grasp the target object based on the target grasping force and the grasping position.

[0049] To achieve the above objectives, a third aspect of this application provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the method described in the first aspect.

[0050] To achieve the above objectives, a fourth aspect of the present application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method described in the first aspect.

[0051] The object grasping method, object grasping device, electronic device, and computer-readable storage medium proposed in this application acquire images of a target object from multiple shooting angles, obtaining multiple image frames. Based on these multiple image frames, the grasping force and grasping position are accurately estimated, thereby accurately grasping the object. The grasping force and grasping position are related to the object's material. To obtain the optimal grasping force and optimal grasping position for the target object, the material type of the target object needs to be obtained. The target object includes visible parts and unknown parts. To obtain the material type of the visible parts, a first object recognition is performed on each image frame to obtain the material type of the visible parts of the target object. To enable the robotic arm to more accurately perceive and detect the target object in the working area, three-dimensional reconstruction is performed based on multiple image frames to obtain a three-dimensional model of the object, thus obtaining the complete three-dimensional shape information of the target object. The three-dimensional model of the object is material-labeled according to the material types of the visible parts in all image frames to obtain the target material type, thus obtaining the material type of the target object in three-dimensional space. The system performs second object recognition based on image frames to obtain the material type and proportion of the unknown parts of the target object. This allows for accurate 3D object modeling based on visible and unknown components. A pre-set material database is queried based on the target material type to obtain the first material physical property, and the same database is queried based on the unknown material type to obtain the second material physical property. The optimal gripping force and optimal gripping position are then calculated based on these two properties. The target gripping force is calculated using the object's 3D model, the first and second material physical properties, the proportion of the unknown part, and the robotic arm's contact area with the object to obtain the optimal gripping force for the robotic arm. The optimal gripping position is also calculated based on the object's 3D model, the first and second material physical properties, allowing the robotic arm to adaptively adjust the gripping force and position according to the different target objects. Grasping the target object based on the optimal gripping force and position improves the accuracy of object grasping. Attached Figure Description

[0052] Figure 1 This is a flowchart of the object grasping method provided in the embodiments of this application;

[0053] Figure 2 yes Figure 1 The flowchart of step S120 in the middle;

[0054] Figure 3 yes Figure 1 The flowchart of step S140 in the middle;

[0055] Figure 4 yes Figure 1The flowchart of step S170 in the process;

[0056] Figure 5 yes Figure 4 The flowchart of step S430 in the middle;

[0057] Figure 6 yes Figure 5 The flowchart of step S520 in the text;

[0058] Figure 7 yes Figure 1 The flowchart of step S180 in the process;

[0059] Figure 8 This is a schematic diagram of the object grasping device provided in the embodiments of this application;

[0060] Figure 9 This is a schematic diagram of the hardware structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0061] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0062] It should be noted that although functional modules are divided in the device schematic diagram and a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first," "second," etc., in the specification, claims, and the aforementioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0063] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0064] With the development of artificial intelligence and machine vision technologies, robotic arms can be used to perform object grasping tasks. In these technologies, the robotic arm grasps objects based on a fixed grasping force and position. However, this grasping process can be hampered by insufficient or excessive force, or poor grasping position, leading to inaccurate object grasping. Therefore, improving the accuracy of object grasping has become a pressing issue.

[0065] Based on this, embodiments of this application provide an object grasping method, an object grasping device, an electronic device, and a computer-readable storage medium, aiming to improve the accuracy of object grasping.

[0066] The object grasping method, object grasping device, electronic device, and computer-readable storage medium provided in this application are specifically described through the following embodiments. First, the object grasping method in the embodiments of this application is described.

[0067] The object grasping method provided in this application relates to the field of artificial intelligence technology. This object grasping method can be applied to a terminal, a server, or software running on either a terminal or a server. In some embodiments, the terminal can be a smartphone, tablet, laptop, desktop computer, etc.; the server can be configured as an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application implementing the object grasping method, but is not limited to the above forms.

[0068] This application can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0069] Figure 1 This is an optional flowchart of the object grasping method provided in the embodiments of this application. The object grasping method is applied to a robotic arm and may include, but is not limited to, steps S110 to S190.

[0070] Step S110: Acquire images of the target object from multiple shooting angles to obtain multiple image frames;

[0071] Step S120: For each image frame, perform first object recognition on the image frame to obtain the material type of the visible parts of the target object;

[0072] Step S130: Perform 3D reconstruction based on multiple image frames to obtain a 3D model of the object;

[0073] Step S140: Material labeling of the 3D model of the object is performed based on the material types of the visible parts of all image frames to obtain the target material type;

[0074] Step S150: Perform second object recognition based on the image frame to obtain the material type and proportion of the unknown part of the target object.

[0075] Step S160: Query the preset material database according to the target material type to obtain the first material physical property; query the material database according to the unknown material type to obtain the second material physical property.

[0076] Step S170: Calculate the target grasping force based on the object's 3D model, the physical properties of the first material, the physical properties of the second material, the proportion of the unknown part, and the object contact area of ​​the robotic arm.

[0077] Step S180: Calculate the grab position based on the object's 3D model, the physical properties of the first material, and the physical properties of the second material;

[0078] Step S190: Grasp the target object based on the target grasping force and grasping position.

[0079] In step S110 of some embodiments, the target object is the object to be grasped by the robotic arm in the current working scene. If the current working scene is an industrial manufacturing scene, the target object can be mechanical parts, electronic products, etc. If the current working scene is a logistics warehousing scene, the target object can be express parcels, goods, etc. The robotic arm is equipped with an RGB-D camera, which captures images of the target object from multiple shooting angles to obtain image frames matching the shooting angles. The image frames include RGB images and depth images. Under a single shooting angle, part of the target object may be occluded. By capturing images of the target object from different angles, a more comprehensive and complete view of the target object can be obtained, thereby improving the robotic arm's adaptability to complex and dynamic working environments.

[0080] Please see Figure 2 In some embodiments, step S120 may include, but is not limited to, steps S210 to S240:

[0081] Step S210: Perform object detection on the image frame to obtain the two-dimensional coordinates of the object;

[0082] Step S220: Perform preliminary instance segmentation on the image frame based on the two-dimensional coordinates of the object to obtain the object instance segmentation image;

[0083] Step S230: Perform further instance segmentation on the object instance segmentation image according to the preset cue point density to obtain the component instance segmentation image;

[0084] Step S240: Perform material identification based on the object instance segmentation image and the component instance segmentation image to obtain the material type of the visible component.

[0085] In step S210 of some embodiments, in order to locate the target object from the image frame, an object detection model can be used to detect the image frame and obtain a detection box containing the target object. If the detection box is rectangular, the coordinates of the upper left and lower right corners of the detection box are used as the two-dimensional coordinates of the object. If the detection box is not rectangular, the detection box is sampled to obtain multiple sampling points, and the coordinates of the multiple sampling points are used as the two-dimensional coordinates of the object. The object detection model can be a zero-shot object recognition model such as the GroundingDINO model, or a discriminative object recognition model such as the YOLO model.

[0086] In step S220 of some embodiments, instance segmentation provides pixel-level localization information. When multiple target objects of the same category exist in an image frame, in order to more accurately distinguish each object instance, the two-dimensional coordinates of the object are used as cue information and input together with the image frame into a zero-shot instance segmentation model for preliminary instance segmentation to extract the target object in the detection box and remove irrelevant background, thereby obtaining the object instance segmentation image of the target object. The zero-shot instance segmentation model can be a SAM (Segment Anything Model) model.

[0087] In step S230 of some embodiments, the target object includes multiple components, such as a hammer including a hammerhead, a handle, and a connecting part. To perform finer-grained component-level segmentation of the target object, a cue point density is obtained. The cue point density indicates the number of cue points distributed within a unit image region, and the cue points indicate the position of the component in the object instance segmentation image. Cue points are set in the object instance segmentation image according to the cue point density. The object instance segmentation image with cue points is then input into the zero-shot instance segmentation model in step S220 for further instance segmentation to obtain the component instance segmentation image.

[0088] In step S240 of some embodiments, the robotic arm may fail to grasp the object or damage it due to insufficient grasping force, excessive grasping force, or poor grasping position during object grasping. Related technologies employ methods such as soft robotic arms and optical tactile sensing to address the problem of inaccurate object grasping. Soft robotic arms utilize the deformability and compliance properties of soft materials, enabling the gripper to automatically adapt to the geometry and weight of various objects. However, this automatic adaptability requires careful selection of the design scheme of the soft material and gripper mechanical structure to meet the object grasping requirements of specific application scenarios, thus limiting the universality of the robotic arm in all application scenarios. Optical tactile sensing methods require installing a camera in each fingertip of the robotic arm gripper, located behind soft and transparent artificial skin, to convert optical observations marked on the skin into force estimates; the electronic skin detects the force applied by the gripper to the object through electrical signals. However, optical tactile sensing methods typically face durability challenges, resulting in a shorter lifespan for the robotic arm, and require a significant additional space to install the camera and skin.

[0089] This application embodiment, without introducing additional mechanical structures or complex sensors, utilizes the visual input of a camera to predict the material constituting the target object. This allows the robotic arm to accurately perceive the material of the target object, thereby accurately estimating the gripping force and position, and achieving accurate gripping of the target object. Even if the target object is composed of multiple different materials, the robotic arm can grip the target object without slipping, and will not damage fragile, easily deformable, or delicate target objects due to excessive force. This significantly improves the robotic arm's automated and intelligent handling and operation capabilities, and ensures the safety and integrity of the gripped object.

[0090] Specifically, component material estimation prompts are obtained, which indicate the material of the component to be output by the large model. The component instance segmentation image, the object instance segmentation image, and the component material estimation prompts are input into the visual language large model for material recognition to obtain the visible component material type. The visible component material type refers to the material type of each component on the surface of the target object, such as glass, ceramic, wood, plastic, etc. The visual language large model can be GPT-4V, GPT-4o, LLaVA, CogVLM, Qwen-VL, etc.

[0091] Through the above steps S210 to S240, the material of each component in the target object can be predicted, so that the robotic arm can adjust the gripping force and gripping position according to the material of each component, thereby accurately gripping the target object.

[0092] In step S130 of some embodiments, a multi-view geometry perspective difference cloud reconstruction algorithm, a super depth-of-field fusion algorithm, or a Gaussian splashing algorithm can be invoked to perform three-dimensional reconstruction based on all image frames of the target object, so as to map the target object from a two-dimensional image to a three-dimensional shape structure and obtain a three-dimensional model of the object.

[0093] It should be noted that the 3D object model obtained by the view difference point cloud reconstruction algorithm or the super depth-of-field fusion algorithm based on multi-view geometry is a 3D point cloud model, while the 3D object model obtained by the Gaussian splash algorithm is a Gaussian sphere. The 3D object model has 3D spatial coordinates. If the 3D object model is a 3D point cloud model, then the 3D spatial coordinates are the coordinates of the 3D point cloud that constitutes the model; if the 3D object model is a Gaussian sphere, then the 3D spatial coordinates are the coordinates of the center of the Gaussian sphere.

[0094] Please see Figure 3 In some embodiments, step S140 may include, but is not limited to, steps S310 to S330:

[0095] Step S310: Perform a two-dimensional projection on the three-dimensional model of the object to obtain the component region;

[0096] Step S320: For each image frame, obtain the original material type of the object's 3D model based on the part region and the material type of the visible parts;

[0097] Step S330: Determine the target material type based on the original material type obtained from each image frame.

[0098] In step S310 of some embodiments, the intrinsic calibration parameters and pose parameters of the camera are obtained, and the three-dimensional spatial coordinates of the object's three-dimensional model are mapped in two dimensions based on the intrinsic calibration parameters and pose parameters to obtain the component region. The object's three-dimensional model includes multiple three-dimensional point clouds or multiple Gaussian spheres, and the component region is a mapping of multiple three-dimensional point clouds or Gaussian spheres onto a two-dimensional plane, that is, the component region includes multiple mapping points, and the mapping points have two-dimensional spatial coordinates.

[0099] The formula for calculating a two-dimensional mapping is defined as follows:

[0100]

[0101] Among them, (X) w ,Y w Z w (x) represents the three-dimensional spatial coordinates. w Y is the x-axis. w Z is the ordinate. w (u,v) represents the vertical coordinate; (u,v) represents the two-dimensional spatial coordinates of the mapping point, where u is the horizontal coordinate and v is the vertical coordinate; f x f y cx and c y For the internal parameter calibration parameter, f x f is the focal length of the camera along the x-axis. y Let c be the focal length of the camera along the y-axis. x and c y `r` represents the principal point coordinates; `r` and `t` are pose parameters; `r` is the rotation matrix used to rotate points in the world coordinate system to the camera coordinate system; `r`... ij The element in the i-th row and j-th column of the rotation matrix; [t x ,t y ,t z ] is the translation vector, used to translate the rotated point from the camera coordinate system to the camera's position; t x t is the translation along the x-axis. y t is the translation along the y-axis. z This represents the translation along the z-axis.

[0102] In step S320 of some embodiments, referring to step S230, a component instance segmentation image of each image frame is obtained. The component instance segmentation image includes component segmentation regions of each component, and each component segmentation region has a visible component material type. If a mapping point of a component region falls into a component segmentation region of the component instance segmentation image, the mapping point is marked as the visible component material type of that component segmentation region, and the original material type of the mapping point is obtained. This process continues until all mapping points in the component regions are marked, and the original material type of the object's 3D model is obtained.

[0103] In step S330 of some embodiments, a majority vote is performed on the original material types obtained from each image frame to obtain the target material type. It should be noted that when multiple materials receive the same number of votes, one material can be randomly selected. For example, if a camera captures five image frames of a target object, five component instance segmentation images will be obtained. If a mapping point falls into one of the five component instance segmentation images and is labeled with the original material types iron, aluminum, aluminum, iron, and iron, where aluminum receives 2 votes and iron receives 3 votes, and since 3 is greater than 2, iron is chosen as the target material type for that mapping point.

[0104] Through steps S310 to S330, the material type of each component of the target object in three-dimensional space can be obtained, so that the robotic arm can accurately perceive the target object based on the material type of each component, thereby accurately grasping the target object.

[0105] In step S150 of some embodiments, in addition to the visible part of the object's surface, the target object also includes an invisible part (the contents of the target object), i.e., an unknown part. For example, whether a stainless steel cup contains water and the amount of water can affect the robotic arm's object-grabbing performance. To accurately predict the optimal gripping force and optimal gripping position of the target object, material estimation prompts for the invisible parts are obtained. These prompts, along with all image frames, are input into a large visual language model for second object recognition, yielding the material type and proportion of the unknown part of the target object. The material type of the unknown part refers to the material type of the unknown part, and the proportion of the unknown part refers to the percentage of the unknown part within the target object. Taking a stainless steel cup as an example, the material type of the unknown part indicates whether the cup contains contents, and the proportion of the unknown part can be the percentage of the contents, etc.

[0106] In step S160 of some embodiments, the material database stores material types and material physical properties matching those material types. These material physical properties are physical attribute parameters related to the material type, including material density, the coefficient of friction between the robotic arm gripper and the contact surface of that material type, maximum bending stress, and maximum bending curvature. A query is performed on the material database based on the target material type to obtain a first material physical property, which is the material physical property of the target material type. A query is also performed on the material database based on the unknown material types to obtain a second material physical property, which is the material physical property of the unknown material types.

[0107] Please see Figure 4 In some embodiments, step S170 may include, but is not limited to, steps S410 to S430:

[0108] Step S410: Calculate the first gripping force based on the object's 3D model, the physical properties of the first material, the physical properties of the second material, and the proportion of the unknown part;

[0109] Step S420: Calculate the second gripping force based on the physical properties of the first material, the physical properties of the second material, and the object contact area of ​​the robotic arm;

[0110] Step S430: Determine the target gripping force based on the first gripping force and the second gripping force.

[0111] In step S410 of some embodiments, the first gripping force is the minimum gripping force that the robotic arm's gripper can use to grasp the target object. The first material physical properties include the coefficient of friction between the robotic arm gripper and each component (target material type) in the object's 3D model, the material density of each component, and the volume of each component. The second material physical property includes the material density of the unknown portion. Specifically, the average coefficient of friction is calculated by averaging the coefficients of friction of each component. Based on the object's 3D model, the average coefficient of friction, the material density of each component, the volume of each component, the material density of the unknown portion, and the proportion of the unknown portion, the first gripping force is calculated. The formula for calculating the first gripping force is expressed as:

[0112]

[0113] Among them, F min ρ is the first gripping force; g is the gravitational constant, with a value of 9.8; μ is the average friction coefficient between the gripper and the target object; θ is the upward lifting angle of the gripper, generally 0°; O is the target object, including the object's 3D model and unknown parts; v is the volume of the component or the volume of the unknown part; ρ is the material density of the component or the material density of the unknown part.

[0114] It should be noted that the 3D model of the object comprises multiple 3D point clouds or multiple Gaussian spheres. The volume of a component can be expressed as the volume of the 3D point cloud or the volume of the Gaussian sphere. The volume of the 3D point cloud can be a pre-defined constant, and the volume of the Gaussian sphere can be represented by its standard deviation. The volume of the 3D model of the object can be determined based on the volumes of all 3D point clouds or all Gaussian spheres. Multiplying the proportion of the unknown part by the volume of the 3D model of the object yields the volume of the unknown part.

[0115] In step S420 of some embodiments, the object contact area of ​​the robotic arm is the surface area of ​​the gripper in contact with the target object, which is a constant. The second gripping force is the maximum gripping force that the gripper of the robotic arm can exert to grasp the target object without damaging it. The first material physical properties also include the maximum bending stress, maximum bending curvature, and Young's modulus corresponding to the material of each component. The second material physical properties also include the maximum bending stress, maximum bending curvature, and Young's modulus of the unknown portion. The target maximum bending stress is obtained by averaging the maximum bending stress of each component and the maximum bending stress of the unknown portion. The target maximum bending curvature is obtained by averaging the maximum bending curvature of each component and the maximum bending curvature of the unknown portion. The target Young's modulus is obtained by averaging the Young's modulus of each component and the Young's modulus of the unknown portion. The second gripping force is calculated based on the target maximum bending stress, target maximum bending curvature, target Young's modulus, object contact area of ​​the robotic arm, and object contact thickness of the robotic arm. The object contact thickness is the depth or thickness of the contact between the gripper and the target object, which is a constant. The formula for calculating the second gripping force is as follows:

[0116]

[0117] Among them, F max The second gripping force; A is the contact area of ​​the robotic arm with the object; σ y κ represents the target maximum bending stress; E represents the target Young's modulus; d represents the contact thickness of the object; max The target is the maximum bending curvature.

[0118] In step S430 of some embodiments, an optimal gripping force is determined based on a first gripping force and a second gripping force to obtain a target gripping force.

[0119] Steps S410 to S430 above can obtain the optimal gripping force, thereby enabling the robotic arm to grip the target object with appropriate force.

[0120] Please see Figure 5 In some embodiments, step S430 may include, but is not limited to, steps S510 to S530:

[0121] Step S510: Compare the first gripping force and the second gripping force to obtain the gripping force relationship; the gripping force relationship is used to indicate whether the first gripping force is less than the second gripping force or whether the first gripping force is greater than or equal to the second gripping force.

[0122] Step S520: Determine the gripping force limit based on the gripping force relationship, the first gripping force, and the second gripping force; the gripping force limit includes a lower gripping force limit and an upper gripping force limit;

[0123] Step S530: Determine the target gripping force based on the lower limit gripping force and the upper limit gripping force.

[0124] In step S510 of some embodiments, the first gripping force and the second gripping force are compared to obtain a gripping force relationship. The gripping force relationship is used to characterize the magnitude relationship between the first gripping force and the second gripping force, and can indicate that the first gripping force is less than the second gripping force, or that the first gripping force is greater than or equal to the second gripping force.

[0125] In step S520 of some embodiments, there may be parameter estimation errors in the first gripping force and the second gripping force. In order to obtain the optimal gripping force of the robotic arm on the target object, so that the target object can be gripped without slipping and without damage, the first gripping force and the second gripping force are corrected according to the gripping force relationship to obtain gripping force limits. The gripping force limits are the limit values ​​of the gripping force that the robotic arm gripper can apply to the target object. The gripping force limits include a lower limit gripping force and an upper limit gripping force. The lower limit gripping force is the minimum limit value of the gripping force, and the upper limit gripping force is the maximum limit value of the gripping force.

[0126] In step S530 of some embodiments, a set of gripping forces is constructed based on the lower limit gripping force, the upper limit gripping force, and the gripping forces between the lower limit gripping force and the upper limit gripping force. A gripping force is randomly selected from the set of gripping forces as the target gripping force. If the lower limit gripping force is represented as F1 and the upper limit gripping force is represented as F2, then the set of gripping forces is represented as [F1, F2].

[0127] Alternatively, the target gripping force can be obtained by calculating the median of the lower and upper gripping forces. The formula for median calculation is defined as follows:

[0128]

[0129] By using steps S510 to S530, the optimal gripping force can be obtained, avoiding the inability to grip the object or the object slipping due to insufficient gripping force, and the damage to the object due to excessive gripping force.

[0130] Please see Figure 6 In some embodiments, step S520 may include, but is not limited to, step S610 or step S620:

[0131] Step S610: If the gripping force relationship indicates that the first gripping force is greater than or equal to the second gripping force, then the second gripping force is taken as the lower limit gripping force and the first gripping force is taken as the upper limit gripping force.

[0132] Step S620: If the gripping force relationship indicates that the first gripping force is less than the second gripping force, then the difference between the first gripping force and the second gripping force is calculated to obtain the gripping difference value; the lower limit gripping force is calculated based on the first gripping force, the gripping difference value and the preset weight factor; the upper limit gripping force is calculated based on the first gripping force, the gripping difference value and the preset weight factor.

[0133] In step S610 of some embodiments, under normal circumstances, the first gripping force cannot be too small to ensure that the robotic arm does not slip when gripping the object, and the second gripping force cannot be too large to ensure that the object is not damaged when gripping it. Therefore, the first gripping force is generally less than the second gripping force. In abnormal situations, such as when the target object is too heavy but the surface of the object is very fragile, the first gripping force may be greater than or equal to the second gripping force. If the gripping force relationship indicates that the first gripping force is greater than or equal to the second gripping force, it indicates that an abnormal situation has occurred. In order for the robotic arm to accurately grip the target object, the second gripping force is used as the lower limit gripping force, and the first gripping force is used as the upper limit gripping force.

[0134] In step S620 of some embodiments, if the gripping force relationship indicates that the first gripping force is less than the second gripping force, in order to obtain the optimal gripping force, this embodiment further narrows the range of gripping force values. Specifically, the difference between the second gripping force and the first gripping force is calculated to obtain a gripping difference value. The value of the preset weighting factor is greater than 0 and less than 1. The gripping difference value is multiplied by the preset weighting factor to obtain the intermediate gripping force. The first gripping force and the intermediate gripping force are added together to obtain the lower limit gripping force. The second gripping force is subtracted from the intermediate gripping force to obtain the upper limit gripping force.

[0135] Through the above steps S610 to S620, the minimum gripping force and the maximum gripping force can be corrected according to the gripping force relationship, so as to improve the estimation accuracy of the optimal gripping force.

[0136] In some embodiments, the formula for calculating the target gripping force is defined as:

[0137]

[0138] Among them, F * F represents the force with which a target is grasped. min Indicates the first grasping force; F max Indicates the second gripping force; γ is the preset weighting factor; ΔF is the capture difference, expressed as ΔF = max[0, F max -F min ]; Indicates that f is a state of f min and f max A grasping force between them.

[0139] Please see Figure 7 In some embodiments, step S180 may include, but is not limited to, steps S710 to S740:

[0140] Step S710: Obtain the first cross-sectional area of ​​the object's 3D model in the first direction, and calculate the first coordinates based on the first cross-sectional area, the first material physical properties, and the second material physical properties;

[0141] Step S720: Obtain the second cross-sectional area of ​​the object's 3D model in the second direction, and calculate the second coordinates based on the second cross-sectional area, the physical properties of the first material, and the physical properties of the second material.

[0142] Step S730: Obtain the third cross-sectional area of ​​the object's 3D model in the third direction; calculate the third coordinate based on the third cross-sectional area, the physical properties of the first material, and the physical properties of the second material; any two of the first direction, the second direction, and the third direction are perpendicular to each other.

[0143] Step S740: Determine the grab position based on the first coordinate, the second coordinate, and the third coordinate.

[0144] In step S710 of some embodiments, the first direction is the x-axis direction of the three-dimensional model of the object, and the first cross-sectional area is the cross-sectional area at a certain position along the x-axis direction. The first coordinate is calculated based on the first cross-sectional area, the material density of each component, and the material density of the unknown part. The first coordinate is the abscissa of the gripping position. The method for calculating the first coordinate is as follows:

[0145]

[0146] Where l and l′ represent a position on one side of the x-axis direction of the object's 3D model and a position on the other side of the x-axis direction, respectively; ρ x (l) and ρ x (l′) represent the material density at positions l and l′ along the x-axis, respectively, including the material density of the component and the material density of the unknown part; S x (l) and S x (l′) represent the first cross-sectional area at positions l and l′, respectively; x is the integration limit.

[0147] Solving for x yields the first coordinate. Specifically, the mass of the 3D point cloud or Gaussian sphere is accumulated along both sides of the x-axis. If the accumulated mass on one side is greater than that on the other, accumulation begins from the other side first, until the accumulated mass on both sides is equal. x is then used as the first coordinate. It should be noted that the mass of the 3D point cloud or Gaussian sphere is obtained through the material density and the first cross-sectional area.

[0148] In step S720 of some embodiments, the second direction is the y-axis direction of the three-dimensional model of the object, the second cross-sectional area is the cross-sectional area at a certain position along the y-axis, and the second coordinate is calculated based on the second cross-sectional area, the material density of each component, and the material density of the unknown part. The second coordinate is the ordinate of the gripping position. The method for calculating the second coordinate is as follows:

[0149]

[0150] Where l and l′ represent a position on one side of the y-axis direction of the object's 3D model and a position on the other side of the y-axis direction, respectively; ρ y (l) and ρ y (l′) represent the material density at positions l and l′ along the y-axis, respectively, including the material density of the component and the material density of the unknown part; S y (l) and S y (l′) represents the second cross-sectional area at positions l and l′, respectively; y is the integration limit.

[0151] The mass of the 3D point cloud or Gaussian sphere is accumulated along both sides of the y-axis. If the accumulated mass on one side is greater than that on the other, the accumulation begins from the other side first, until the accumulated mass on both sides is equal. The integration limit y is then used as the second coordinate. It should be noted that the mass of the 3D point cloud or Gaussian sphere is obtained through the material density and the second cross-sectional area.

[0152] In step S730 of some embodiments, the third direction is the z-axis direction of the three-dimensional model of the object, the third cross-sectional area is the cross-sectional area at a certain position along the z-axis, and the third coordinate is calculated based on the third cross-sectional area, the material density of each component, and the material density of the unknown part. The third coordinate is the vertical coordinate of the gripping position. The method for calculating the third coordinate is as follows:

[0153]

[0154] Where l and l′ represent a position on one side of the z-axis direction of the object's 3D model and a position on the other side of the z-axis direction, respectively; ρ z (l) and ρ z (l′) represent the material density at positions l and l′ along the z-axis, respectively, including the material density of the component and the material density of the unknown part; S z (l) and S z (l′) represents the third cross-sectional area at positions l and l′, respectively; z is the integration limit.

[0155] The mass of the 3D point cloud or Gaussian sphere is accumulated along both sides of the z-axis. If the accumulated mass on one side is greater than that on the other side, the accumulation begins from the other side first, until the accumulated mass on both sides is equal. The integration limit z is then used as the third coordinate. It should be noted that the mass of the 3D point cloud or Gaussian sphere is obtained through the material density and the third cross-sectional area.

[0156] In step S740 of some embodiments, the first coordinate, the second coordinate, and the third coordinate are used as the grab position. If the first coordinate is x... * The second coordinate is y * The third coordinate is z * The capture position is represented as P. * =(x * ,y * ,z * ).

[0157] By using steps S710 to S740, the optimal gripping point can be obtained to avoid damage to or slipping of the target object.

[0158] In step S190 of some embodiments, the robotic arm applies a target grasping force to the grasping position to grasp the target object, so that the robotic arm can universally and safely grasp objects of any material, improving the accuracy of object grasping.

[0159] Please see Figure 8 This application also provides an object grasping device for use in a robotic arm, which can realize the above-described object grasping method. The object grasping device includes:

[0160] The acquisition module 810 is used to acquire images of the target object from multiple shooting angles, thereby obtaining multiple image frames;

[0161] The first recognition module 820 is used to perform first object recognition on each image frame to obtain the material type of the visible parts of the target object.

[0162] The 3D reconstruction module 830 is used to perform 3D reconstruction based on multiple image frames to obtain a 3D model of the object.

[0163] The marking module 840 is used to mark the material of the 3D model of the object according to the material type of the visible parts of all image frames, so as to obtain the target material type;

[0164] The second recognition module 850 is used to perform second object recognition based on the image frame to obtain the material type and proportion of the unknown part of the target object.

[0165] The query module 860 is used to query the preset material database according to the target material type to obtain the first material physical property, and to query the material database according to the unknown material type to obtain the second material physical property;

[0166] The first calculation module 870 is used to calculate the target grasping force based on the object's three-dimensional model, the physical properties of the first material, the physical properties of the second material, the proportion of the unknown part, and the object contact area of ​​the robotic arm.

[0167] The second calculation module 880 is used to calculate the grab position based on the object's three-dimensional model, the physical properties of the first material, and the physical properties of the second material.

[0168] The grasping module 890 is used to grasp target objects based on the grasping force and grasping position.

[0169] The specific implementation of this object grasping device is basically the same as the specific implementation of the object grasping method described above, and will not be repeated here.

[0170] This application also provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the above-described object grasping method. This electronic device can be any smart terminal, including tablet computers, in-vehicle computers, etc.

[0171] Please see Figure 9 , Figure 9The hardware structure of an electronic device according to another embodiment is illustrated. The electronic device includes:

[0172] The processor 910 can be implemented using a general-purpose central processing unit (CPU), microprocessor, application specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this application.

[0173] The memory 920 can be implemented as a read-only memory (ROM), static storage device, dynamic storage device, or random access memory (RAM). The memory 920 can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 920 and is called and executed by the processor 910 using the object grasping method of the embodiments of this application.

[0174] The input / output interface 930 is used to implement information input and output;

[0175] The communication interface 940 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0176] Bus 950 transmits information between various components of the device (e.g., processor 910, memory 920, input / output interface 930, and communication interface 940);

[0177] The processor 910, memory 920, input / output interface 930 and communication interface 940 are connected to each other within the device via bus 950.

[0178] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described object grasping method.

[0179] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0180] The embodiments described in this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided by the embodiments of this application. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.

[0181] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of this application, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0182] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0183] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0184] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0185] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0186] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0187] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0188] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0189] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0190] The preferred embodiments of the present application have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present application. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and substance of the embodiments of the present application shall be within the scope of the claims of the present application.

Claims

1. A method for grasping an object, characterized in that, Applied to robotic arms, the method includes: Acquire images of the target object from multiple shooting angles to obtain multiple image frames; For each image frame, a first object recognition is performed on the image frame to obtain the material type of the visible components of the target object; A 3D model of the object is obtained by performing 3D reconstruction based on multiple image frames. The object's 3D model is material-marked based on the material types of the visible components in all image frames to obtain the target material type; Obtain material estimation prompts for invisible components, input the material estimation prompts for invisible components and all the image frames into a visual language big model for second object recognition, and obtain the material type and proportion of the unknown part of the target object; The first material physical property is obtained by querying the preset material database according to the target material type, and the second material physical property is obtained by querying the material database according to the unknown material type. The target grasping force is calculated based on the object's 3D model, the physical properties of the first material, the physical properties of the second material, the proportion of the unknown part, and the object contact area of ​​the robotic arm. Calculate the grab position based on the object's 3D model, the physical properties of the first material, and the physical properties of the second material; Grasp the target object based on the target grasping force and the grasping position.

2. The method according to claim 1, characterized in that, The step of calculating the target grasping force based on the object's 3D model, the physical properties of the first material, the physical properties of the second material, the proportion of the unknown portion, and the object contact area of ​​the robotic arm includes: The first gripping force is calculated based on the object's 3D model, the physical properties of the first material, the physical properties of the second material, and the proportion of the unknown portion. The second gripping force is calculated based on the physical properties of the first material, the physical properties of the second material, and the object contact area of ​​the robotic arm. The target gripping force is determined based on the first gripping force and the second gripping force.

3. The method according to claim 2, characterized in that, Determining the target gripping force based on the first gripping force and the second gripping force includes: By comparing the first gripping force and the second gripping force, a gripping force relationship is obtained; the gripping force relationship is used to indicate that the first gripping force is less than the second gripping force or that the first gripping force is greater than or equal to the second gripping force. A gripping force limit is determined based on the gripping force relationship, the first gripping force, and the second gripping force; the gripping force limit includes a lower gripping force limit and an upper gripping force limit; The target gripping force is determined based on the lower limit gripping force and the upper limit gripping force.

4. The method according to claim 3, wherein determining the gripping force limit based on the gripping force relationship, the first gripping force, and the second gripping force comprises: If the gripping force relationship indicates that the first gripping force is greater than or equal to the second gripping force, then the second gripping force is taken as the lower limit gripping force, and the first gripping force is taken as the upper limit gripping force; If the gripping force relationship indicates that the first gripping force is less than the second gripping force, then the difference between the first gripping force and the second gripping force is calculated to obtain a gripping difference value; the lower limit gripping force is calculated based on the first gripping force, the gripping difference value, and a preset weighting factor; the upper limit gripping force is calculated based on the first gripping force, the gripping difference value, and the preset weighting factor.

5. The method according to claim 1, characterized in that, The step of calculating the grab position based on the object's 3D model, the physical properties of the first material, and the physical properties of the second material includes: Obtain the first cross-sectional area of ​​the three-dimensional model of the object in the first direction, and calculate the first coordinate based on the first cross-sectional area, the first material physical properties, and the second material physical properties; Obtain the second cross-sectional area of ​​the object's 3D model in the second direction, and calculate the second coordinates based on the second cross-sectional area, the physical properties of the first material, and the physical properties of the second material; Obtain the third cross-sectional area of ​​the object's 3D model in a third direction, and calculate the third coordinate based on the third cross-sectional area, the physical properties of the first material, and the physical properties of the second material; any two of the first direction, the second direction, and the third direction are mutually perpendicular. The grab position is determined based on the first coordinate, the second coordinate, and the third coordinate.

6. The method according to any one of claims 1 to 5, characterized in that, The first object recognition process, which involves identifying the visible components of the target object from the image frame, includes: Object detection is performed on the image frame to obtain the two-dimensional coordinates of the object; The image frame is initially segmented based on the two-dimensional coordinates of the object to obtain an object instance segmentation image; The object instance segmentation image is further segmented based on the preset cue point density to obtain the component instance segmentation image; Material identification is performed based on the segmented images of the object instance and the segmented images of the component instance to obtain the material type of the visible component.

7. The method according to any one of claims 1 to 5, characterized in that, The step of material labeling the 3D model of the object based on the material types of the visible components in all image frames to obtain the target material type includes: The component region is obtained by performing a two-dimensional projection on the three-dimensional model of the object; For each image frame, the original material type of the object's 3D model is obtained based on the component region and the material type of the visible component; The target material type is determined based on the original material type obtained from each image frame.

8. An object grasping device, characterized in that, Applied to robotic arms, the device includes: The acquisition module is used to acquire images of the target object from multiple shooting angles, resulting in multiple image frames. The first recognition module is used to perform a first object recognition on each image frame to obtain the material type of the visible parts of the target object; The 3D reconstruction module is used to perform 3D reconstruction based on multiple image frames to obtain a 3D model of the object. The marking module is used to mark the material of the three-dimensional model of the object according to the material type of the visible parts in all image frames, so as to obtain the target material type; The second recognition module is used to obtain material estimation prompts for invisible parts, input the material estimation prompts for invisible parts and all the image frames into the visual language big model for second object recognition, and obtain the material type and proportion of the unknown part of the target object. The query module is used to query a preset material database according to the target material type to obtain a first material physical property, and to query the material database according to the unknown material type to obtain a second material physical property; The first calculation module is used to calculate the target grasping force based on the three-dimensional model of the object, the physical properties of the first material, the physical properties of the second material, the proportion of the unknown part, and the object contact area of ​​the robotic arm. The second calculation module is used to calculate the grab position based on the three-dimensional model of the object, the physical properties of the first material, and the physical properties of the second material. The grasping module is used to grasp the target object based on the target grasping force and the grasping position.

9. An electronic device, characterized in that, The electronic device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the method according to any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the method described in any one of claims 1 to 7.

Images