Object grabbing method, object grabbing device, electronic equipment and storage medium

Abstract

The embodiment of the invention provides an object grabbing method, an object grabbing device, electronic equipment and a storage medium, and belongs to the technical field of artificial intelligence. The method comprises the steps of performing three-dimensional reconstruction according to a plurality of image frames to obtain an object three-dimensional model, performing material marking on the object three-dimensional model according to visible part material types of all the image frames to obtain a target material type, performing object identification according to the image frames to obtain an unknown part material type and an unknown part proportion of a target object, and performing object identification according to the unknown part material type and the unknown part proportion. The material database is inquired according to the target material type and the unknown part material type, first and second material physical attributes are obtained, the target grabbing force is calculated according to the object three-dimensional model, the first and second material physical attributes, the unknown part proportion and the object contact area of the mechanical arm, and the target grabbing force is obtained. According to the object three-dimensional model, the first material physical attribute and the second material physical attribute, the grabbing position is calculated, the target object is grabbed according to the target grabbing force and the grabbing position, and the object grabbing accuracy can be improved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to an object grasping method, an object grasping device, an electronic device and a storage medium. Background Art

[0002] With the development of artificial intelligence and machine vision technology, robotic arms can be used to perform object grasping tasks. In related technologies, robotic arms grasp objects based on fixed grasping force and grasping position, which makes it impossible to accurately grasp objects due to insufficient force, excessive force or poor grasping position during the grasping process. Therefore, how to improve the accuracy of object grasping has become an urgent problem to be solved. Summary of the invention

[0003] The main purpose of the embodiments of the present application is to provide an object grasping method, an object grasping device, an electronic device and a storage medium, aiming to improve the accuracy of object grasping.

[0004] To achieve the above-mentioned purpose, a first aspect of an embodiment of the present application proposes an object grasping method, which is applied to a robotic arm, and the method comprises:

[0005] Acquire images of the target object at multiple shooting angles to obtain multiple image frames;

[0006] For each of the image frames, performing first object recognition on the image frame to obtain a material type of a visible component of the target object;

[0007] Performing three-dimensional reconstruction based on the plurality of image frames to obtain a three-dimensional model of the object;

[0008] Marking the material of the three-dimensional model of the object according to the material type of the visible components of all image frames to obtain a target material type;

[0009] Performing a second object recognition according to the image frame to obtain the material type of the unknown part and the proportion of the unknown part of the target object;

[0010] A preset material database is queried according to the target material type to obtain a first material physical property, and a material database is queried according to the unknown part material type to obtain a second material physical property;

[0011] Calculating a target grasping force according to the three-dimensional model of the object, the physical properties of the first material, the physical properties of the second material, the proportion of the unknown part, and the object contact area of ​​the robotic arm;

[0012] Calculating a grasping position according to the three-dimensional model of the object, the physical properties of the first material, and the physical properties of the second material;

[0013] The target object is grasped according to the target grasping force and the grasping position.

[0014] In some embodiments, calculating the target grasping force according to the three-dimensional model of the object, the physical properties of the first material, the physical properties of the second material, the proportion of the unknown part, and the object contact area of ​​the robotic arm includes:

[0015] Calculating a first grasping force according to the three-dimensional model of the object, the physical properties of the first material, the physical properties of the second material, and the proportion of the unknown part;

[0016] Calculating a second grasping force according to the physical properties of the first material, the physical properties of the second material, and an object contact area of ​​the robotic arm;

[0017] The target grasping force is determined according to the first grasping force and the second grasping force.

[0018] In some embodiments, determining the target grasping force according to the first grasping force and the second grasping force includes:

[0019] Comparing the first grasping force and the second grasping force to obtain a grasping force relationship; the grasping force relationship is used to indicate that the first grasping force is less than the second grasping force or the first grasping force is greater than or equal to the second grasping force;

[0020] determining a gripping force limit value according to the gripping force relationship, the first gripping force and the second gripping force; the gripping force limit value includes a lower gripping force limit and an upper gripping force limit;

[0021] The target grasping force is determined according to the lower grasping force limit and the upper grasping force limit.

[0022] In some embodiments, determining the gripping force limit value according to the gripping force relationship, the first gripping force, and the second gripping force includes:

[0023] If the gripping force relationship indicates that the first gripping force is greater than or equal to the second gripping force, the second gripping force is used as the lower gripping force limit, and the first gripping force is used as the upper gripping force limit;

[0024] If the grasping force relationship indicates that the first grasping force is smaller than the second grasping force, the difference between the first grasping force and the second grasping force is calculated to obtain a grasping difference; the lower limit grasping force is calculated based on the first grasping force, the grasping difference and a preset weight factor; the upper limit grasping force is calculated based on the first grasping force, the grasping difference and the preset weight factor.

[0025] In some embodiments, calculating the grasping position according to the object three-dimensional model, the first material physical property and the second material physical property includes:

[0026] Acquire a first cross-sectional area of ​​the three-dimensional model of the object in a first direction, and calculate a first coordinate according to the first cross-sectional area, the first material physical property, and the second material physical property;

[0027] Acquire a second cross-sectional area of ​​the three-dimensional model of the object in a second direction, and calculate a second coordinate according to the second cross-sectional area, the first material physical property, and the second material physical property;

[0028] Obtaining a third cross-sectional area of ​​the three-dimensional model of the object in a third direction, and calculating a third coordinate according to the third cross-sectional area, the first material physical property, and the second material physical property; any two of the first direction, the second direction, and the third direction are perpendicular to each other;

[0029] The grasping position is determined according to the first coordinate, the second coordinate and the third coordinate.

[0030] In some embodiments, performing first object recognition on the image frame to obtain the material type of the visible component of the target object includes:

[0031] Performing object detection on the image frame to obtain two-dimensional coordinates of the object;

[0032] Performing preliminary instance segmentation on the image frame according to the two-dimensional coordinates of the object to obtain an object instance segmentation image;

[0033] Performing instance segmentation again on the object instance segmentation image according to a preset hint point density to obtain a component instance segmentation image;

[0034] Material recognition is performed based on the object instance segmentation image and the component instance segmentation image to obtain the material type of the visible component.

[0035] In some embodiments, the step of marking the material of the three-dimensional model of the object according to the material type of the visible components of all image frames to obtain the target material type includes:

[0036] Performing two-dimensional projection on the three-dimensional model of the object to obtain a component area;

[0037] For each image frame, obtaining the original material type of the three-dimensional model of the object according to the component area and the material type of the visible component;

[0038] The target material type is determined according to the original material type obtained from each image frame.

[0039] To achieve the above-mentioned purpose, a second aspect of an embodiment of the present application provides an object grasping device, which is applied to a robotic arm, and the device comprises:

[0040] An acquisition module is used to acquire images of a target object at multiple shooting angles to obtain multiple image frames;

[0041] A first recognition module, configured to perform a first object recognition on each of the image frames to obtain a material type of a visible component of the target object;

[0042] A three-dimensional reconstruction module, used for performing three-dimensional reconstruction according to the plurality of image frames to obtain a three-dimensional model of the object;

[0043] A marking module, used for marking the material of the three-dimensional model of the object according to the material type of the visible components of all image frames to obtain a target material type;

[0044] A second recognition module is used to perform a second object recognition according to the image frame to obtain the material type of the unknown part and the proportion of the unknown part of the target object;

[0045] A query module, configured to query a preset material database according to the target material type to obtain a first material physical property, and query the material database according to the unknown part material type to obtain a second material physical property;

[0046] A first calculation module, configured to calculate a target grasping force according to the three-dimensional model of the object, the physical properties of the first material, the physical properties of the second material, the proportion of the unknown part, and the object contact area of ​​the robotic arm;

[0047] A second calculation module, used for calculating a grasping position according to the three-dimensional model of the object, the physical properties of the first material and the physical properties of the second material;

[0048] A grasping module is used to grasp the target object according to the target grasping force and the grasping position.

[0049] To achieve the above objectives, a third aspect of an embodiment of the present application proposes an electronic device, which includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the method described in the first aspect is implemented.

[0050] To achieve the above-mentioned purpose, the fourth aspect of an embodiment of the present application proposes a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method described in the first aspect is implemented.

[0051] The object grasping method, object grasping device, electronic device and computer-readable storage medium proposed in the embodiments of the present application obtain multiple image frames by acquiring images of the target object at multiple shooting angles, so as to accurately estimate the grasping force and grasping position based on the multiple image frames, thereby accurately grasping the object. The grasping force and grasping position are related to the material of the object. In order to obtain the best grasping force and the best grasping position for grasping the target object, it is necessary to obtain the material type of the target object. The target object includes visible parts and unknown parts. In order to obtain the material type of the visible parts, for each image frame, the image frame is firstly identified to obtain the material type of the visible parts of the target object. In order to enable the robot arm to more accurately perceive and detect the target object in the working area, three-dimensional reconstruction is performed based on multiple image frames to obtain a three-dimensional model of the object to obtain the complete three-dimensional shape information of the target object. The three-dimensional model of the object is material-marked according to the material type of the visible parts of all image frames to obtain the target material type, so as to obtain the material type of the target object in the three-dimensional space dimension. The second object is identified according to the image frame, and the material type of the unknown part of the target object and the proportion of the unknown part are obtained, so as to obtain the material type of the unknown part and the proportion of the unknown part in the target object, thereby accurately modeling the three-dimensional object based on the visible parts and the unknown parts. According to the target material type, the preset material database is queried to obtain the first material physical property, and according to the unknown part material type, the material database is queried to obtain the second material physical property, so as to calculate the best grasping force and the best grasping position based on the first material physical property and the second material physical property. According to the three-dimensional model of the object, the first material physical property, the second material physical property, the proportion of the unknown part and the object contact area of ​​the manipulator, the target grasping force is calculated to obtain the best grasping force of the manipulator for the target object. According to the three-dimensional model of the object, the first material physical property and the second material physical property, the grasping position is calculated to obtain the best grasping position of the manipulator for the target object, so that the manipulator can adjust the grasping force and grasping position according to the different target objects. The target object is grasped according to the grasping force and grasping position, and the target object is grasped by the best grasping force and the best grasping position, thereby improving the accuracy of object grasping. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 is a flow chart of the object grasping method provided in an embodiment of the present application;

[0053] Figure 2 yes Figure 1 Flow chart of step S120 in FIG.

[0054] Figure 3 yes Figure 1 Flow chart of step S140 in FIG.

[0055] Figure 4 yes Figure 1Flowchart of step S170 in FIG.

[0056] Figure 5 yes Figure 4 Flow chart of step S430 in FIG.

[0057] Figure 6 yes Figure 5 Flow chart of step S520 in FIG.

[0058] Figure 7 yes Figure 1 Flow chart of step S180 in FIG.

[0059] Figure 8 is a schematic diagram of the structure of an object grasping device provided in an embodiment of the present application;

[0060] Fig. 9 It is a schematic diagram of the hardware structure of the electronic device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0061] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0062] It should be noted that, although the functional modules are divided in the device schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first", "second", etc. in the specification, claims and the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0063] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0064] With the development of artificial intelligence and machine vision technology, robotic arms can be used to perform object grasping tasks. In related technologies, robotic arms grasp objects based on fixed grasping force and grasping position, which makes it impossible to accurately grasp objects due to insufficient force, excessive force or poor grasping position during the grasping process. Therefore, how to improve the accuracy of object grasping has become an urgent problem to be solved.

[0065] Based on this, the embodiments of the present application provide an object grasping method, an object grasping device, an electronic device and a computer-readable storage medium, aiming to improve the accuracy of object grasping.

[0066] The object grasping method, object grasping device, electronic device and computer-readable storage medium provided in the embodiments of the present application are specifically illustrated through the following embodiments. First, the object grasping method in the embodiments of the present application is described.

[0067] The object grasping method provided in the embodiment of the present application relates to the field of artificial intelligence technology. The object grasping method provided in the embodiment of the present application can be applied to a terminal, can be applied to a server side, or can be software running in a terminal or a server side. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, etc.; the server side can be configured as an independent physical server, or can be configured as a server cluster or a distributed system composed of multiple physical servers, and can also be configured as a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application that implements the object grasping method, etc., but is not limited to the above forms.

[0068] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0069] Figure 1 It is an optional flow chart of the object grasping method provided in an embodiment of the present application. The object grasping method is applied to a robotic arm and may include but is not limited to steps S110 to S190.

[0070] Step S110, acquiring images of the target object at multiple shooting angles to obtain multiple image frames;

[0071] Step S120, for each image frame, performing first object recognition on the image frame to obtain the material type of the visible component of the target object;

[0072] Step S130, performing three-dimensional reconstruction according to the multiple image frames to obtain a three-dimensional model of the object;

[0073] Step S140, marking the material of the object three-dimensional model according to the material types of visible components of all image frames to obtain a target material type;

[0074] Step S150, performing second object recognition according to the image frame to obtain the material type of the unknown part and the proportion of the unknown part of the target object;

[0075] Step S160, querying a preset material database according to the target material type to obtain a first material physical property, and querying a material database according to the unknown part material type to obtain a second material physical property;

[0076] Step S170, calculating the target grasping force according to the three-dimensional model of the object, the physical properties of the first material, the physical properties of the second material, the proportion of the unknown part, and the object contact area of ​​the robot arm;

[0077] Step S180, calculating a grasping position according to the three-dimensional model of the object, the physical properties of the first material, and the physical properties of the second material;

[0078] Step S190, grasping the target object according to the target grasping force and grasping position.

[0079] In step S110 of some embodiments, the target object is an object to be grasped in the current working scene of the robot arm. If the current working scene is an industrial manufacturing scene, the target object may be a mechanical part, an electronic product, etc. If the current working scene is a logistics warehousing scene, the target object may be an express package, a commodity, etc. The robot arm is equipped with an RGB-D camera, which shoots the target object from multiple shooting angles to obtain an image frame that matches the shooting angle, and the image frame includes an RGB image and a depth image. Under a single shooting angle, part of the target object may be obscured. By shooting the target object from different angles, a more comprehensive and complete view of the target object can be obtained, thereby improving the robot arm's ability to adapt to complex and dynamic working environments.

[0080] See also Figure 2 In some embodiments, step S120 may include but is not limited to steps S210 to S240:

[0081] Step S210, performing object detection on the image frame to obtain the two-dimensional coordinates of the object;

[0082] Step S220, performing preliminary instance segmentation on the image frame according to the two-dimensional coordinates of the object to obtain an object instance segmentation image;

[0083] Step S230, performing instance segmentation again on the object instance segmentation image according to a preset hint point density to obtain a component instance segmentation image;

[0084] Step S240 , performing material recognition based on the object instance segmentation image and the component instance segmentation image to obtain the material type of the visible component.

[0085] In step S210 of some embodiments, in order to locate the target object from the image frame, the image frame may be subjected to object detection by a target detection model to obtain a detection frame, which contains the target object. If the detection frame is a rectangle, the coordinates of the upper left corner and the lower right corner of the detection frame are used as the two-dimensional coordinates of the object. If the detection frame is non-rectangular, the detection frame is sampled to obtain multiple sampling points, and the coordinates of the multiple sampling points are used as the two-dimensional coordinates of the object. The target detection model can be a zero-sample object recognition model such as the GroundingDINO model, or a discriminative object recognition model such as the YOLO model.

[0086] In step S220 of some embodiments, instance segmentation provides pixel-level positioning information. When there are multiple target objects of the same category in the image frame, in order to more accurately distinguish each object instance, the object two-dimensional coordinates are used as prompt information and input into the zero-sample instance segmentation model together with the image frame for preliminary instance segmentation to extract the target object in the detection frame and remove irrelevant background to obtain an object instance segmentation image of the target object. The zero-sample instance segmentation model can be a SAM (Segment Anything Model) model.

[0087] In step S230 of some embodiments, the target object includes multiple parts, such as a hammer including a hammer head, a hammer handle, and a connecting part. In order to perform more fine-grained component-level segmentation on the target object, the cue point density is obtained. The cue point density is used to indicate the number of cue points distributed in a unit image area. The cue points are used to indicate the position of the parts in the object instance segmentation image. Cue points are set in the object instance segmentation image according to the cue point density, and the object instance segmentation image with cue points is input into the zero-sample instance segmentation model of step S220 for re-instance segmentation to obtain a component instance segmentation image.

[0088] In step S240 of some embodiments, the robot arm cannot grasp the object or the object is damaged due to insufficient grasping force, excessive grasping force or poor grasping position during the object grasping process. In the related art, soft robot arms, optical tactile sensing and other methods are used to solve the problem of inaccurate object grasping. The soft robot arm uses the deformation and compliance characteristics of soft materials to enable the gripper of the robot arm to automatically adapt to the geometry and weight of various objects. However, this automatic adaptability of the robot arm requires careful selection of the design scheme of the soft material and the mechanical structure of the gripper to meet the object grasping requirements in specific application scenarios, thereby limiting the universality of the robot arm in all application scenarios. The optical tactile sensing method requires a camera to be installed in each fingertip of the robot arm gripper, which is located behind the soft and transparent artificial skin to convert the optical observation of the skin mark into a force estimate; while the electronic skin detects the force applied by the gripper to the object through electrical signals. However, the optical tactile sensing method usually faces challenges related to durability, resulting in a short service life of the robot arm and requiring additional large space to install the camera and skin.

[0089] The embodiment of the present application uses the visual input of the camera to predict the material constituting the target object without introducing additional mechanical structures or complex sensors, so that the robot arm can accurately sense the material of the target object, and then accurately estimate the grasping force and grasping position to achieve accurate grasping of the target object. Even if the target object is composed of multiple different materials, the robot arm can grasp the target object without slipping, and will not cause fragile, deformable, and fragile target objects to be damaged due to excessive force. It can greatly improve the robot arm's automated and intelligent processing and operation capabilities of objects, and ensure the safety and integrity of the grasped objects.

[0090] Specifically, obtain the component material estimation prompt word, which is used to indicate the material of the component output by the large model. Input the component instance segmentation image, the object instance segmentation image and the component material estimation prompt word into the visual language large model for material recognition to obtain the visible component material type. The visible component material type is the material type of each component on the surface of the target object, such as glass, ceramic, wood, plastic, etc. The visual language large model can be GPT-4V, GPT-4o, LLaVA, CogVLM, Qwen-VL, etc.

[0091] Through the above steps S210 to S240, the material of each component in the target object can be predicted, so that the robot arm can adjust the grasping force and grasping position according to the material of each component, thereby accurately grasping the target object.

[0092] In step S130 of some embodiments, the perspective difference point cloud reconstruction algorithm, super depth of field fusion algorithm or Gaussian splash algorithm of multi-perspective geometry can be called to perform three-dimensional reconstruction based on all image frames of the target object to map the target object from a two-dimensional image to a three-dimensional shape structure to obtain a three-dimensional model of the object.

[0093] It should be noted that the 3D model of an object obtained by the perspective difference point cloud reconstruction algorithm based on multi-perspective geometry or the super depth of field fusion algorithm is a 3D point cloud model, and the 3D model of an object obtained by the Gaussian splash algorithm is a Gaussian sphere. The 3D model of an object has 3D space coordinates. If the 3D model of an object is a 3D point cloud model, the 3D space coordinates are the coordinates of the 3D point cloud constituting the model; if the 3D model of an object is a Gaussian sphere, the 3D space coordinates are the coordinates of the center of the Gaussian sphere.

[0094] See also Figure 3 In some embodiments, step S140 may include but is not limited to steps S310 to S330:

[0095] Step S310, performing two-dimensional projection on the three-dimensional model of the object to obtain a component area;

[0096] Step S320, for each image frame, obtaining the original material type of the object's three-dimensional model according to the component area and the visible component material type;

[0097] Step S330, determining a target material type according to the original material type obtained from each image frame.

[0098] In step S310 of some embodiments, the intrinsic calibration parameters and the pose parameters of the camera are obtained, and the three-dimensional space coordinates of the three-dimensional model of the object are two-dimensionally mapped according to the intrinsic calibration parameters and the pose parameters to obtain the component area. The three-dimensional model of the object includes multiple three-dimensional point clouds or multiple Gaussian spheres, and the component area is the mapping of the multiple three-dimensional point clouds or Gaussian spheres on the two-dimensional plane, that is, the component area includes multiple mapping points, and the mapping points have two-dimensional space coordinates.

[0099] The calculation formula of the two-dimensional mapping is defined as:

[0100]

[0101] Among them, (X w ,Y w ,Z w ) is the three-dimensional space coordinate, X w is the horizontal axis, Y w is the vertical coordinate, Z w is the vertical coordinate; (u, v) is the two-dimensional space coordinate of the mapping point, u is the horizontal coordinate, and v is the vertical coordinate; f x 、f y 、cx and c y is the internal calibration parameter, f x is the focal length of the camera along the x-axis, f y is the focal length of the camera along the y axis, c x and c y are the principal point coordinates; r and t are the pose parameters; r is the rotation matrix, which is used to rotate the point in the world coordinate system to the camera coordinate system; r ij is the element in the i-th row and j-th column of the rotation matrix; [t x ,t y ,t z ] is the translation vector, which is used to translate the rotated point from the camera coordinate system to the camera position; t x is the translation of the x-axis, t y is the translation of the y axis, t z is the translation amount on the z-axis.

[0102] In step S320 of some embodiments, referring to step S230, a component instance segmentation image of each image frame is obtained, the component instance segmentation image includes component segmentation regions of each component, each component segmentation region has a visible component material type of the component. If a mapping point of a component region falls into a component segmentation region of the component instance segmentation image, the mapping point is marked as the visible component material type of the component segmentation region, and the original material type of the mapping point is obtained, until all mapping points in the component region are marked, and the original material type of the three-dimensional model of the object is obtained.

[0103] In step S330 of some embodiments, a majority vote is performed on the original material types obtained from each image frame to obtain a target material type. It should be noted that when multiple materials obtain the same number of highest votes, a material can be randomly selected. For example, when 5 image frames of the target object are taken by a camera, 5 component instance segmentation images will be obtained. After a mapping point falls into the 5 component instance segmentation images, the marked original material types are iron, aluminum, aluminum, iron and iron, respectively, where the number of votes for aluminum is 2 and the number of votes for iron is 3. Since 3 is greater than 2, iron is used as the target material type of the mapping point.

[0104] Through the above steps S310 to S330, the material type of each component of the target object in the three-dimensional space can be obtained, so that the robot arm can accurately perceive the target object based on the material type of each component, thereby accurately grasping the target object.

[0105] In step S150 of some embodiments, in addition to the visible part of the object surface, the target object also includes an invisible part inside (the content of the target object), i.e., an unknown part. The invisible part, such as whether a stainless steel cup is filled with water and the amount of water, will also affect the object grasping performance of the robot arm. In order to accurately predict the optimal grasping force and the optimal grasping position of the target object, the invisible part material estimation prompt word is obtained, and the invisible part material estimation prompt word and all image frames are input into the visual language large model for the second object recognition to obtain the unknown part material type and the unknown part proportion of the target object. The unknown part material type is the material type of the unknown part, and the unknown part proportion is the proportion of the unknown part in the target object. Taking a stainless steel water cup as an example, the unknown part material type is used to indicate whether the stainless steel water cup is filled with content, and the unknown part proportion can be the proportion of the content, etc.

[0106] In step S160 of some embodiments, the material database stores material types and material physical properties that match the material types. The material physical properties are physical property parameters related to the material types, including material density, friction coefficient between the mechanical arm gripper and the contact surface of the material type, maximum bending stress, maximum bending curvature, and other parameters. The material database is queried according to the target material type to obtain a first material physical property, which is the material physical property of the target material type. The material database is queried according to the unknown part material type to obtain a second material physical property, which is the material physical property of the unknown part material type.

[0107] See also Figure 4 In some embodiments, step S170 may include but is not limited to steps S410 to S430:

[0108] Step S410, calculating a first grasping force according to the three-dimensional model of the object, the physical properties of the first material, the physical properties of the second material, and the proportion of the unknown part;

[0109] Step S420, calculating a second grasping force according to the physical properties of the first material, the physical properties of the second material, and the object contact area of ​​the robot arm;

[0110] Step S430: determining a target grasping force according to the first grasping force and the second grasping force.

[0111] In step S410 of some embodiments, the first grasping force is the minimum grasping force that the gripper of the robot arm can grasp the target object. The first material physical property includes the friction coefficient between the robot arm gripper and each component (target material type) in the three-dimensional model of the object, the material density of each component and the volume of each component. The second material physical property includes the material density of the unknown part. Specifically, the friction coefficient of each component is averaged to obtain the average friction coefficient. The first grasping force is calculated based on the three-dimensional model of the object, the average friction coefficient, the material density of each component, the volume of each component, the material density of the unknown part and the proportion of the unknown part. The calculation formula of the first grasping force is expressed as:

[0112]

[0113] Among them, F min is the first grasping force; g is the gravity constant, which is 9.8; μ is the average friction coefficient between the gripper and the target object; θ is the upward lifting angle of the gripper, which is generally 0°; O is the target object, including the 3D model of the object and the unknown part; v is the volume of the component or the volume of the unknown part; ρ is the material density of the component or the material density of the unknown part.

[0114] It should be noted that the 3D model of the object includes multiple 3D point clouds or multiple Gaussian spheres, the volume of the component can be the volume of the 3D point cloud or the volume of the Gaussian sphere, the volume of the 3D point cloud can be a preset constant, and the volume of the Gaussian sphere can be represented by the standard deviation. The volume of the 3D model of the object can be determined according to the volume of all 3D point clouds or all Gaussian spheres, and the volume of the unknown part can be obtained by multiplying the proportion of the unknown part by the volume of the 3D model of the object.

[0115] In step S420 of some embodiments, the object contact area of ​​the robot arm is the surface area of ​​the contact between the clamp and the target object, which is a constant. The second grasping force is the maximum grasping force at which the clamp of the robot arm can grasp the target object without damaging the target object. The first material physical property also includes the maximum bending stress, the maximum bending curvature, and the Young's modulus corresponding to the material of each component. The second material physical property also includes the maximum bending stress, the maximum bending curvature, and the Young's modulus of the unknown part. The maximum bending stress of each component and the maximum bending stress of the unknown part are averaged to obtain the target maximum bending stress. The maximum bending curvature of each component and the maximum bending curvature of the unknown part are averaged to obtain the target maximum bending curvature. The Young's modulus of each component and the Young's modulus of the unknown part are averaged to obtain the target Young's modulus. The second grasping force is calculated based on the target maximum bending stress, the target maximum bending curvature, the target Young's modulus, the object contact area of ​​the robot arm, and the object contact thickness of the robot arm. The object contact thickness is the depth or thickness of the contact between the clamp and the target object, which is a constant. The calculation formula of the second grasping force is expressed as:

[0116]

[0117] Among them, F max is the second grasping force; A is the object contact area of ​​the robot arm; σ y is the target maximum bending stress; E is the target Young's modulus; d is the contact thickness of the object; κ max is the target maximum bending curvature.

[0118] In step S430 of some embodiments, an optimal grasping force is determined according to the first grasping force and the second grasping force to obtain a target grasping force.

[0119] The above steps S410 to S430 can obtain the best grasping force, so that the robot arm can grasp the target object with appropriate force.

[0120] See also Figure 5 In some embodiments, step S430 may include but is not limited to steps S510 to S530:

[0121] Step S510, comparing the first grasping force and the second grasping force to obtain a grasping force relationship; the grasping force relationship is used to indicate that the first grasping force is less than the second grasping force or the first grasping force is greater than or equal to the second grasping force;

[0122] Step S520, determining a gripping force limit value according to the gripping force relationship, the first gripping force, and the second gripping force; the gripping force limit value includes a lower gripping force limit and an upper gripping force limit;

[0123] Step S530, determining a target grasping force according to the lower grasping force limit and the upper grasping force limit.

[0124] In step S510 of some embodiments, the first grasping force and the second grasping force are compared to obtain a grasping force relationship, which is used to characterize the magnitude relationship between the first grasping force and the second grasping force, and may indicate that the first grasping force is smaller than the second grasping force, or that the first grasping force is greater than or equal to the second grasping force.

[0125] In step S520 of some embodiments, there may be parameter estimation errors between the first grasping force and the second grasping force. In order to obtain the best grasping force of the manipulator for the target object so that the target object can be grasped without slipping and damage, the first grasping force and the second grasping force are corrected according to the grasping force relationship to obtain a grasping force limit value, which is the limit value of the grasping force that the manipulator gripper can apply to the target object. The grasping force limit value includes a lower grasping force limit and an upper grasping force limit, wherein the lower grasping force limit is the minimum limit value of the grasping force, and the upper grasping force limit is the maximum limit value of the grasping force.

[0126] In step S530 of some embodiments, a grasping force set is constructed according to the lower grasping force, the upper grasping force, and the grasping force between the lower grasping force and the upper grasping force, and a grasping force is randomly selected from the grasping force set as the target grasping force. If the lower grasping force is represented by F1 and the upper grasping force is represented by F2, the grasping force set is represented by [F1, F2].

[0127] Alternatively, the lower limit grasping force and the upper limit grasping force are median-calculated to obtain the target grasping force. The formula for median calculation is defined as:

[0128]

[0129] Through the above steps S510 to S530, the optimal grasping force can be obtained, thereby avoiding the inability to grasp the object or the object slipping due to too small a grasping force, and avoiding the object being damaged due to too large a grasping force.

[0130] See also Figure 6 In some embodiments, step S520 may include but is not limited to step S610 or step S620:

[0131] Step S610, if the gripping force relationship indicates that the first gripping force is greater than or equal to the second gripping force, the second gripping force is used as the lower limit gripping force and the first gripping force is used as the upper limit gripping force;

[0132] Step S620, if the grasping force relationship indicates that the first grasping force is less than the second grasping force, the difference between the first grasping force and the second grasping force is calculated to obtain the grasping difference; the lower limit grasping force is calculated according to the first grasping force, the grasping difference and the preset weight factor; the upper limit grasping force is calculated according to the first grasping force, the grasping difference and the preset weight factor.

[0133] In step S610 of some embodiments, under normal circumstances, the first grasping force cannot be too small to ensure that the robot arm does not slip when grasping the object, and the second grasping force cannot be too large to ensure that the object is not damaged when the robot arm grasps the object, so the first grasping force is generally smaller than the second grasping force. Under abnormal circumstances, such as when the target object is too heavy but the surface of the object is very fragile, the first grasping force will be greater than or equal to the second grasping force. If the grasping force relationship indicates that the first grasping force is greater than or equal to the second grasping force, it means that an abnormal situation has occurred. In order to enable the robot arm to accurately grasp the target object, the second grasping force is used as the lower limit grasping force and the first grasping force is used as the upper limit grasping force.

[0134] In step S620 of some embodiments, if the grasping force relationship indicates that the first grasping force is less than the second grasping force, in order to obtain the optimal grasping force, the embodiments of the present application further narrow the grasping force value range. Specifically, the difference between the second grasping force and the first grasping force is calculated to obtain the grasping difference. The value of the preset weight factor is greater than 0 and less than 1. The grasping difference is multiplied by the preset weight factor to obtain the intermediate grasping force. The first grasping force and the intermediate grasping force are added to obtain the lower limit grasping force. The intermediate grasping force is subtracted from the second grasping force to obtain the upper limit grasping force.

[0135] Through the above steps S610 to S620, the minimum grasping force and the maximum grasping force can be corrected according to the grasping force relationship to improve the estimation accuracy of the optimal grasping force.

[0136] In some embodiments, the calculation formula of the target grasping force is defined as:

[0137]

[0138] Among them, F * represents the target grasping force; F min Indicates the first grasping force; F max represents the second grasping force; γ is the preset weight factor; ΔF is the capture difference, expressed as ΔF=max[0,F max -F min ]; Indicates that f is in f min and f max A grip between.

[0139] See also Figure 7 In some embodiments, step S180 may include but is not limited to steps S710 to S740:

[0140] Step S710, obtaining a first cross-sectional area of ​​the three-dimensional model of the object in a first direction, and calculating a first coordinate according to the first cross-sectional area, a first material physical property, and a second material physical property;

[0141] Step S720, obtaining a second cross-sectional area of ​​the three-dimensional model of the object in a second direction, and calculating a second coordinate according to the second cross-sectional area, the first material physical property, and the second material physical property;

[0142] Step S730, obtaining a third cross-sectional area of ​​the three-dimensional model of the object in a third direction, and calculating a third coordinate according to the third cross-sectional area, the first material physical property, and the second material physical property; any two of the first direction, the second direction, and the third direction are perpendicular to each other;

[0143] Step S740, determining the grasping position according to the first coordinate, the second coordinate and the third coordinate.

[0144] In step S710 of some embodiments, the first direction is the x-axis direction of the three-dimensional model of the object, and the first cross-sectional area is the cross-sectional area of ​​a certain position in the x-axis direction. The first coordinate is calculated according to the first cross-sectional area, the material density of each component, and the material density of the unknown part, and the first coordinate is the horizontal coordinate of the grasping position. The calculation method of the first coordinate can be:

[0145]

[0146] Wherein, l and l′ represent a position on one side of the x-axis direction and a position on the other side of the x-axis direction of the three-dimensional model of the object respectively; ρ x (l) and ρ x (l′) represents the material density at position l and position l′ in the x-axis direction, including the material density of the component and the material density of the unknown part; S x (l) and S x (l′) represents the first cross-sectional area at positions l and l′ respectively; x is the integration limit.

[0147] Solve for x to get the first coordinate. Specifically, along both sides of the x-axis, the mass of the three-dimensional point cloud or Gaussian sphere is accumulated. When the accumulated mass on one side is greater than that on the other side, the other side is accumulated first until the accumulated mass on both sides is equal, and x is used as the first coordinate. It should be noted that the mass of the three-dimensional point cloud or Gaussian sphere is obtained by the material density and the first cross-sectional area.

[0148] In step S720 of some embodiments, the second direction is the y-axis direction of the three-dimensional model of the object, the second cross-sectional area is the cross-sectional area of ​​a certain position in the y-axis direction, and the second coordinate is calculated according to the second cross-sectional area, the material density of each component and the material density of the unknown part, and the second coordinate is the ordinate of the grasping position. The calculation method of the second coordinate is:

[0149]

[0150] Where l and l′ represent a position on one side of the y-axis direction and a position on the other side of the y-axis direction of the three-dimensional model of the object respectively; ρ y (l) and ρ y (l′) represents the material density at position l and position l′ in the y-axis direction, including the material density of the component and the material density of the unknown part; S y (l) and S y (l′) represents the second cross-sectional area at positions l and l′ respectively; y is the integration limit.

[0151] The mass of the 3D point cloud or Gaussian sphere is accumulated along both sides of the y-axis. When the accumulated mass on one side is greater than that on the other side, the other side is accumulated first until the accumulated mass on both sides is equal. The integration limit y is used as the second coordinate. It should be noted that the mass of the 3D point cloud or Gaussian sphere is obtained through the material density and the second cross-sectional area.

[0152] In step S730 of some embodiments, the third direction is the z-axis direction of the three-dimensional model of the object, the third cross-sectional area is the cross-sectional area of ​​a certain position in the z-axis direction, and the third coordinate is calculated according to the third cross-sectional area, the material density of each component and the material density of the unknown part, and the third coordinate is the vertical coordinate of the grasping position. The calculation method of the third coordinate is:

[0153]

[0154] Wherein, l and l′ represent a position on one side of the z-axis direction of the three-dimensional model of the object and a position on the other side of the z-axis direction respectively; ρ z (l) and ρ z (l′) represents the material density at position l and position l′ in the z-axis direction, including the material density of the component and the material density of the unknown part; S z (l) and S z (l′) represents the third cross-sectional area at positions l and l′ respectively; z is the integration limit.

[0155] The mass of the 3D point cloud or Gaussian sphere is accumulated along both sides of the z axis. When the mass accumulated on one side is greater than that on the other side, the other side is accumulated first until the mass accumulated on both sides is equal. The integration limit z is used as the third coordinate. It should be noted that the mass of the 3D point cloud or Gaussian sphere is obtained through the material density and the third cross-sectional area.

[0156] In step S740 of some embodiments, the first coordinate, the second coordinate, and the third coordinate are used as the grasping position. * , the second coordinate is y * , the third coordinate is z * , then the grasping position is represented by P * =(x * ,y * ,z * ).

[0157] Through the above steps S710 to S740, an optimal grasping position can be obtained to prevent the target object from being damaged or slipping.

[0158] In step S190 of some embodiments, the robotic arm applies a target grasping force to the grasping position to grasp the target object, so that the robotic arm can universally and safely grasp objects of any material, thereby improving the accuracy of object grasping.

[0159] See also Figure 8 The embodiment of the present application also provides an object grasping device, which is applied to a robot arm and can implement the above-mentioned object grasping method. The object grasping device includes:

[0160] An acquisition module 810 is used to acquire images of a target object at multiple shooting angles to obtain multiple image frames;

[0161] A first recognition module 820 is used to perform a first object recognition on each image frame to obtain a material type of a visible component of the target object;

[0162] A 3D reconstruction module 830 is used to perform 3D reconstruction based on multiple image frames to obtain a 3D model of the object;

[0163] A marking module 840 is used to mark the material of the object three-dimensional model according to the material types of the visible components of all image frames to obtain a target material type;

[0164] The second recognition module 850 is used to perform second object recognition according to the image frame to obtain the material type and proportion of the unknown part of the target object;

[0165] A query module 860 is used to query a preset material database according to the target material type to obtain a first material physical property, and query the material database according to the unknown part material type to obtain a second material physical property;

[0166] A first calculation module 870, configured to calculate a target grasping force according to a three-dimensional model of the object, physical properties of the first material, physical properties of the second material, a proportion of an unknown part, and an object contact area of ​​the manipulator;

[0167] A second calculation module 880, for calculating a grasping position according to the three-dimensional model of the object, the physical properties of the first material, and the physical properties of the second material;

[0168] The grasping module 890 is used to grasp the target object according to the target grasping force and grasping position.

[0169] The specific implementation of the object grasping device is basically the same as the specific implementation of the above-mentioned object grasping method, and will not be repeated here.

[0170] The embodiment of the present application also provides an electronic device, the electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the object grasping method when executing the computer program. The electronic device can be any intelligent terminal including a tablet computer, a car computer, etc.

[0171] See also Fig. 9 , Fig. 9The hardware structure of an electronic device of another embodiment is illustrated, and the electronic device includes:

[0172] The processor 910 may be implemented by a general-purpose central processing unit (CPU), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;

[0173] The memory 920 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 920 can store an operating system and other application programs. When the technical solution provided in the embodiment of this specification is implemented by software or firmware, the relevant program code is stored in the memory 920, and the processor 910 calls and executes the object grasping method of the embodiment of this application;

[0174] Input / output interface 930, used to implement information input and output;

[0175] Communication interface 940, used to realize communication interaction between the device and other devices, which can be realized through wired mode (such as USB, network cable, etc.) or wireless mode (such as mobile network, WIFI, Bluetooth, etc.);

[0176] bus 950 , which transmits information between the various components of the device (e.g., processor 910 , memory 920 , input / output interface 930 , and communication interface 940 );

[0177] The processor 910 , the memory 920 , the input / output interface 930 , and the communication interface 940 are connected to each other in communication within the device via a bus 950 .

[0178] An embodiment of the present application also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the above-mentioned object grasping method is implemented.

[0179] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0180] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0181] Those skilled in the art will appreciate that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0182] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0183] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0184] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0185] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0186] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0187] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0188] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0189] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store programs.

[0190] The preferred embodiments of the present application are described above with reference to the accompanying drawings, but the scope of the rights of the present application is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present application should be within the scope of the rights of the present application.

Claims

1. A method for grasping an object, characterized in that: Applied to a robotic arm, the method comprises: Acquire images of the target object at multiple shooting angles to obtain multiple image frames; For each of the image frames, performing first object recognition on the image frame to obtain a material type of a visible component of the target object; Performing three-dimensional reconstruction based on the plurality of image frames to obtain a three-dimensional model of the object; Marking the material of the three-dimensional model of the object according to the material types of the visible components of all image frames to obtain a target material type; Performing a second object recognition according to the image frame to obtain a material type of an unknown part and a proportion of an unknown part of the target object; A preset material database is queried according to the target material type to obtain a first material physical property, and a material database is queried according to the unknown part material type to obtain a second material physical property; Calculating a target grasping force according to the three-dimensional model of the object, the physical properties of the first material, the physical properties of the second material, the proportion of the unknown part, and the object contact area of ​​the robotic arm; Calculating a grasping position according to the three-dimensional model of the object, the physical properties of the first material, and the physical properties of the second material; The target object is grasped according to the target grasping force and the grasping position.

2. The method according to claim 1, characterized in that The calculating the target grasping force according to the three-dimensional model of the object, the physical properties of the first material, the physical properties of the second material, the proportion of the unknown part and the object contact area of ​​the robot arm includes: Calculating a first grasping force according to the three-dimensional model of the object, the physical properties of the first material, the physical properties of the second material, and the proportion of the unknown part; Calculating a second grasping force according to the physical properties of the first material, the physical properties of the second material, and an object contact area of ​​the robotic arm; The target grasping force is determined according to the first grasping force and the second grasping force.

3. The method according to claim 2, characterized in that The step of determining the target grasping force according to the first grasping force and the second grasping force includes: Comparing the first grasping force and the second grasping force to obtain a grasping force relationship; the grasping force relationship is used to indicate that the first grasping force is less than the second grasping force or the first grasping force is greater than or equal to the second grasping force; determining a gripping force limit value according to the gripping force relationship, the first gripping force and the second gripping force; the gripping force limit value includes a lower gripping force limit and an upper gripping force limit; The target grasping force is determined according to the lower grasping force limit and the upper grasping force limit.

4. The method according to claim 3, wherein determining the gripping force limit value according to the gripping force relationship, the first gripping force and the second gripping force comprises: If the gripping force relationship indicates that the first gripping force is greater than or equal to the second gripping force, the second gripping force is used as the lower gripping force limit, and the first gripping force is used as the upper gripping force limit; If the grasping force relationship indicates that the first grasping force is smaller than the second grasping force, the difference between the first grasping force and the second grasping force is calculated to obtain a grasping difference; the lower limit grasping force is calculated based on the first grasping force, the grasping difference and a preset weight factor; the upper limit grasping force is calculated based on the first grasping force, the grasping difference and the preset weight factor.

5. The method according to claim 1, characterized in that The calculating the grasping position according to the object three-dimensional model, the first material physical property and the second material physical property comprises: Acquire a first cross-sectional area of ​​the three-dimensional model of the object in a first direction, and calculate a first coordinate according to the first cross-sectional area, the first material physical property, and the second material physical property; Acquire a second cross-sectional area of ​​the three-dimensional model of the object in a second direction, and calculate a second coordinate according to the second cross-sectional area, the first material physical property, and the second material physical property; Obtaining a third cross-sectional area of ​​the three-dimensional model of the object in a third direction, and calculating a third coordinate according to the third cross-sectional area, the first material physical property, and the second material physical property; any two of the first direction, the second direction, and the third direction are perpendicular to each other; The grasping position is determined according to the first coordinate, the second coordinate and the third coordinate.

6. The method according to any one of claims 1 to 5, characterized in that: The performing first object recognition on the image frame to obtain the material type of the visible component of the target object includes: Performing object detection on the image frame to obtain two-dimensional coordinates of the object; Performing preliminary instance segmentation on the image frame according to the two-dimensional coordinates of the object to obtain an object instance segmentation image; Performing instance segmentation again on the object instance segmentation image according to a preset hint point density to obtain a component instance segmentation image; Material recognition is performed based on the object instance segmentation image and the component instance segmentation image to obtain the material type of the visible component.

7. The method according to any one of claims 1 to 5, characterized in that: The step of marking the material of the three-dimensional model of the object according to the material types of the visible components of all image frames to obtain the target material type includes: Performing two-dimensional projection on the three-dimensional model of the object to obtain a component area; For each image frame, obtaining the original material type of the three-dimensional model of the object according to the component area and the material type of the visible component; The target material type is determined according to the original material type obtained from each image frame.

8. An object grasping device, characterized in that: Applied to a robotic arm, the device comprises: An acquisition module is used to acquire images of a target object at multiple shooting angles to obtain multiple image frames; A first recognition module, configured to perform a first object recognition on each of the image frames to obtain a material type of a visible component of the target object; A three-dimensional reconstruction module, used for performing three-dimensional reconstruction according to the plurality of image frames to obtain a three-dimensional model of the object; A marking module, used for marking the material of the three-dimensional model of the object according to the material type of the visible components of all image frames to obtain a target material type; A second recognition module is used to perform a second object recognition according to the image frame to obtain the material type of the unknown part and the proportion of the unknown part of the target object; A query module, configured to query a preset material database according to the target material type to obtain a first material physical property, and query the material database according to the unknown part material type to obtain a second material physical property; A first calculation module, configured to calculate a target grasping force according to the three-dimensional model of the object, the physical properties of the first material, the physical properties of the second material, the proportion of the unknown part, and the object contact area of ​​the robotic arm; A second calculation module, used for calculating a grasping position according to the three-dimensional model of the object, the physical properties of the first material and the physical properties of the second material; A grasping module is used to grasp the target object according to the target grasping force and the grasping position.

9. An electronic device, characterized in that: The electronic device comprises a memory and a processor, the memory stores a computer program, and the processor implements the method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images