Workpiece grabbing method and device, electronic equipment and storage medium

Abstract

The invention discloses a workpiece grabbing method and device, electronic equipment and a storage medium, and the method comprises the steps: synchronously shooting a first image and a second image of workpieces stacked in a material frame through a first camera and a second camera; wherein the first camera is mounted at the tail end of the mechanical arm, and the second camera is mounted at the edge of the material frame; inputting the first image and the second image into a deep learning model, and constructing a three-dimensional model of each workpiece in the material frame through the deep learning model; and target detection is conducted based on the three-dimensional model, the three-dimensional coordinates of the outer contour of each workpiece relative to the material frame are obtained, and according to the three-dimensional coordinates, the target workpiece is grabbed through the mechanical arm. According to the technical scheme, the workpieces in the material frame can be accurately grabbed without adopting automatic feeding equipment to arrange the workpieces in order.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of automation grabbing, and in particular to a workpiece grabbing method and device, electronic equipment and a storage medium. BACKGROUND

[0002] In a factory production scene, a large number of repeatedly used workpieces are placed in a frame after use, and subsequent workpieces need to be picked up by a mechanical arm to realize recycling. In the prior art, in order to facilitate the grabbing of the mechanical arm, an automatic feeding device is usually additionally arranged to arrange the workpieces in the frame neatly and then move them to a fixed position. This method not only increases the equipment procurement cost, but also consumes a lot of time in the process of arranging the workpieces, which seriously affects the overall efficiency of workpiece grabbing. Therefore, there is an urgent need for a method that can directly and accurately grab irregularly stacked workpieces in the frame without relying on a feeding device. SUMMARY

[0003] The present application provides a workpiece grabbing method and device, electronic equipment and a storage medium to accurately grab irregularly stacked workpieces in a frame.

[0004] According to one aspect of the present application, a workpiece grabbing method is provided, comprising:

[0005] The first camera and the second camera are used to synchronously capture a first image and a second image of the workpieces stacked in the frame, wherein the first camera is installed at the end of the mechanical arm, and the second camera is installed at the edge of the frame;

[0006] The first image and the second image are input into a deep learning model, and a three-dimensional model of each workpiece in the frame is constructed by the deep learning model;

[0007] Target detection is performed based on the three-dimensional model to obtain the three-dimensional coordinates of the outer contour of each workpiece relative to the frame, and a grabbing action is performed on the target workpiece by the mechanical arm according to the three-dimensional coordinates.

[0008] According to another aspect of the present application, a workpiece grabbing device is provided, comprising:

[0009] An image acquisition module is configured to synchronously capture a first image and a second image of the workpieces stacked in the frame by a first camera and a second camera, wherein the first camera is installed at the end of the mechanical arm, and the second camera is installed at the edge of the frame;

[0010] A model construction module is configured to input the first image and the second image into a deep learning model, and construct a three-dimensional model of each workpiece in the frame by the deep learning model;

[0011] The grabbing execution module performs target detection based on the three-dimensional model to obtain three-dimensional coordinates of an outer contour of each workpiece relative to the material frame, and performs a grabbing action on the target workpiece through the mechanical arm according to the three-dimensional coordinates.

[0012] According to another aspect of the present application, an electronic device is provided, the electronic device comprising:

[0013] at least one processor; and a memory connected with the at least one processor in communication; wherein the memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor to enable the at least one processor to perform the workpiece grabbing method of any embodiment of the present application.

[0014] According to another aspect of the present application, a computer readable storage medium is provided, the computer readable storage medium storing computer instructions for enabling a processor to implement the workpiece grabbing method of any embodiment of the present application when executed by the processor.

[0015] The technical solution of the embodiment of the present application synchronously collects double-view images of workpieces in the material frame through the first camera and the second camera, uses a deep learning model to complete feature extraction, cross-view matching, three-dimensional modeling and target detection, optimizes geometric reasoning accuracy in combination with camera parameters, improves reliability of the three-dimensional model through a point cloud refinement network, and finally realizes precise grabbing of the mechanical arm based on the workpiece gravity center coordinates, thereby solving the problems of high cost and low grabbing efficiency caused by dependence on the feeding equipment in the prior art, and the problem of inability to adapt to irregularly stacked workpieces, achieving the beneficial effects of no need for additional feeding equipment, reduced production cost, improved grabbing efficiency and stability, and adaptation to scattered workpiece grabbing scenes.

[0016] It should be understood that the content described in this part is not intended to identify key or important features of the embodiments of the present application, nor is it used to limit the scope of the present application. Other features of the present application will become apparent through the following description. BRIEF DESCRIPTION OF DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced as follows. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0018] Figure 1 A flowchart of a workpiece grabbing method provided by the embodiment of the present application is provided.

[0019] Figure 2 A flowchart of another workpiece grabbing method provided by the embodiment of the present application is provided.

[0020] Figure 3 A structural schematic diagram of a workpiece grabbing device provided by an embodiment of the present application is shown in the figure.

[0021] Figure 4 A structural schematic diagram of an electronic device for implementing a workpiece grabbing method of an embodiment of the present application is shown in the figure. DETAILED DESCRIPTION

[0022] In order to enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the accompanying drawings of the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative efforts should fall within the protection scope of the present application.

[0023] It should be noted that the terms "first", "second", and the like in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and do not necessarily indicate a specific order or a chronological sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device that includes a series of steps or units does not necessarily have to be limited to those steps or units clearly listed, but can include other steps or units that are not clearly listed or inherent to the process, method, product or device.

[0024] Figure 1 A flowchart of a workpiece grabbing method provided by an embodiment of the present application is shown in the figure. The embodiment can be applicable to the case of accurately grabbing workpieces irregularly stacked in a material frame. The method can be executed by a workpiece grabbing device, which can be realized in the form of hardware and / or software, and can be configured in an electronic device. As shown in the figure, the method specifically includes the following steps: Figure 1

[0025] S110, synchronously capturing, by a first camera and a second camera, a first image and a second image of the workpieces stacked in the material frame.

[0026] The first camera is installed at the end of a mechanical arm, and the second camera is installed at the edge of the material frame. The first camera and the second camera are image acquisition devices with synchronous capturing function, and are used to acquire image information of the workpieces in the material frame under different viewing angles.

[0027] ​Specifically, the first camera can be installed at the end of the mechanical arm, and the second camera can be installed at the edge of the material frame. The two cameras are controlled to start synchronously to capture the first image and the second image of the workpieces stacked in the material frame.

[0028] S120, input the first image and the second image into a deep learning model, and construct a three-dimensional model of each workpiece in the material frame through the deep learning model.

[0029] The deep learning model can be a neural network model capable of feature extraction and three-dimensional modeling, and can be used to convert two-dimensional images into three-dimensional spatial information.

[0030] Specifically, the first image and the second image obtained by photographing are simultaneously input into the deep learning model, and the three-dimensional model corresponding to each workpiece is constructed by processing the images through the deep learning model.

[0031] In some embodiments, the three-dimensional model of each workpiece in the material frame is constructed through the deep learning model, including:

[0032] The shared image encoder in the deep learning model is used to extract features from the first image and the second image respectively, to obtain first multi-scale features of the first image and second multi-scale features of the second image.

[0033] The cross-view matching relationship between the first multi-scale features and the second multi-scale features is established based on an attention mechanism, and geometric reasoning is performed in combination with camera parameters.

[0034] The depth information of each pixel is estimated through the cross-view matching relationship and camera geometric constraints, and the pixel points are back-projected to a three-dimensional space through a differentiable triangulation process to obtain a preliminary point cloud of the workpiece.

[0035] The preliminary point cloud is denoised and completed through confidence weighting, double-view consistency checking, and a point cloud refinement network to obtain a three-dimensional point set corresponding to each workpiece, and the three-dimensional point set is taken as the three-dimensional model of the workpiece.

[0036] The shared image encoder can be a component in a deep learning model for extracting image features, and can process two images and output multi-scale features; the first multi-scale feature and the second multi-scale feature are feature information containing local texture and global structure obtained after feature extraction of the first image and the second image respectively; the cross-view matching relationship can represent the corresponding relationship of feature points in the two images; the camera parameters can be parameters describing the imaging characteristics and spatial position of the camera; the differentiable triangulation process can be an algorithm process of mapping a pixel point to a three-dimensional space; the preliminary point cloud can be original spatial data composed of a large number of discrete three-dimensional points; the point cloud refinement network can be a neural network module for optimizing point cloud data; and the three-dimensional point set can be a three-dimensional data set that can completely represent the spatial form of the workpiece after optimization.

[0037] Specifically, the shared image encoder in the deep learning model can be started to perform feature extraction operations on the first image and the second image respectively to obtain the first multi-scale feature corresponding to the first image and the second multi-scale feature corresponding to the second image; the first multi-scale feature and the second multi-scale feature are processed based on an attention mechanism to establish a cross-view matching relationship therebetween, and spatial geometric reasoning is performed in combination with preset camera parameters; the depth information of each pixel is calculated through the deep learning model by using the established cross-view matching relationship and the geometric constraint caused by the camera, and all pixel points are back projected to a three-dimensional space through a differentiable triangulation process to form a preliminary point cloud of the workpiece.

[0038] Further, the preliminary point cloud is preprocessed in a confidence weighting and double-view consistency checking manner, the sparse areas in the preliminary point cloud are completed through a point cloud refinement network, and noise points are removed therefrom to obtain a three-dimensional point set corresponding to each workpiece, and the three-dimensional point set is taken as a three-dimensional model of the workpiece. Through multi-step feature processing and point cloud optimization, an accurate and complete three-dimensional model of the workpiece can be constructed to provide reliable data support for subsequent target detection and grasping.

[0039] In some embodiments, the geometric reasoning in combination with the camera parameters comprises:

[0040] determining a spatial geometric relationship based on intrinsic parameters and extrinsic parameters in the camera parameters, wherein the intrinsic parameters of the camera parameters include focal length, principal point coordinates and pixel size, and the extrinsic parameters include a rotation matrix and a translation vector;

[0041] calculating a disparity search range according to a preset range of scene depth, a horizontal focal length of the camera parameters and a baseline length between optical centers of the two cameras;

[0042] determining the disparity of the corresponding points in the first image and the second image based on the disparity search range, and completing the geometric reasoning.

[0043] The intrinsic parameter can be understood as a parameter describing the imaging geometry of the camera itself, which is irrelevant to the camera installation position; the extrinsic parameter can be understood as a parameter describing the spatial position and attitude of the camera relative to the world coordinate system; the spatial geometric relationship can refer to the positional relationship between the camera and the workpiece, and the positional relationship between the two cameras; the preset range of scene depth can be the distance interval of the workpiece to the camera according to the actual production scene; the horizontal focal length can be the focal length parameter in the horizontal direction in the camera intrinsic parameter; the baseline length can be the straight line distance between the optical centers of the first camera and the second camera; the disparity search range can be the horizontal pixel difference interval for searching for corresponding points of images; and the disparity can be understood as the difference value of the horizontal pixel position of the same three-dimensional point in two images.

[0044] Specifically, the intrinsic parameter and the extrinsic parameter of the camera can be acquired first, the intrinsic parameter including a focal length, a principal point coordinate and a pixel size, and the extrinsic parameter including a rotation matrix and a translation vector, the spatial geometric relationship between the camera and the workpiece and the spatial geometric relationship between the two cameras are determined based on the intrinsic parameter and the extrinsic parameter of the camera; the disparity search range is calculated by a formula according to the preset range of scene depth set according to the actual production scene, the horizontal focal length in the camera intrinsic parameter and the baseline length between the optical centers of the two cameras; and then the disparity of the corresponding points is searched and determined in the first image and the second image based on the calculated disparity search range, and the geometric reasoning process is completed. In this way, the disparity is accurately calculated through the camera parameters and the preset range, the search amount and the mismatch probability in the matching process are reduced, and the efficiency and accuracy of the geometric reasoning are improved.

[0045] Exemplarily, the disparity search range is determined as follows: with the prior depth

[0046] If there is a general range of scene depth , the disparity search range can be calculated as follows:

[0047]

[0048] : horizontal focal length in pixels (intrinsic parameter of the camera ).

[0049] : baseline length between the optical centers of the two cameras (meters or millimeters, and use the same length unit).

[0050] The search amount can be significantly reduced and the mismatch can be reduced.

[0051] In some embodiments, the point cloud refinement network is a network module based on a graph neural network or a network module based on PointNet++, and the point cloud refinement network is used to complete the sparse area in the preliminary point cloud and eliminate the noise points in the preliminary point cloud.

[0052] In practical applications, the depth information of each pixel can be estimated by a model using the matching relationship and camera geometric constraints. Through a differentiable triangulation process, the pixel points can be back-projected into a three-dimensional space to obtain a preliminary point cloud of the object. Then, point cloud fusion and refinement can be performed. As can be understood, the point clouds from the two images usually have noise and sparsity. The point cloud can be denoised and completed by confidence weighting, double-view consistency checking, and a point cloud refinement network, such as a graph neural network or a PointNet++ module, to obtain a more complete and uniform three-dimensional point set. Each workpiece can obtain a one-to-one corresponding three-dimensional point set, and the three-dimensional point set is used to represent the three-dimensional model of the corresponding workpiece.

[0053] S130, target detection is performed based on the three-dimensional model to obtain three-dimensional coordinates of the outer contour of each workpiece relative to the frame, and a grabbing action is performed on the target workpiece by the mechanical arm according to the three-dimensional coordinates.

[0054] The three-dimensional coordinates can be understood as the position parameters of the outer contour of the workpiece in the space coordinate system with the frame as the reference.

[0055] Specifically, target detection is performed based on the constructed three-dimensional model to obtain three-dimensional coordinates of the outer contour of each workpiece relative to the frame, and the mechanical arm adjusts the action according to the three-dimensional coordinates to complete the grabbing of the target workpiece. This method does not require additional feeding equipment and directly realizes the grabbing of irregularly stacked workpieces, which reduces the equipment cost, saves the workpiece alignment step, and improves the grabbing efficiency.

[0056] The technical scheme of the embodiment of the application synchronously collects double-view images of the workpieces in the frame by the first camera and the second camera, uses a deep learning model to complete feature extraction, cross-view matching, three-dimensional modeling and target detection, optimizes the geometric reasoning accuracy in combination with camera parameters, improves the reliability of the three-dimensional model through a point cloud refinement network, and finally realizes accurate grabbing of the mechanical arm based on the workpiece gravity center coordinates, thereby solving the problems of high cost and low grabbing efficiency caused by the dependence on feeding equipment in the prior art, and the problems of being unable to adapt to irregularly stacked workpieces. The beneficial effects of not requiring additional feeding equipment, reducing production cost, improving grabbing efficiency and stability, and simultaneously adapting to scattered workpiece grabbing scenarios are achieved.

[0057] Figure 2 The flowchart of another workpiece grabbing method provided by the embodiment of the application is further optimized for the determination of the three-dimensional coordinates of the outer contour of each workpiece relative to the frame and the grabbing of the target workpiece based on the foregoing embodiment. As shown in Figure 2 The method specifically includes the following steps:

[0058] S210, the first camera and the second camera are used to synchronously capture a first image and a second image of the workpieces stacked in the frame.

[0059] S220. Input the first image and the second image into a deep learning model, and construct a three-dimensional model of each workpiece in the material frame through the deep learning model.

[0060] S230. Divide the feature space corresponding to the three-dimensional model into a grid of fixed size.

[0061] The feature space is the space formed by the digital features corresponding to the three-dimensional model; the grid is a fixed-size unit region obtained by dividing the feature space.

[0062] S240. Process each grid using a convolutional neural network to predict the candidate bounding box, candidate bounding box confidence, and workpiece category probability of each workpiece within the grid.

[0063] Here, the candidate box can be a predicted bounding box that may contain a workpiece; the candidate box confidence can be understood as the probability that the candidate box actually contains a workpiece; the workpiece category probability can be the probability that the workpiece in the candidate box belongs to a preset category.

[0064] Specifically, the feature space corresponding to the constructed 3D model can be divided into multiple grids of fixed size; then, based on a convolutional neural network, feature extraction and data analysis are performed on each grid to predict the candidate boxes of the workpieces that may exist in each grid, the confidence level of the candidate boxes, and the probability of the workpiece category.

[0065] In some embodiments, processing each grid using a convolutional neural network includes: extracting features of the 3D model using the backbone network of the convolutional neural network, wherein the backbone network is a Darknet network or a CSPDarknet network; and predicting workpiece information within each grid on feature maps at different scales using the detection head of the convolutional neural network, wherein the workpiece information includes candidate boxes, candidate box confidence, and workpiece category probability.

[0066] The backbone network can be understood as the part of the convolutional neural network responsible for extracting basic features; Darknet is a deep learning network architecture used for object detection; CSPDarknet is a network architecture based on CSPNet with higher feature extraction efficiency; the detection head is the part of the convolutional neural network responsible for analyzing features and outputting prediction results; and the feature map can be an image output from different layers of the convolutional neural network, containing feature information at different scales.

[0067] Specifically, a Darknet network or a CSPDarknet network can be used as a backbone network of the convolutional neural network to hierarchically extract features of the three-dimensional model and obtain feature maps of different scales; a detection head of the convolutional neural network is used to analyze and process the feature maps of each scale, to predict the workpiece information in each grid on the feature maps of different scales, and to output the candidate box, the candidate box confidence and the workpiece class probability of each workpiece in each grid. Through the special backbone network and multi-scale feature prediction, the efficiency and accuracy of feature extraction are improved, and accurate detection of workpieces of different sizes is ensured.

[0068] S250, according to the candidate box, the candidate box confidence and the workpiece class probability, filtering out the target bounding box of each workpiece.

[0069] Among them, the target bounding box can be a bounding box that can accurately surround the outer contour of the workpiece after filtering.

[0070] Specifically, according to the preset confidence threshold and class probability threshold, all candidate boxes are filtered, and the candidate boxes meeting the requirements are retained as the target bounding box of each workpiece; the spatial coordinate information corresponding to each target bounding box is extracted to obtain the three-dimensional coordinates of the outer contour of each workpiece relative to the upper edge of the material frame.

[0071] S260, extracting the coordinate information corresponding to the target bounding box to obtain the three-dimensional coordinates of the outer contour of each workpiece relative to the upper edge of the material frame.

[0072] S270, according to the three-dimensional coordinates, the mechanical arm performs a grabbing action on the target workpiece.

[0073] In some embodiments, according to the three-dimensional coordinates, the mechanical arm performs a grabbing action on the target workpiece, including:

[0074] Obtaining the x-axis coordinate, y-axis coordinate and z-axis coordinate of all points in the three-dimensional coordinates;

[0075] Summing up all the x-axis coordinates, y-axis coordinates and z-axis coordinates respectively to obtain the x-axis coordinate sum, y-axis coordinate sum and z-axis coordinate sum;

[0076] Based on the x-axis coordinate sum, the y-axis coordinate sum, the z-axis coordinate sum and the number of three-dimensional coordinate points of the outer contour of the workpiece, the three-dimensional candidate coordinates of the center of gravity of the workpiece relative to the upper edge of the material frame are obtained;

[0077] According to the three-dimensional candidate coordinates, the mechanical arm adjusts its posture and position so that the grabbing component is aligned with the three-dimensional candidate coordinates corresponding to the center of gravity of the target workpiece, and the target workpiece is grabbed.

[0078] Wherein, the x-axis coordinate, the y-axis coordinate and the z-axis coordinate can be understood as three dimension parameters of the workpiece outer contour three-dimensional coordinates in the space rectangular coordinate system; the coordinate total is the accumulation result of all coordinate values corresponding to the dimension; the three-dimensional candidate coordinates can be the space coordinates representing the workpiece gravity center position; the grabbing component is an execution mechanism on the mechanical arm for contacting and grabbing the workpiece.

[0079] Specifically, from the acquired three-dimensional coordinates of the workpiece outer contour relative to the frame, the x-axis coordinates, the y-axis coordinates and the z-axis coordinates of all points are extracted; the x-axis coordinates, the y-axis coordinates and the z-axis coordinates are summed respectively to obtain the x-axis coordinate total, the y-axis coordinate total and the z-axis coordinate total; then, the ratios of the x-axis coordinate total, the y-axis coordinate total and the z-axis coordinate total to the number of the workpiece outer contour three-dimensional coordinate points are calculated to obtain the three-dimensional candidate coordinates of the workpiece gravity center relative to the frame upper edge; when the mechanical arm receives the three-dimensional candidate coordinates, the posture and the space position of the mechanical arm are adjusted to make the grabbing component accurately align the three-dimensional candidate coordinates corresponding to the gravity center of the target workpiece, the grabbing component is controlled to be closed to complete the grabbing of the target workpiece, and the grabbing based on the gravity center coordinates can improve the stability and the success rate of the mechanical arm grabbing and avoid the situation of grabbing deviation or workpiece falling off.

[0080] The technical scheme of the embodiment of the application synchronously collects the double-view images of the workpieces in the frame through the first camera and the second camera, uses a deep learning model to complete feature extraction, cross-view matching, three-dimensional modeling and target detection, optimizes the geometric reasoning accuracy in combination with camera parameters, improves the reliability of the three-dimensional model through a point cloud refinement network, and finally realizes accurate grabbing of the mechanical arm based on the workpiece gravity center coordinates, thereby solving the problems of high cost, low grabbing efficiency and inability to adapt to irregularly stacked workpieces caused by the dependence on the feeding equipment in the prior art, achieving the beneficial effects of no need for additional feeding equipment, reduced production cost, improved grabbing efficiency and stability, and adaptation to scattered workpiece grabbing scenes.

[0081] Figure 3 A structure schematic diagram of a workpiece grabbing device provided by the embodiment of the application is shown in Figure 3 The device comprises:

[0082] An image acquisition module 310 is configured to synchronously capture a first image and a second image of the workpieces stacked in the frame through a first camera and a second camera; wherein the first camera is installed at the end of the mechanical arm, and the second camera is installed at the edge of the frame.

[0083] A model construction module 320 is configured to input the first image and the second image into a deep learning model, and construct a three-dimensional model of each workpiece in the frame through the deep learning model.

[0084] The grabbing execution module 330 performs target detection based on the three-dimensional model, obtains three-dimensional coordinates of an outer contour of each workpiece relative to the material frame, and performs grabbing actions on the target workpiece through the mechanical arm according to the three-dimensional coordinates.

[0085] The technical scheme of the embodiment of the application synchronously collects double-view images of workpieces in the material frame through the first camera and the second camera, uses a deep learning model to complete feature extraction, cross-view matching, three-dimensional modeling and target detection, optimizes geometric reasoning accuracy in combination with camera parameters, improves reliability of a three-dimensional model through a point cloud refinement network, and finally realizes precise grabbing of a mechanical arm based on a workpiece gravity center coordinate, thereby solving the problems of high cost and low grabbing efficiency caused by dependence on a feeding device in the prior art, and the problem of inability to adapt to irregularly stacked workpieces, achieving the beneficial effects of no need for additional feeding devices, reduced production cost, improved grabbing efficiency and stability, and simultaneous adaptation to a scattered workpiece grabbing scene.

[0086] In some embodiments, the model construction module 320 comprises:

[0087] The feature extraction submodule is configured to perform feature extraction on the first image and the second image respectively through a shared image encoder in the deep learning model, to obtain first multi-scale features of the first image and second multi-scale features of the second image.

[0088] The cross-view matching submodule is configured to establish a cross-view matching relationship between the first multi-scale features and the second multi-scale features based on an attention mechanism, and perform geometric reasoning in combination with camera parameters.

[0089] The preliminary point cloud determination module is configured to estimate depth information of each pixel through the deep learning model by using the cross-view matching relationship and camera geometric constraints, and to project the pixel points back to a three-dimensional space through a differentiable triangulation process to obtain preliminary point clouds of the workpieces.

[0090] The three-dimensional model establishment module is configured to denoise and complete the preliminary point clouds by confidence weighting, double-view consistency checking and a point cloud refinement network, to obtain a three-dimensional point set corresponding to each workpiece, and to take the three-dimensional point set as the three-dimensional model of the workpiece.

[0091] In some embodiments, the cross-view matching submodule is specifically configured to:

[0092] Determine a spatial geometric relationship based on intrinsic parameters and extrinsic parameters in the camera parameters, wherein the intrinsic parameters of the camera parameters include focal length, principal point coordinates and pixel size, and the extrinsic parameters include a rotation matrix and a translation vector.

[0093] According to a preset range of scene depth, a horizontal focal length of the camera parameters and a baseline length between optical centers of the two cameras, a parallax search range is calculated.

[0094] Based on the parallax search range, a parallax of a corresponding point in the first image and the second image is determined, and geometric reasoning is completed.

[0095] In some embodiments, the point cloud refinement network is a graph neural network-based network module or a PointNet++-based network module, which is used to complete the sparse areas in the preliminary point cloud and eliminate the noise points in the preliminary point cloud.

[0096] In some embodiments, the grasping execution module 330 comprises:

[0097] A grid division sub-module is configured to divide a feature space corresponding to the three-dimensional model into fixed-size grids.

[0098] A prediction sub-module is configured to process each of the grids by a convolutional neural network to predict a candidate box, a candidate box confidence and a workpiece category probability of a workpiece in each of the grids.

[0099] A bounding box screening sub-module is configured to screen a target bounding box of each workpiece according to the candidate box, the candidate box confidence and the workpiece category probability.

[0100] A three-dimensional coordinate determination sub-module is configured to extract coordinate information corresponding to the target bounding box to obtain a three-dimensional coordinate of an outer contour of each workpiece relative to an upper edge of a material frame.

[0101] In some embodiments, the prediction sub-module is specifically configured to:

[0102] extract features of the three-dimensional model by a backbone network of the convolutional neural network, the backbone network being a Darknet network or a CSPDarknet network;

[0103] predict workpiece information in each of the grids on feature maps of different scales by a detection head of the convolutional neural network, the workpiece information including the candidate box, the candidate box confidence and the workpiece category probability.

[0104] In some embodiments, the grasping execution module 330 is specifically configured to:

[0105] obtain x-axis coordinates, y-axis coordinates and z-axis coordinates of all points in the three-dimensional coordinates;

[0106] sum all the x-axis coordinates, all the y-axis coordinates and all the z-axis coordinates to obtain a total sum of the x-axis coordinates, a total sum of the y-axis coordinates and a total sum of the z-axis coordinates;

[0107] Based on the x-axis coordinate sum, the y-axis coordinate sum, the z-axis coordinate sum, and the number of three-dimensional coordinate points of the outer contour of the workpiece, a three-dimensional candidate coordinate of the center of gravity of the workpiece relative to the upper edge of the frame is obtained.

[0108] According to the three-dimensional candidate coordinate, the posture and position of the mechanical arm are adjusted so that the grabbing component is aligned with the three-dimensional candidate coordinate corresponding to the center of gravity of the target workpiece, and the grabbing of the target workpiece is completed.

[0109] The workpiece grabbing device provided in the embodiments of the present application can execute the workpiece grabbing method provided in any of the embodiments of the present application, and has the corresponding function modules and beneficial effects of the execution method.

[0110] Figure 4 A structural schematic diagram of an electronic device for implementing the workpiece grabbing method of the embodiments of the present application. The electronic device is intended to represent various forms of digital computers, such as laptops, desktops, tablets, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular telephones, smartphones, wearable devices (such as helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions, are meant to be examples only, and are not intended to limit the implementations of the present application described and / or claimed in this document.

[0111] As shown in Figure 4 The electronic device 10 includes at least one processor 11, and a memory, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., which is communicatively connected to the at least one processor 11, wherein the memory stores a computer program that can be executed by the at least one processor. The processor 11 can execute various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 to the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other through a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0112] The plurality of components in the electronic device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, a loudspeaker, etc.; a storage unit 18, such as a magnetic disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network, such as the Internet, and / or various telecommunication networks.

[0113] The processor 11 can be various general and / or special purpose processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various specialized artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The processor 11 performs various methods and processes described above, such as the workpiece grasping method.

[0114] In some embodiments, the workpiece grasping method can be implemented as a computer program tangibly embodied in a computer readable storage medium, such as the storage unit 18. In some embodiments, part or all of the computer program can be loaded and / or installed onto the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded onto the RAM 13 and executed by the processor 11, one or more steps of the workpiece grasping method described above can be performed. Alternatively, in other embodiments, the processor 11 can be configured to perform the workpiece grasping method by any other suitable means, such as by means of firmware.

[0115] Various implementations of the systems and techniques described above can be realized in digital electronic circuitry, integrated circuitry, a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a system on a chip (SOC), a complex programmable logic device (CPLD), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include implementation in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which can be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0116] Computer programs used to implement the methods of the application can be written in any combination of one or more programming languages. These computer programs can be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the computer program, when executed, implements the functions / acts specified in the flowcharts and / or block diagrams. The computer program can be executed entirely on a machine, partially on a machine, partially on a machine as a stand-alone software package, and partially on a machine or a remote machine or a server.

[0117] In the context of the present application, a computer-readable storage medium can be a tangible medium that can contain or store a computer program for use by or in connection with an instruction execution system, apparatus, or device. A computer-readable storage medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium can be a machine-readable signal medium. More specific examples of a machine-readable storage medium will include one or more lines of a program of instructions in a transitory signal, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0118] To provide for interaction with a user, the systems and techniques described here can be implemented on an electronic device having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the electronic device. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including acoustic, speech, or tactile input.

[0119] The systems and techniques described here can be implemented in a computing system that includes a back end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front end component (e.g., a user computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0120] The computing system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system, to solve the defects of large management difficulty and weak business scalability in traditional physical host and VPS service.

[0121] It should be understood that the various forms of flow shown above can be reordered, added to, or have steps deleted. For example, the steps described in the present application can be performed in parallel, in series, or in a different order, as long as the desired results of the technical solutions of the present application can be achieved, and this is not limited herein.

[0122] The above detailed description does not constitute a limitation on the protection scope of the present application. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent replacements, and improvements made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A workpiece gripping method, characterized in that, include: The first camera and the second camera simultaneously capture first and second images of the workpieces stacked inside the material frame; wherein, the first camera is installed at the end of the robotic arm and the second camera is installed at the edge of the material frame; The first image and the second image are input into a deep learning model, and a three-dimensional model of each workpiece in the material frame is constructed through the deep learning model. Target detection is performed based on the three-dimensional model to obtain the three-dimensional coordinates of the outer contour of each workpiece relative to the material frame, and the robotic arm performs a gripping action on the target workpiece according to the three-dimensional coordinates.

2. The method according to claim 1, characterized in that, The construction of a 3D model for each workpiece within the material frame using the deep learning model includes: The shared image encoder in the deep learning model is used to extract features from the first image and the second image respectively, to obtain the first multi-scale feature of the first image and the second multi-scale feature of the second image. A cross-view matching relationship between the first multi-scale feature and the second multi-scale feature is established based on an attention mechanism, and geometric inference is performed in combination with camera parameters; Using the cross-view matching relationship and camera geometric constraints, the depth information of each pixel is estimated through the deep learning model, and the pixel points are back-projected into three-dimensional space through a differentiable triangulation process to obtain the preliminary point cloud of the workpiece. By using confidence weighting, dual-view consistency checking, and point cloud refinement network, the preliminary point cloud is denoised and completed to obtain a three-dimensional point set corresponding to each workpiece, and the three-dimensional point set is used as the three-dimensional model of the workpiece.

3. The method according to claim 2, characterized in that, The geometric reasoning based on camera parameters includes: The spatial geometric relationship is determined based on the intrinsic and extrinsic parameters in the camera parameters, wherein the intrinsic parameters of the camera parameters include focal length, principal point coordinates and pixel size, and the extrinsic parameters include rotation matrix and translation vector; The parallax search range is calculated based on the preset range of scene depth, the horizontal focal length of the camera parameters, and the baseline length between the optical centers of the two cameras. Based on the disparity search range, the disparity of corresponding points in the first image and the second image is determined to complete geometric reasoning.

4. The method according to claim 2, characterized in that, The point cloud refinement network is a network module based on graph neural networks or a network module based on PointNet++. The point cloud refinement network is used to complete the sparse regions in the initial point cloud and remove noise points in the initial point cloud.

5. The method according to claim 1, characterized in that, The target detection based on the three-dimensional model, obtaining the three-dimensional coordinates of each workpiece's outer contour relative to the material frame, includes: The feature space corresponding to the three-dimensional model is divided into a grid of fixed size; Each grid is processed by a convolutional neural network to predict the candidate bounding box, the confidence score of the candidate bounding box, and the probability of the workpiece category within each grid. Based on the candidate boxes, the confidence scores of the candidate boxes, and the probability of the workpiece category, the target bounding box of each workpiece is selected. Extract the coordinate information corresponding to the target bounding box to obtain the three-dimensional coordinates of the outer contour of each workpiece relative to the upper edge of the material frame.

6. The method according to claim 5, characterized in that, The process of processing each grid using a convolutional neural network includes: The features of the 3D model are extracted through the backbone network of the convolutional neural network, wherein the backbone network is a Darknet network or a CSPDarknet network. The detection head of the convolutional neural network predicts the workpiece information within each grid on feature maps of different scales. The workpiece information includes candidate boxes, candidate box confidence, and workpiece category probability.

7. The method according to claim 1, characterized in that, The step of performing a gripping action on the target workpiece using a robotic arm based on the three-dimensional coordinates includes: Obtain the x-axis, y-axis, and z-axis coordinates of all points in the three-dimensional coordinate system; Summing up all the x-axis coordinates, y-axis coordinates, and z-axis coordinates respectively yields the sum of the x-axis coordinates, the sum of the y-axis coordinates, and the sum of the z-axis coordinates. Based on the sum of the x-axis coordinates, the sum of the y-axis coordinates, the sum of the z-axis coordinates, and the number of three-dimensional coordinate points of the outer contour of the workpiece, the three-dimensional candidate coordinates of the center of gravity of the workpiece relative to the upper edge of the material frame are obtained. Based on the three-dimensional candidate coordinates, the robotic arm adjusts its posture and position so that the gripping component is aligned with the three-dimensional candidate coordinates corresponding to the center of gravity of the target workpiece, thereby completing the gripping of the target workpiece.

8. A workpiece gripping device, characterized in that, include: The image acquisition module is used to simultaneously capture a first image and a second image of the workpieces stacked in the material frame using a first camera and a second camera; wherein, the first camera is installed at the end of the robotic arm and the second camera is installed at the edge of the material frame; The model building module is used to input the first image and the second image into a deep learning model, and to build a three-dimensional model of each workpiece in the material frame through the deep learning model; The grasping execution module performs target detection based on the three-dimensional model to obtain the three-dimensional coordinates of the outer contour of each workpiece relative to the material frame, and performs grasping action on the target workpiece by means of the robotic arm according to the three-dimensional coordinates.

9. An electronic device, characterized in that, The electronic device includes: At least one processor; and a memory communicatively connected to the at least one processor; The memory stores a computer program that can be executed by the at least one processor, which is then executed by the at least one processor to enable the at least one processor to perform the workpiece gripping method according to any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that cause a processor to execute the workpiece gripping method according to any one of claims 1-7.

Images