Grabbing method and equipment for industrial parts of feeding and discharging production line and medium
By processing RGB and depth images with statistical methods to calculate surface normals and adjust robotic joints, the method addresses the challenge of grasping varied industrial parts, improving production line efficiency and stability.
Patent Information
- Application Number
- CN202510461456.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-04-14
- Publication Date
- 2025-07-15
AI Technical Summary
When handling industrial parts with varying shapes, sizes and materials, existing robotics have problems such as inaccurate identification, unstable grasping and low loading and unloading efficiency, which seriously restricts the efficiency and stability of the production line.
By obtaining RGB images, depth images and point cloud data of industrial parts, the target segmentation mask is obtained using SAM model and depth image processing technology, combining point cloud data calculation normal vectors and pose information, the robot joint angle is adjusted to achieve accurate capture.
It realizes accurate grasp of variable industrial parts, and improves the operating efficiency and stability of the loading and unloading production line.
Smart Images
Figure CN120308630A_ABST
Abstract
Description
Technical Field
[0001] This application relates to robot technology in the field of industrial automation, and particularly to a method, device, and medium for grasping industrial parts on a loading and unloading production line. Background Art
[0002] In the automated production lines of discrete manufacturing, the loading and unloading of workpieces is a key link. However, existing robot technologies have problems such as inaccurate recognition, unstable grasping, and low loading and unloading efficiency when dealing with industrial parts with diverse shapes, sizes, and materials, which severely restrict the efficiency and stability of the production line. Summary of the Invention
[0003] In view of the above problems and technical requirements, this application proposes a method, device, and medium for grasping industrial parts on a loading and unloading production line to solve the problem of grasping diverse industrial parts by robots in the prior art and achieve precise grasping and efficient loading and unloading of industrial parts.
[0004] An embodiment of the application provides a method for grasping industrial parts on a loading and unloading production line, and the method includes:
[0005] Based on the operation of the loading and unloading production line, obtain RGB images, depth images, and point cloud data including industrial parts;
[0006] Based on the RGB image and the depth image, obtain a target segmentation mask, and obtain the contour of the industrial part and its position in the RGB image through the target segmentation mask;
[0007] Extract target point cloud data corresponding to the contour and the position from the point cloud data;
[0008] Calculate the normal vector corresponding to the target point cloud data based on a statistical method, and determine the pose information of the industrial part based on the state and distribution of the normal vector;
[0009] Based on the pose information, adjust the target angles of the joints of the robot to control the robot to grasp the industrial part.
[0010] According to the method for grasping industrial parts on a loading and unloading production line provided by the embodiment of the application, obtaining a target segmentation mask based on the RGB image and the depth image includes:
[0011] Input the RGB image into a pre-trained SAM model to obtain an initial segmentation mask output by the SAM model;
[0012] Extract an effective segmentation mask from the initial segmentation mask based on the depth image.
[0013] According to the method for grasping industrial parts of the loading and unloading production line provided by the embodiment of the present application, extracting an effective segmentation mask from the initial segmentation mask based on the depth image includes:
[0014] Obtaining the depth value of each pixel point carried in the depth image;
[0015] Removing the initial segmentation mask corresponding to the depth value when it exceeds the preset effective range, and taking the remaining initial segmentation mask as the effective segmentation mask.
[0016] According to the method for grasping industrial parts of the loading and unloading production line provided by the embodiment of the present application, calculating the normal vector corresponding to the target point cloud data based on a statistical method includes:
[0017] Performing the following normal vector determination process for each target point cloud data:
[0018] Determining the best neighborhood radius corresponding to the current target point cloud data, obtaining the local neighborhood point set corresponding to the best neighborhood radius; performing a plane fitting operation on the target point cloud data in the local neighborhood point set to obtain a fitting plane; determining the normal vector corresponding to the fitting plane;
[0019] Wherein, the direction of the normal vector points outwards from the viewing point.
[0020] According to the method for grasping industrial parts of the loading and unloading production line provided by the embodiment of the present application, determining the best neighborhood radius corresponding to the current target point cloud data includes:
[0021] Based on a preset information entropy calculation formula, obtaining the information entropy corresponding to different neighborhood radii;
[0022] Wherein, the information entropy calculation formula includes:
[0023]
[0024] Wherein, S represents the information entropy corresponding to different neighborhood radii, p i represents the probability distribution of the i-th target point cloud data in the to-be-determined local neighborhood point set corresponding to the current neighborhood radius, and n represents the total number of target point cloud data in the to-be-determined local neighborhood point set;
[0025] Determining the neighborhood radius in the state of the smallest information entropy among the obtained multiple information entropies as the best neighborhood radius corresponding to the current target point cloud data.
[0026] According to the method for grasping industrial parts of the loading and unloading production line provided by the embodiment of the present application, performing a plane fitting operation on the target point cloud data in the local neighborhood point set to obtain a fitting plane includes:
[0027] Calculate the distance between each target point cloud data in the local neighborhood point set and the current target point cloud data;
[0028] Configure Gaussian weights for each target point cloud data based on the distance and the weight calculation formula;
[0029] Among them, the weight calculation formula includes:
[0030]
[0031] Among them, ω i represents the Gaussian weight corresponding to the i-th target point cloud data, d i represents the distance between the i-th target point cloud data and the current target point cloud data, and σ represents the standard deviation of the Gaussian function, which is obtained based on the optimal neighborhood radius;
[0032] Perform a plane fitting operation on the target point cloud data with configured Gaussian weights by using the least squares method to obtain the fitting plane.
[0033] According to the industrial part grasping method of the loading and unloading production line provided by the embodiment of the present application, adjusting the target angles of the joints of the robot based on the pose information includes:
[0034] Based on a preset information conversion relationship, convert the pose information to the corresponding target pose information of the robot;
[0035] Use the inverse kinematics algorithm to calculate the target pose information to obtain the target angles of the joints.
[0036] According to the industrial part grasping method of the loading and unloading production line provided by the embodiment of the present application, after obtaining the contour of the industrial part, it further includes:
[0037] Use an edge enhancement algorithm to enhance the contour.
[0038] The embodiment of the present application also provides an electronic device, including a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the program, it implements the steps of the industrial part grasping method of the loading and unloading production line as described in any one of the above.
[0039] The embodiment of the present application also provides a non-transitory computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, it implements the steps of the industrial part grasping method of the loading and unloading production line as described in any one of the above.
[0040] The grasping method, device and medium for industrial parts of the loading and unloading production line provided by the embodiments of the present application obtain an RGB image, a depth image and point cloud data including industrial parts based on the operation condition of the loading and unloading production line; obtain a target segmentation mask based on the RGB image and the depth image, and map the target segmentation mask to the RGB image to obtain the contour of the industrial part and its position in the RGB image. The present application can effectively obtain the two-dimensional features and three-dimensional features of industrial parts through the RGB image and the depth image, and can obtain the contour and position of industrial parts more accurately, providing an effective data basis for the determination of subsequent pose information; furthermore, obtain the target point cloud data corresponding to the industrial part based on the contour and position; calculate the normal vector corresponding to the target point cloud data by a statistical method, and determine the pose information of the industrial part based on the state and distribution of the normal vector; adjust the target angles of the joints of the robot based on the pose information to control the robot to grasp industrial parts. The present application accurately identifies industrial parts, effectively evaluates their pose information, and adjusts the joint angles of the robot based on the pose information to achieve accurate grasping of industrial parts. It can be seen that the present application can accurately grasp various types of industrial parts, improving the operation efficiency of the loading and unloading production line. BRIEF DESCRIPTION OF THE DRAWINGS
[0041] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the following drawings are some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.
[0042] Figure 1 is one of the flow charts of the grasping method for industrial parts of the loading and unloading production line provided by the embodiments of the present application;
[0043] Figure 2 is another flow chart of the grasping method for industrial parts of the loading and unloading production line provided by the embodiments of the present application;
[0044] Figure 3 is still another flow chart of the grasping method for industrial parts of the loading and unloading production line provided by the embodiments of the present application;
[0045] Figure 4 is yet another flow chart of the grasping method for industrial parts of the loading and unloading production line provided by the embodiments of the present application;
[0046] Figure 5 is still yet another flow chart of the grasping method for industrial parts of the loading and unloading production line provided by the embodiments of the present application;
[0047] Figure 6It is the sixth flowchart of the method for grasping industrial parts on the loading and unloading production line provided by the embodiments of the present application;
[0048] Figure 7 It is the seventh flowchart of the method for grasping industrial parts on the loading and unloading production line provided by the embodiments of the present application;
[0049] Figure 8 It is the eighth flowchart of the method for grasping industrial parts on the loading and unloading production line provided by the embodiments of the present application;
[0050] Figure 9 It is the structural schematic diagram of the electronic device provided by the embodiments of the present application. Detailed implementation manners
[0051] To make the objectives, technical solutions, and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Apparently, the described embodiments are some but not all of the embodiments of the present application. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present application without creative efforts shall fall within the scope of protection of the present invention.
[0052] The embodiments of the present application provide a method for grasping industrial parts on a loading and unloading production line. This method can be applied to intelligent terminals and also to servers. This application takes the example of this method being applied in a server for illustration. This is for illustrative purposes only and is not used to limit the protection scope of the present application. Some other descriptions in the embodiments are also for illustrative purposes and will not be repeated hereinafter. The specific implementation of this method is as Figure 1 shown:
[0053] Step 101: Based on the operation conditions of the loading and unloading production line, obtain the RGB image, depth image, and point cloud data including industrial parts.
[0054] Step 102: Obtain the target segmentation mask based on the RGB image and the depth image, so as to obtain the contour of the industrial part and its position in the RGB image through the target segmentation mask.
[0055] Step 103: Extract the target point cloud data corresponding to the contour and position from the point cloud data.
[0056] Step 104: Calculate the normal vector corresponding to the target point cloud data based on a statistical method, and determine the pose information of the industrial part based on the state and distribution of the normal vector.
[0057] Step 105: Adjust the target angles of the joints of the robot based on the pose information to control the robot to grasp the industrial part.
[0058] Among them, the depth image can reflect the three-dimensional structure of the scene by assigning a value representing the distance to each pixel point. The depth image focuses on information such as the position and shape of objects, rather than directly on color information.
[0059] Among them, the operating conditions include the current state of the loading and unloading production line and the loading and unloading requirements. For example, the current state is the operating state and the target type of industrial part. The target type of industrial part changes as the loading and unloading progresses and is determined according to the actual operation process. This application does not impose any restrictions on which specific part it is, and it can be determined based on the actual scenario.
[0060] Among them, the target type of industrial part can be obtained through a demand instruction sent by a third-party device or through user input. This application does not impose any restrictions on the specific acquisition method.
[0061] Specifically, an RGBD camera is used to collect the aligned RGB image, depth image, and point cloud data. In addition, the camera's internal parameters need to be obtained.
[0062] The method for grasping industrial parts on the loading and unloading production line provided by the embodiments of this application obtains an RGB image, a depth image, and point cloud data including industrial parts based on the operating conditions of the loading and unloading production line; obtains a target segmentation mask based on the RGB image and the depth image, and maps the target segmentation mask to the RGB image to obtain the contour of the industrial part and its position in the RGB image. This application can effectively obtain the two-dimensional and three-dimensional features of industrial parts through the RGB image and the depth image, and can more accurately obtain the contour and position of industrial parts, providing an effective data basis for the subsequent determination of pose information; furthermore, based on the contour and position, the target point cloud data corresponding to the industrial part is obtained; and the normal vector corresponding to the target point cloud data is calculated based on a statistical method, and the pose information of the industrial part is determined based on the state and distribution of the normal vector; the target angles of the respective joints of the robot are adjusted based on the pose information to control the robot to grasp industrial parts. This application accurately identifies industrial parts, effectively evaluates their pose information, and adjusts the angles of the respective joints of the robot based on the pose information to achieve accurate grasping of industrial parts. It can be seen that this application can accurately grasp various types of industrial parts, improving the operation efficiency of the loading and unloading production line.
[0063] In a specific embodiment, the specific implementation of obtaining the target segmentation mask based on the RGB image and the depth image is as follows Figure 2 :
[0064] Step 201: Input the RGB image into a pre-trained SAM model to obtain the initial segmentation mask output by the SAM model.
[0065] Step 202: Extract a valid segmentation mask from the initial segmentation mask based on the depth image.
[0066] Among them, obtaining a segmentation mask is a technique in computer vision for precisely separating objects in an image from the background. It achieves a fine-grained division of the image region by classifying and labeling each pixel. Each pixel point is assigned a label to indicate whether it belongs to the foreground, background, or different object categories. Such label information forms a two-dimensional matrix, namely the segmentation mask.
[0067] Among them, the SAM model belongs to a prompt-based model, which includes an image encoder, a prompt encoder, and a mask decoder. The input image is converted into a feature vector through the image encoder, the user prompt is converted into a prompt vector by the prompt encoder, and the mask decoder fuses the image features and prompt features to generate and output the segmentation mask. During the use of this application, the user prompt is the default prompt, that is, the entire image is boxed.
[0068] Moreover, the SAM model does not need to collect a large amount of data sets for specialized training for each task and can directly obtain the desired results, saving operation time.
[0069] Of course, this application can use a sample data set including industrial parts to fine-tune the SAM model to more quickly and accurately predict the segmentation mask.
[0070] This application can accurately identify variable industrial parts through the SAM model, providing an effective data basis for subsequent grasping.
[0071] In a specific embodiment, the specific implementation of extracting a valid segmentation mask from the initial segmentation mask based on the depth image can be seen in Figure 3 :
[0072] Step 301: Obtain the depth value of each pixel point carried in the depth image.
[0073] Step 302: Remove the initial segmentation mask corresponding to when the depth value exceeds the preset valid range, and use the remaining initial segmentation mask as the valid segmentation mask.
[0074] Specifically, this application removes image light interference and background debris based on the depth value to obtain a valid segmentation mask. It uses depth information to analyze the three-dimensional structure of the object to eliminate the deficiencies of two-dimensional features and ensure the accuracy of the valid segmentation mask.
[0075] In a specific embodiment, after obtaining the contour of the industrial part, the edge enhancement algorithm is used to enhance the contour.
[0076] Specifically, for example, classic edge detection algorithms such as Sobel and Canny are used to enhance physical edge features and optimize the initial segmentation mask. For example, the Sobel operator calculates the gradient amplitudes in the horizontal and vertical directions of the image to determine edge pixel points. Through morphological processing (erosion, dilation) and mask operations, the object contour is refined, the mask accuracy is improved, and the risk of missegmentation is reduced.
[0077] Specifically, the present application uses an edge enhancement algorithm to refine and highlight the edge features (corresponding to the contour) in the image, improve the clarity of the contour and the contrast with the background, and improve the accuracy of the effective segmentation mask.
[0078] In a specific embodiment, the specific implementation of calculating the normal vector corresponding to the target point cloud data based on a statistical method includes:
[0079] The following normal vector determination process is performed for each target point cloud data. For details, see Figure 4 :
[0080] Step 401: Determine the optimal neighborhood radius corresponding to the current target point cloud data, and obtain the local neighborhood point set corresponding to the optimal neighborhood radius.
[0081] Step 402: Perform a plane fitting operation on the target point cloud data in the local neighborhood point set to obtain a fitting plane.
[0082] Step 403: Determine the normal vector corresponding to the fitting plane.
[0083] Among them, the direction of the normal vector points outward from the viewing point.
[0084] Among them, the statistical method includes an algorithm obtained by improving the PCA algorithm (corresponding to the normal vector determination process) to obtain the normal vector using the improved PCA algorithm.
[0085] Specifically, due to the direction ambiguity of the normal vector, that is, the direction of the normal vector can be two opposite directions, it is necessary to orient the normal vector according to the viewing point, that is, make the direction of the normal vector point outward from the viewing point.
[0086] The viewing point position is determined in advance, the coordinate position of the current target point cloud data is obtained, and when the relative position relationship between the viewing point position and the coordinate position satisfies formula (1), it is determined that the direction of the normal vector points outward from the viewing point.
[0087] Among them, formula (1) includes:
[0088]
[0089] Among them, represents the normal vector, V p represents the viewing point position, P i represents the coordinate position of the i-th target point cloud data.
[0090] Among them, if the formula (1) is not satisfied, the direction of the normal vector can be flipped.
[0091] In a specific embodiment, for the specific implementation of determining the optimal neighborhood radius corresponding to the current target point cloud data, refer to Figure 5 :
[0092] Step 501: Based on a preset information entropy calculation formula, obtain the information entropy corresponding to different neighborhood radii.
[0093] Step 502: Determine the neighborhood radius in the state of the minimum information entropy among the obtained multiple information entropies as the optimal neighborhood radius corresponding to the current target point cloud data.
[0094] Among them, the information entropy calculation formula is shown in formula (2):
[0095]
[0096] Among them, S represents the information entropy corresponding to different neighborhood radii, and p i represents the probability distribution of the i-th target point cloud data in the to-be-determined local neighborhood point set corresponding to the current neighborhood radius, and n represents the total number of target point cloud data in the to-be-determined local neighborhood point set.
[0097] Specifically, the optimal neighborhood radius is determined based on the principle of minimum information entropy constraint. The information entropy in this application is used to measure the distribution of the to-be-determined local neighborhood point set. The smaller the information entropy, the more concentrated the point set distribution.
[0098] In a specific embodiment, for the specific implementation of performing a plane fitting operation on the target point cloud data in the local neighborhood point set, refer to Figure 6 :
[0099] Step 601: Calculate the distance between each target point cloud data in the local neighborhood point set and the current target point cloud data.
[0100] Step 602: Configure Gaussian weights for each target point cloud data based on the distance and weight calculation formula.
[0101] Step 603: Use the least squares method to perform a plane fitting operation on the target point cloud data with Gaussian weights configured, and obtain a fitting plane.
[0102] Among them, the weight calculation formula is shown in formula (3):
[0103]
[0104] Among them, ω i represents the Gaussian weight corresponding to the i-th target point cloud data, and di represents the distance between the i-th target point cloud data and the current target point cloud data, and σ represents the standard deviation of the Gaussian function, which is used to control the attenuation speed of the Gaussian weight and is obtained based on the optimal neighborhood radius.
[0105] Based on the difference in the spatial distribution of the local point cloud, that is, the closer the points are, the closer the geometric features they represent are to the sampling points, this application uses an improved PCA method to estimate the normal vector, improving the accuracy of the normal vector estimation.
[0106] In a specific embodiment, for the specific implementation of determining the pose information of the industrial part based on the state and distribution of the normal vector, refer to Figure 7 :
[0107] Step 701, obtain the standard pose information corresponding to the recognized industrial part.
[0108] Step 702, determine the direction of the normal vector based on the state of the normal vector.
[0109] Step 703, compare the deviation between the standard direction and the direction of the normal vector to obtain the direction and tilt angle of the industrial part, and determine the direction, tilt angle, and position of the industrial part as the pose information.
[0110] Among them, the state of the normal vector includes the direction of the normal vector arrow, and it is required that the direction view of the normal vector faces outward.
[0111] Among them, the standard pose information includes the standard direction of the standard normal vector and the angle between the standard direction and the preset reference standard direction.
[0112] This application provides an effective data basis for the planning of the grasping path of the robotic arm by determining the pose information of the industrial part.
[0113] In a specific embodiment, for the specific implementation of adjusting the target angles of the joints of the robot based on the pose information, refer to Figure 8 :
[0114] Step 801, based on the preset information conversion relationship, convert the pose information to the corresponding target pose information of the robot.
[0115] Step 802, use the inverse kinematics algorithm to calculate the target pose information to obtain the target angles of the joints.
[0116] Specifically, through the hand-eye calibration relationship between the RGBD camera and the grasping structure of the robot (for example, the robotic arm), the pose information of the industrial part in the camera coordinate system is converted to the base coordinate system of the robotic arm, and the inverse kinematics algorithm is used to solve to determine the target angles that each joint needs to reach, realizing the precise movement of the end effector of the robotic arm point-to-point to the target position and performing the grasping of the industrial part.
[0117] In a specific embodiment, the specific implementation of the hand-eye calibration process includes:
[0118] First, image acquisition is performed: Images are taken at different positions and postures within the working space of the robotic arm using a checkerboard calibration board, and the joint angles and position information of the robotic arm corresponding to each image are recorded. Then, image preprocessing is carried out: Sub-pixel corner detection, denoising processing, and quality assessment are performed on the captured checkerboard images to ensure the high reliability of the image data used for solving the transformation relationship. Finally, the transformation relationship between coordinate systems is solved: Based on the Tsai-Lenz algorithm and the singular value decomposition (SVD) optimization method, the transformation relationship between the camera coordinate system and the end effector coordinate system of the robotic arm is solved, and the object pose information is accurately transformed to the robotic arm base coordinate system.
[0119] Next, the present application will be specifically described based on the actual application scenario of the loading and unloading production line:
[0120] Specifically, it mainly includes:
[0121] First, data acquisition and preprocessing operations.
[0122] By interacting with the production line through signals, the current state and loading and unloading requirements of the production line are obtained.
[0123] An RGBD camera is used to collect RGB images, depth images, and point cloud data, and at the same time, the camera internal parameters are obtained to ensure the accuracy and integrity of the image data, providing a basis for subsequent recognition and positioning.
[0124] Second, target recognition and positioning operations.
[0125] The RGB image is input into the pre-trained SAM model to generate an initial segmentation mask. By combining the depth image to remove the invalid parts, a valid segmentation mask is obtained to accurately locate the position and contour of the workpiece.
[0126] An edge enhancement algorithm is used to strengthen the workpiece contour, improve the accuracy of the segmentation mask, and further optimize the target recognition effect.
[0127] Third, point cloud data extraction and processing operations.
[0128] According to the enhanced contour and position information, the target point cloud data corresponding to the workpiece is extracted from the point cloud data to obtain detailed three-dimensional structure information.
[0129] The extracted point cloud data is transmitted to the production line control system for overall production scheduling and optimization.
[0130] Specifically, point cloud data is the data basis for accurately determining the pose information of industrial parts and can also provide the data basis for accurate part information for production scheduling. Through point cloud data, the processing quality of industrial parts can be evaluated, and for industrial parts with quality problems, rework or adjustment can be arranged in a timely manner. In addition, the position and state of industrial parts during the loading and unloading process can be obtained through point cloud data. The production line control system can control the production progress based on the position and state during the loading and unloading process, and take effective adjustment measures in a timely manner for abnormal progress to avoid the entire production process. Furthermore, production scheduling can be optimized through point cloud data, an optimal part handling path can be planned, the idle running time and energy consumption of the robot can be reduced, and production efficiency can be improved.
[0131] Fourth, pose estimation operation.
[0132] Calculate the normal vector corresponding to the target point cloud data based on a statistical method, and determine the pose information of the industrial part based on the state and distribution of the normal vector;
[0133] For each target point cloud data, determine the optimal neighborhood radius, obtain the local neighborhood point set, determine the optimal neighborhood radius based on the principle of minimum information entropy, use Gaussian weights and the least squares method for plane fitting, calculate the normal vector, and determine the workpiece pose, including the direction and tilt angle.
[0134] Fifth, grasping and loading / unloading planning operation.
[0135] Based on the workpiece pose information, calculate the target angles of each joint of the robot through a preset information conversion relationship and inverse kinematics algorithm, and plan the grasping path to ensure that the robot can stably and accurately grasp the workpiece.
[0136] When planning the loading / unloading path, comprehensively consider the production line layout and the requirements of the loading / unloading process, use the collision detection algorithm to generate a collision-free loading / unloading path to ensure the smooth progress of the loading / unloading process.
[0137] During the planning process, the loading / unloading system communicates with the production line control system in real time, and dynamically adjusts the grasping and handling strategies according to the current state and requirements of the production line. This real-time communication runs through the entire loading / unloading process to ensure the close linkage between the loading / unloading process and the production line.
[0138] Sixth, loading / unloading execution and monitoring operation.
[0139] The robot grasps the workpiece according to the planned path and transports it to the designated position of the processing equipment to ensure the accurate placement of the workpiece and wait for the processing completion signal. After processing is completed, the robot grasps the workpiece again and transports it to the finished product collection area to achieve the efficient flow of the workpiece.
[0140] Throughout the process, parameters such as torque and position during loading and unloading are monitored in real time to ensure the operation quality.
[0141] Specifically, by monitoring the torque, it is ensured that the force exerted by the robot is within a safe range, preventing excessive torque from damaging the workpiece and avoiding the robot's own overload; by monitoring the torque and position parameters, it is detected whether the grasping is stable and whether there is a position deviation, avoiding collision problems caused by position deviation; by monitoring the parameter changes in real time, it helps to detect potential faults or abnormal situations in a timely manner, and an alarm can be issued and corresponding measures can be taken in a timely manner.
[0142] Specifically, through visual inspection or force sensor feedback, it is verified whether the workpiece is correctly grasped and placed. If successful, the next process is continued; if failed, the reasons are analyzed and the loading and unloading operation is performed again, forming a closed-loop control to improve the success rate of loading and unloading.
[0143] Throughout the execution process, continuous communication with the production line control system is carried out to real-time feedback the working state of the robot and the processing progress of the workpiece, so that the production line can be coordinated and optimized as a whole.
[0144] This application accurately solves the conversion relationship between the camera and the robotic arm, constructs an accurate mapping of the two coordinate systems, and ensures the accurate positioning and grasping of the end effector of the robotic arm.
[0145] In order to better adapt to complex and changeable industrial scenarios, this application improves the robot's recognition and grasping capabilities of industrial parts by integrating deep learning and geometric features, and realizes the grasping of industrial parts with multiple shapes, types, and sizes. And through the SAM pre-trained model, industrial parts of different sizes, shapes, and materials are segmented, the segmentation mask is further optimized by combining the edge enhancement algorithm and depth information, and the normal vector of the point cloud within the segmentation mask is estimated and the grasping posture is estimated by the improved PCA method, so as to realize the grasping of industrial parts by the robot. It can be seen that this application integrates bimodal information and deep learning, and fully considers the spatial characteristics of the local point cloud, and can be applied to complex industrial scenarios to realize the grasping of variable industrial parts.
[0146] Figure 9 An example of the physical structure diagram of an electronic device is shown as Figure 9 shown. The electronic device may include: a processor 901, a communication interface 902, a memory 903, and a communication bus 904. Among them, the processor 901, the communication interface 902, and the memory 903 complete mutual communication through the communication bus 904. The processor 901 can call the logical instructions in the memory 903 to execute the method for grasping industrial parts of the loading and unloading production line.
[0147] In addition, when the logical instructions in the above-mentioned memory 903 are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes: various media such as USB flash drives, mobile hard disks, read-only memories (ROM, Read-Only Memory), random access memories (RAM, Random Access Memory), magnetic disks, or optical discs that can store program codes.
[0148] On the other hand, the present invention also provides a computer program product. The computer program product includes a computer program stored on a non-transitory computer-readable storage medium. The computer program includes program instructions. When the program instructions are executed by a computer, the computer can execute the grasping method of industrial parts on the loading and unloading production line provided by the above-mentioned various methods.
[0149] On yet another aspect, the present invention also provides a non-transitory computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, it is configured to execute the grasping method of industrial parts on the loading and unloading production line provided by the above-mentioned various embodiments.
[0150] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separated. The components shown as units may or may not be physical units, that is, they may be located in one place, or may be distributed to multiple network units. One can select some or all of the modules according to actual needs to achieve the purpose of the solution of this embodiment. Those of ordinary skill in the art can understand and implement it without creative efforts.
[0151] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.
[0152] Finally, it should be noted that the above is only the preferred implementation of the present application, and the present application is not limited to the above embodiments. It is understood that other improvements and changes directly derived or associated by those skilled in the art without departing from the spirit and concept of the present application should be considered to be included in the scope of protection of the present application.
Claims
1. A grasping method for industrial parts of a loading and unloading production line, characterized in that, The method includes: Based on the operation of the loading and unloading production line, obtaining an RGB image, a depth image, and point cloud data including industrial parts; Based on the RGB image and the depth image, obtaining a target segmentation mask, and obtaining the contour of the industrial part and its position in the RGB image through the target segmentation mask; Extracting target point cloud data corresponding to the contour and the position from the point cloud data; Calculating the normal vector corresponding to the target point cloud data based on a statistical method, and determining the pose information of the industrial part based on the state and distribution of the normal vector; Based on the pose information, adjusting the target angles of the joints of the robot to control the robot to grasp the industrial part.
2. The grasping method of industrial parts of the loading and unloading production line according to claim 1, characterized in that, Obtaining a target segmentation mask based on the RGB image and the depth image, including: Inputting the RGB image into a pre-trained SAM model to obtain an initial segmentation mask output by the SAM model; Extracting an effective segmentation mask from the initial segmentation mask based on the depth image.
3. The grasping method of industrial parts of the loading and unloading production line according to claim 2, characterized in that, Extracting an effective segmentation mask from the initial segmentation mask based on the depth image, including: Obtaining the depth value of each pixel point carried in the depth image; Removing the initial segmentation mask corresponding to the depth value when it exceeds the preset effective range, and taking the remaining initial segmentation mask as the effective segmentation mask.
4. The grasping method of industrial parts of the loading and unloading production line according to any one of claims 1-3, characterized in that, Calculating the normal vector corresponding to the target point cloud data based on a statistical method, including: Performing the following normal vector determination process for each target point cloud data: Determining the best neighborhood radius corresponding to the current target point cloud data, obtaining the local neighborhood point set corresponding to the best neighborhood radius; performing a plane fitting operation on the target point cloud data within the local neighborhood point set to obtain a fitting plane; determining the normal vector corresponding to the fitting plane; Wherein, the direction of the normal vector is outward from the viewing point.
5. The gripping method of industrial parts of the loading and unloading production line according to claim 4, characterized in that, Determining the best neighborhood radius corresponding to the current target point cloud data, including: Based on a preset information entropy calculation formula, obtaining the information entropy corresponding to different neighborhood radii; Wherein, the information entropy calculation formula includes: Among them, S represents the information entropy corresponding to different neighborhood radii, and p i represents the probability distribution of the i-th target point cloud data in the to-be-determined local neighborhood point set corresponding to the current neighborhood radius, and n represents the total number of target point cloud data in the to-be-determined local neighborhood point set; Determining the neighborhood radius in the state of the minimum information entropy among the obtained multiple information entropies as the best neighborhood radius corresponding to the current target point cloud data.
6. The grasping method of industrial parts of the loading and unloading production line according to claim 4, characterized in that Performing a plane fitting operation on the target point cloud data within the local neighborhood point set to obtain a fitting plane, including: Calculating the distance between each target point cloud data within the local neighborhood point set and the current target point cloud data; Configuring Gaussian weights for each target point cloud data based on the distance and a weight calculation formula; Wherein, the weight calculation formula includes: where, ω i represents the Gaussian weight corresponding to the i-th target point cloud data, d i represents the distance between the i-th target point cloud data and the current target point cloud data, and σ represents the standard deviation of the Gaussian function, which is obtained based on the optimal neighborhood radius; Using the least squares method to perform a plane fitting operation on the target point cloud data with configured Gaussian weights to obtain the fitting plane.
7. The grasping method of industrial parts of the loading and unloading production line according to any one of claims 1-3, characterized in that, Adjusting the target angles of the joints of the robot based on the pose information, including: Based on a preset information conversion relationship, converting the pose information to the corresponding target pose information of the robot; Using an inverse kinematics algorithm to calculate the target pose information to obtain the target angles of the joints.
8. The grasping method of industrial parts of the loading and unloading production line according to any one of claims 1-3, characterized in that, After obtaining the contour of the industrial part, it further includes: Enhancing the contour using an edge enhancement algorithm.
9. An electronic device, comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, When the processor executes the program, the steps of the method for grasping industrial parts of the loading and unloading production line according to any one of claims 1 to 8 are implemented.
10. A non-transitory computer-readable storage medium storing a computer program thereon, characterized in that, When the computer program is executed by the processor, the steps of the method for grasping industrial parts of the loading and unloading production line according to any one of claims 1 to 8 are implemented.