Steel spline grabbing method and system

Abstract

The invention discloses a steel spline grabbing method and system, and the method comprises the steps: carrying out the color space conversion of a designated region of an original image of a steel spline, and outputting a designated color channel gray image; finding a target position in the specified color channel grayscale image through a target detection algorithm, and carrying out target classification; detecting and segmenting a target classification result by adopting an image segmentation model to obtain a steel spline image; performing BLOB analysis on the steel spline image to obtain steel spline camera coordinates; the steel spline camera coordinates are calibrated and converted to obtain mechanical arm world coordinates of the large spline grabbing robot; and the steel sample strip is grabbed according to the world coordinates of the mechanical arm. According to the invention, a robot automatic sampling assembly line system in a steel production and manufacturing process can realize intelligent grabbing of steel splines, so that the working efficiency is improved, and the labor cost is saved.

Description

Technical Field

[0001] The present invention belongs to the technical field of intelligent manufacturing, and particularly relates to a method and system for grasping steel splines. Background Art

[0002] The automatic sampling pipeline system consists of parts such as a large spline handling robot, a spline buffer station, a feeding mechanism, a cutting device, a scrap steel separation conveyor belt, a sample number marking device, a small sample bar code placing robot, and a spline logistics box.

[0003] In the actual steel production process, the existing steel spline grasping link has a low degree of intelligence, resulting in low work efficiency and being prone to errors. Summary of the Invention

[0004] The purpose of the present invention is to overcome the defects of the prior art. The present invention provides a method and system for grasping steel splines, which can more intelligently realize the grasping of steel splines through a calibration method, improving work efficiency.

[0005] To achieve the expected effect, the present invention adopts the following technical solutions:

[0006] The present invention discloses a method for grasping steel splines, including:

[0007] After performing color space conversion on a specified area of the original steel spline image, output a grayscale image of a specified color channel;

[0008] Find the target position in the grayscale image of the specified color channel through a target detection algorithm and perform target classification;

[0009] Use an image segmentation model to detect and segment the result of the target classification to obtain a steel spline image;

[0010] Perform BLOB analysis on the steel spline image to obtain the camera coordinates of the steel spline;

[0011] Calibrate and convert the camera coordinates of the steel spline to obtain the robotic arm world coordinates of the large spline handling robot;

[0012] Grasp the steel spline according to the robotic arm world coordinates.

[0013] Further, the original steel spline image is taken from any one of a camera memory card, a local memory, and a software development kit.

[0014] Further, the content of the color space conversion is at least one of grayscale, HSV, HSI, and YUV spaces.

[0015] Further, the target detection algorithm is any one of the YOLO series and the RCNN series.

[0016] Further, the image segmentation model is any one of SegNet, DeepLab, Mask R-CNN, U-Net, and Gated SCNN.

[0017] Further, the detecting and segmenting the foreground target from the result of the target classification by using the image segmentation model includes: using a position correction tool to correct the target motion offset and assist in precise positioning for the region of interest in the target classification result, and then using the image segmentation model to detect and segment the region of interest to obtain the foreground target.

[0018] Further, the calibrating and converting the steel spline camera coordinates to the robotic arm world coordinates of the large spline gripper includes: selecting N points on the steel spline image for calibration to obtain the steel spline camera coordinates, and converting the camera coordinates of the N points to the robotic arm world coordinates of the large spline gripper, where N is a positive integer greater than or equal to 4.

[0019] Further, the grasping of the steel spline according to the robotic arm world coordinate system includes: analyzing and calculating the center point coordinates and the rotation angle of the steel spline according to the robotic arm world coordinate system, and then grasping the steel spline according to the center point coordinates and the rotation angle of the steel spline.

[0020] The present invention also discloses a steel spline grasping system, including:

[0021] An acquisition module, configured to acquire the original image of the steel spline;

[0022] A grasping module, configured to grasp the steel spline according to the method described in any one of the above.

[0023] The present invention also discloses a computer storage medium, on which computer program instructions are stored, and when the computer program instructions are executed by a processor, the method described in any one of the above is implemented.

[0024] Compared with the prior art, the beneficial effects of the present invention are: The present invention provides a steel spline grasping method and system, which can enable the robotic automatic sampling pipeline system in the steel production and manufacturing process to achieve intelligent grasping of steel splines, improve work efficiency, and save labor costs. Description of the Drawings

[0025] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for the description of the embodiments or the prior art. Obviously, the following-described drawings are only some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0026] Figure 1 It is a schematic diagram of the application scenario of a steel spline grasping system provided by an embodiment of the present invention.

[0027] Figure 2 It is a flowchart of a method for grasping steel splines provided by an embodiment of the present invention.

[0028] Figure 3 It is a flowchart of another method for grasping steel splines provided by an embodiment of the present invention. Detailed implementation manners

[0029] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, rather than all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts belong to the scope of protection of the present invention.

[0030] The present invention is based on a robot automatic sampling pipeline system in the steel production and manufacturing process. The system includes a large spline robot and a small spline robot. The robotic arm of the large spline robot uses a method for grasping steel splines of the present invention to grab the large splines on the spline conveyor in the spline pit to the large spline conveyor line through an electromagnet. The robotic arm of the small spline robot uses a method and system for positioning and packing small steel splines to place the cut small splines into the corresponding spline boxes according to certain rules. The robotic arms of the large spline robot and the small spline robot both use machine vision guidance for operation.

[0031] In addition, the large spline that has been moved into position needs to be transported forward by the feeding mechanism to the laser marking station for sample number marking. The laser marking position is within a square area (marking area 100*100mm) near one end on the front or bottom surface of the spline. The marking form is white primer and black font, and a QR code is marked simultaneously. The content of the marking is data sent from the superior system, and the marking content is stored. After the marking is completed, the feeding mechanism continues to feed the material to the hydraulic shear station and feeds the material according to the size data sent by the automatic sampling pipeline system. The feeding mechanism is linked with the hydraulic shear to cut out small splines according to the agreed size. After the cut small splines come out of the hydraulic shear, when the sensor on the conveyor belt detects that the small splines reach the preset position, the small spline robot takes out the small splines from the small spline conveyor belt and puts them into the corresponding spline boxes. The spline boxes include a sample delivery box, a retained sample box, and a slow cooling box. The spline boxes are fixed in position by using the angle steel at the bottom of the box corresponding to the ground slot, and at the same time, it is ensured that the spline boxes are within the grasping range of the robotic arm of the small spline robot. After sequentially and circularly completing the cutting of the entire large spline, in the case of longer waste materials, the hydraulic shear also cuts the waste materials so that the length of the waste materials is less than or equal to the preset value (for example, 1.2m). The remaining waste materials are conveyed by the conveyor belt to the waste material box located at the head of the conveyor belt and automatically fall into the waste material box according to gravity. The waste material box is a pit below the ground, and a waste material box is placed inside. The distance from the bottom of the waste material box to the elevation of the conveyor belt should be greater than the length of the waste materials to ensure that the waste materials can be completely put into the waste material box.

[0032] The splines in the spline boxes may be stacked regularly or irregularly.

[0033] When stacked regularly, the splines are placed in a grid pattern in the spline boxes and are laid flat from bottom to top.

[0034] The slow-cooled splines are placed disorderly in the slow-cooling sample hopper. When the placement reaches a certain value, the entire slow-cooling sample hopper is lifted and transported into the slow-cooling box.

[0035] See Figures 1 to 3 , the present invention discloses a method for grasping steel splines, including:

[0036] Step 1: After performing color space conversion on a specified area of the original image of the steel spline, output a grayscale image of the specified color channel;

[0037] In a preferred embodiment, the original image of the steel spline is taken from any one of a camera memory card, a local memory, and a software development kit.

[0038] Exemplarily, a camera is installed on the robotic arm of the large spline grasping robot to perform real-time image capture to obtain the original image of the steel spline.

[0039] In a preferred embodiment, the content of the color space conversion is at least one of grayscale, HSV, HSI, and YUV spaces.

[0040] HSV (Hue, Saturation, Value) is a color space created based on the intuitive characteristics of colors, also known as the Hexcone Model. The HSV color model refers to a visible photon set in the three-dimensional color space of H, S, and V, which contains all the colors in a certain color domain.

[0041] The HSI 〔Hue-Saturation-Intensity (Lightness), HSI or HSL〕 color model describes the color characteristics with three parameters of H, S, and I. Among them, H defines the frequency of the color, called hue; S represents the depth of the color, called saturation; I represents intensity or brightness. In the double hexagonal pyramid representation of the HSI color model, I is the intensity axis, and the angular range of the hue H is [0, 2π]. Among them, the angle of pure red is 0, the angle of pure green is 2π / 3, and the angle of pure blue is 4π / 3.

[0042] YUV is a type of true-color color space. Proper nouns such as Y’UV, YUV, YCbCr, and YPbPr can all be called YUV, and there is overlap among them. "Y" represents luminance, and "U" and "V" represent chrominance and concentration.

[0043] Step 2: Use the object detection algorithm to find the object position in the grayscale image of the specified color channel and perform object classification.

[0044] Preferably, the object detection algorithm is any one of the YOLO series and the RCNN series.

[0045] Specifically, the object detection algorithm can be: 1. Traditional object detection algorithms: VJ, HOG, DPM Detector; 2. Deep learning two-stage object detection algorithms: RCNN, SPPNet, Fast RCNN, Faster RCNN; 3. Object detection Trick algorithms: FPN, Cascade RCNN; 4. Deep learning one-stage object detection algorithms: Yolo v1, v2, v3, v4, v5, X, SSD, RetinaNet; 5. Deep learning Anchor-free object detection algorithms: CornerNet, CenterNet, FCOS; 6. Transformer-based object detection algorithms: DETR.

[0046] Step 3: Use an image segmentation model to detect and segment the results of the target classification to obtain a steel spline image;

[0047] Further, the image segmentation model is any one of SegNet, DeepLab, Mask R-CNN, U-Net, and Gated SCNN.

[0048] Specifically, image segmentation refers to dividing an image into several non-overlapping regions according to features such as grayscale, color, spatial texture, and geometric shape, so that these features show consistency or similarity within the same region, while showing obvious differences between different regions. Simply put, it is to separate the target from the background in an image. In the present invention, a pre-trained image segmentation model is used, and then parameters are configured to detect foreground objects.

[0049] SegNet is an architecture based on a deep encoder and decoder, also known as semantic pixel segmentation. It includes performing low-dimensional encoding on the input image, and then using the direction invariance ability in the decoder to restore the image. A segmentation image is then generated at the decoder end.

[0050] DeepLab performs image segmentation while helping to control the extraction of image signals to reduce the number of samples and the amount of data that the network has to process, enabling multi-scale context feature learning to aggregate features from images of different scales. DeepLab uses ImageNet pre-trained ResNet for feature extraction. In addition, DeepLab uses dilated convolution instead of regular convolution, and different dilation rates for each convolution enable the ResNet block to capture multi-scale context information.

[0051] Mask R-CNN mainly consists of three parts: an input layer, a convolutional layer, and a pooling layer. The input layer converts the image into a vector form, the convolutional layer processes the vector, and the pooling layer is used to extract the features of the convolutional layer. In the pooling layer, the mask represents the weight of each channel, and the logits represent the weighted average of all channels.

[0052] U-net includes: an input layer, a core layer, and an output layer. The input layer converts the image into a vector form, the core layer converts the vector into an instance, and the output layer classifies the instance according to predefined labels.

[0053] Gated SCNN uses two streams of image classification and contour prediction in the network, and uses the gated idea, which greatly improves the efficiency and accuracy of image segmentation.

[0054] Further, the detection and segmentation of the foreground target by using the image segmentation model for the target classification result includes: using a position correction tool to correct the target motion offset and assist in precise positioning of the region of interest in the target classification result, and then using the image segmentation model to detect and segment the region of interest to obtain the foreground target.

[0055] Region of Interest: In machine vision and image processing, the area that needs to be processed outlined in the form of a square, circle, ellipse, irregular polygon, etc. from the processed image is called the Region of Interest ROI. Various operators and functions are commonly used in machine vision software such as Halcon, OpenCV, and Matlab to obtain the Region of Interest ROI and perform the next step of image processing.

[0056] Specifically, a benchmark for position offset is established according to the matching points and the angle of the matching box in the template matching result, and then the coordinate rotation offset of the ROI detection box is realized according to the relative position offset between the running point and the reference point in the feature matching result, that is, to make the ROI area able to follow the changes in the image angle and pixels.

[0057] Step Four: Perform BLOB analysis on the steel spline image to obtain the camera coordinates of the steel spline;

[0058] Specifically, Blob analysis in computer vision is to extract and label the connected regions of a binary image after separating the foreground or background. Each labeled Blob represents a foreground target, and then some related features of the Blob can be calculated. Its advantage is that through Blob extraction, information about the relevant region can be obtained.

[0059] Step Five: Calibrate and convert the camera coordinates of the steel spline to obtain the robotic arm world coordinates of the large spline gripper;

[0060] Further, the calibration and conversion of the camera coordinates of the steel spline to obtain the robotic arm world coordinates of the large spline gripper includes: selecting N points on the steel spline image for calibration to obtain the camera coordinates of the steel spline, and converting the camera coordinates of the N points to the robotic arm world coordinates of the large spline gripper, where N is a positive integer greater than or equal to 4.

[0061] In a preferred embodiment, the present invention uses the 9-point calibration method, that is, 9 points are selected for calibration. After measurement, the grasping accuracy in this way is the highest.

[0062] Step Six: Grasp the steel spline according to the robotic arm world coordinates.

[0063] Further, the grasping of the steel spline according to the robot arm world coordinate system includes: analyzing and calculating the central point coordinates and rotation angle of the steel spline according to the robot arm world coordinate system, and then grasping the steel spline according to the central point coordinates and rotation angle of the steel spline.

[0064] Preferably, the robot arm of the large spline grasping robot has 6 moving axes, namely the rotary axis, the vertical arm axis, the horizontal arm axis, the wrist axis, the wrist swing axis, and the wrist transmission axis. Among them,

[0065] The moving range of the rotary axis is [+180°, -180°], and the maximum speed is 75° / s;

[0066] The moving range of the vertical arm axis is [+85°, -60°], and the maximum speed is 50° / s;

[0067] The moving range of the horizontal arm axis is [+60°, -180°], and the maximum speed is 55° / s;

[0068] The moving range of the wrist axis is [+300°, -300°], and the maximum speed is 100° / s;

[0069] The moving range of the wrist swing axis is [+100°, -100°], and the maximum speed is 100° / s;

[0070] The moving range of the wrist transmission axis is [+360°, -360°], and the maximum speed is 160° / s.

[0071] Preferably, the automatic sequence of the large spline grasping robot includes:

[0072] Sequence 0: Stop state;

[0073] Sequence 1: The robot returns to zero;

[0074] Sequence 2: Standby state;

[0075] Sequence 3: The ground track extends;

[0076] Sequence 4: Take a photo of the sample tail 1;

[0077] Sequence 5: Take a photo of the sample tail 2;

[0078] Sequence 6: Absorb the spline;

[0079] Sequence 7: The ground track retracts;

[0080] Sequence 8: Cache the spline;

[0081] Sequence 9: Feed to the shearing line.

[0082] Preferably, the present invention further includes: after completing the automatic sequence of the large spline grabbing robot, entering the automatic sequence of the large spline conveying line, and the automatic sequence of the large spline conveying line includes:

[0083] Step 0: Shutdown state;

[0084] Step 1: Initialization;

[0085] Step 2: Hydraulic shear start;

[0086] Step 3: Feeding servo goes to standby position;

[0087] Step 4: Standby state;

[0088] Step 5: Servo goes to picking position;

[0089] Step 6: Electromagnet drops to pick up material;

[0090] Step 7: Quick positioning;

[0091] Step 8: Slow positioning;

[0092] Step 9: Go to the position of the spline head;

[0093] Step 10: Cut the spline head;

[0094] Step 11: Servo searches for the end of the plate;

[0095] Step 12: Remaining length judgment;

[0096] Step 13: Cut the standard spline;

[0097] Step 14: Cut the waste.

[0098] The present invention can enable the robot automatic sampling pipeline system in the steel production and manufacturing process to achieve intelligent grabbing of steel splines, improve work efficiency, and save labor costs.

[0099] In one embodiment, the present invention may include the following steps:

[0100] Step S102, identifying the target grabbing object from the monitoring information, wherein the above monitoring information is obtained by acquiring the real-time video of the target scene using a depth camera installed on the grabbing robot;

[0101] Step S104, determining the position coordinates of the above target grabbing object in the first space coordinate system of the above grabbing robot, wherein the above first space coordinate system is a coordinate system established with the base of the robotic arm of the above grabbing robot as the origin;

[0102] Step S106: Determine the target navigation point in the fourth space coordinate system based on the above position coordinates, where the fourth space coordinate system is a coordinate system established according to the above target scenario.

[0103] Step S108: Control the above-mentioned grasping robot to move to the above-mentioned target navigation point for grasping.

[0104] In the embodiment of the present invention, the execution subject of the autonomous movement method of the above-mentioned grasping robot provided in the above steps S102 to S108 is the above-mentioned grasping robot. The above-mentioned grasping robot continuously shoots the target scenario by using a depth camera in the autonomous cruising state. The data information captured is the above-mentioned monitoring information, and the target grasping object is identified from the monitoring information, and the position coordinates in the camera coordinate system are determined. Through continuous conversion of multiple preset conversion matrices, the position coordinates of the target grasping object in the target scenario coordinate system are finally obtained. The target navigation point is calculated based on the above position coordinates, so as to control the grasping robot to move to the above target navigation point for grasping operations.

[0105] It should be noted that the above-mentioned monitoring information is obtained by using a depth camera installed on the grasping robot to shoot the target scenario; the above-mentioned coordinate systems include: the robotic arm base coordinate system (the first space coordinate system), the camera coordinate system (the second space coordinate system), the robot coordinate system (the third space coordinate system), and the map coordinate system (the fourth space coordinate system). The above-mentioned first space coordinate system is a coordinate system established with the base of the robotic arm of the above-mentioned grasping robot as the origin; the above-mentioned second space coordinate system is a coordinate system established with the above-mentioned depth camera as the origin; the above-mentioned third space coordinate system is a coordinate system established with the central position of the above-mentioned grasping robot as the origin; the above-mentioned fourth space coordinate system is a coordinate system established according to the above-mentioned target scenario.

[0106] In the embodiment of the present invention, after the camera recognition is completed, the position of the grasping point in the camera coordinate system can be obtained. The grasping coordinate point in the camera coordinate system can be converted to the robotic arm base coordinate system through the calibration parameters between the camera and the robotic arm. The robotic arm is fixed on the robot, and the coordinates in the robotic arm base coordinate system can be converted to the robot coordinate system through position parameters. The robot coordinate system also has position coordinates on the map. Finally, the grasping point recognized by the camera can be converted to the map coordinate system; the navigation process is based on the map coordinate system, and the grasping process is based on the robotic arm base coordinate system.

[0107] It should also be noted that when performing position coordinate conversion between the above-mentioned multiple spatial coordinate systems, the position coordinates in the response coordinate system can be calculated according to a preset conversion matrix. The above-mentioned conversion matrix is determined according to the origin positions of the above-mentioned multiple coordinate systems and the installation positions of the various components of the grasping robot; by continuously converting the position of the target grasping object in the above-mentioned multiple spatial coordinates, the specific position in different spatial coordinate systems can be determined, which can make the autonomous movement and grasping accuracy of the grasping robot higher.

[0108] As an optional embodiment, a schematic diagram of the overall process of the grasping robot's autonomous movement and grasping. The robot enters the autonomous cruising state and continuously monitors the surrounding environmental information through a depth camera during the traveling process; if the robot does not recognize the target object, it continues to autonomously cruise; if the target object is recognized, it stops cruising, calculates the spatial position of the target object's grasping point in the base coordinate system of the robotic arm according to the camera recognition result. If this position is within the graspable range of the robot, grasping is performed. Otherwise, the position coordinates of the target object's grasping point in the robot coordinate system and the position coordinates in the target scene coordinate system are further calculated. A grasping range is determined with the position coordinates in the target scene coordinate system as the center and the arm span of the robotic arm as the radius. A plurality of pose points (i.e., initial grasping points) are evenly generated within the grasping range. At the same time, according to the obstacle information obtained by the depth camera, the non-navigable grasping points among the initial grasping points are removed, and the grasping point with the shortest path is selected from the remaining grasping points as the target navigation point, thereby generating a navigation trajectory and performing autonomous movement to approach the target object for grasping; during the entire grasping process, the robot can determine whether it has received a stop operation signal. If it has received the signal, the robot stops; if it has not received the stop signal, it continues to cruise or grasp.

[0109] It should be noted that the above-mentioned positioning of the robot in the map before uses the amcl (adaptive Monte Carlo Localization) algorithm, and the path planning and robot navigation are completed using the move_base framework.

[0110] Through the embodiments of the present invention, without human participation, the robot will enter the patrol state after startup. Once the target object is recognized, the robot will calculate the relative position between itself and the target object and autonomously plan a path to approach the target object, and approach the target object for grasping. A series of navigation and positioning algorithms such as move_base and amcl can be combined to accurately control the robot to move near the target object and achieve the recognition, positioning, and grasping functions. This method requires no human intervention, has a fast response speed, and high control accuracy.

[0111] In an optional embodiment, before recognizing the target grasping object from the monitoring information, the above-mentioned method further includes: controlling the above-mentioned grasping robot to perform cruise monitoring processing on the above-mentioned target scene; receiving the above-mentioned monitoring information obtained during the above-mentioned cruise monitoring processing.

[0112] As an alternative embodiment, monitoring information is obtained in real time through a depth camera installed on the grasping robot. The above-mentioned depth camera can be installed at a position convenient for monitoring operations according to actual conditions, and a second space coordinate system is established at the installation position.

[0113] In an alternative embodiment, determining the position coordinates of the target grasping object in the first space coordinate system of the grasping robot includes: obtaining the second space coordinates of the target grasping object in the second space coordinate system of the depth camera, where the second space coordinate system is a coordinate system established with the depth camera as the origin; using a first preset transformation relation to convert the second space coordinates in the second space coordinate system into the first space coordinates in the first space coordinate system; and taking the first space coordinates as the position coordinates.

[0114] In the embodiment of the present invention, the spatial position of the target grasping object in the base coordinate system of the robotic arm (the first space coordinate system) is calculated according to the camera recognition result, that is, the position information in the camera coordinate system (the second space coordinate system) is converted into the spatial position in the first space coordinate system. The first preset transformation relation (conversion formula) is as follows:

[0115] PA = TAC PC;

[0116] Wherein, PA represents the coordinates of the grasping point in the base coordinate system of the robotic arm, PC represents the coordinates of the grasping point in the camera coordinate system, and these coordinates are obtained by measurement with the depth camera. TAC represents the transformation matrix from the camera coordinate system to the base coordinate system of the robotic arm, and this matrix can be obtained by means of hand-eye calibration.

[0117] Optionally, PA is used as the above-mentioned position coordinates for subsequent calculation and processing.

[0118] In an alternative embodiment, determining the target navigation point in the fourth space coordinate system based on the position coordinates includes: based on the position coordinates, obtaining the grasping distance, where the grasping distance is the distance between the position coordinates and the origin of the first space coordinate system; judging the magnitude relationship between the grasping distance and the arm span length of the robotic arm of the grasping robot; if the grasping distance is less than the arm span length, then taking the current position of the grasping robot as the target navigation point; if the grasping distance is greater than the arm span length, then determining the third space coordinates of the position coordinates in the third space coordinate system and the fourth space coordinates of the position coordinates in the fourth space coordinate system, and determining the target navigation point based on the third space coordinates and the fourth space coordinates.

[0119] As an alternative embodiment, based on the above position coordinates, the grasping distance is obtained. If the above grasping distance is less than the above arm reach, the current position of the above grasping robot is used as the above target navigation point for the grasping operation; if the above grasping distance is greater than the above arm reach, the third space coordinates of the above position coordinates in the third space coordinate system and the fourth space coordinates of the above position coordinates in the fourth space coordinate system are determined, and the above target navigation point is determined based on the above third space coordinates and the above fourth space coordinates

[0120] It should be noted that the above grasping distance is the distance between the above position coordinates and the origin of the above first space coordinate system, and the judgment method is as follows:

[0121] |PA| ≤ thr1;

[0122] Wherein, |PA| represents the distance from the grasping point to the origin of the robotic arm base coordinate system, and thr1 is the robotic arm grasping threshold, which can be determined according to the arm reach of the robotic arm. For example, the grasping threshold is set to 80% of the arm reach.

[0123] In an alternative embodiment, the determining the third space coordinates of the above position coordinates in the third space coordinate system and determining the above target navigation point in the fourth space coordinate system based on the above third space coordinates includes: converting the above first space coordinates in the above first space coordinate system into the third space coordinates in the above third space coordinate system by using a second preset transformation relation, wherein the above third space coordinate system is a coordinate system established with the central position of the above grasping robot as the origin; converting the above third space coordinates in the above third space coordinate system into the fourth space coordinates in the above fourth space coordinate system by using a third preset transformation relation, wherein the above fourth space coordinate system is a coordinate system established according to the above target scenario; determining the grasping range according to the above fourth space coordinates and the above arm reach; generating a plurality of first initial navigation pose points within the above grasping range, and determining the above target navigation point from the above plurality of first initial navigation pose points.

[0124] In the embodiments of the present invention, the third space coordinates PB of the grasping point of the target object in the robot coordinate system (third space coordinate system) and the fourth space coordinates PW in the map coordinate system (fourth space coordinate system) are calculated. The second preset transformation relation and the third preset transformation relation are as follows:

[0125] Wherein, TBA is the transformation matrix of the robotic arm base coordinate system in the robot coordinate system, which is determined by the robotic arm at the designed position of the robot and obtained during the structural design. TWB is the transformation matrix of the robot coordinate system relative to the map coordinate system, which is calculated by the robot odometer. Since the robot moves in a plane, the coordinate value of the fourth space coordinate PW on the z-axis is transformed to 0, which is equivalent to the projection of the grasping point on the map plane.

[0126] Optionally, after obtaining the above-mentioned fourth spatial coordinate PW, determine the grasping range according to the above-mentioned fourth spatial coordinate and the above-mentioned arm span length; generate a plurality of first initial navigation pose points within the above-mentioned grasping range, and determine the above-mentioned target navigation point from the above-mentioned plurality of first initial navigation pose points.

[0127] As an optional embodiment, n first initial navigation pose points (pose points) relative to the map coordinate system are uniformly generated within a circular ring with the obtained coordinate PW as the center, thr1 value as the outer circle radius, and thr2 value as the inner circle radius.

[0128] It should be noted that the value of the above-mentioned thr2 is less than the value of thr1, and it can be set according to the actual situation. The number n of pose points is a set value, and in actual application, it can be set according to the working environment. If the environment is relatively complex, then n can be set larger (n≥30). If the environment is not complex, then n can be set smaller (15≤n<30). The number of actually generated pose points is normally distributed around n.

[0129] As an optional embodiment, the sampling method of the pose point (the first initial navigation pose point) includes three methods: offset sampling, arc sampling, and star sampling.

[0130] Optionally, the offset sampling method includes: generating a sampling point array centered on PW, with the same offset Δ in the x and y directions of the array. The offset is related to thr1 and the number n of sampling points, and the calculation formula is as follows:

[0131] Among them, the symbol "[]" is the integer part symbol. To ensure the number of samples, a is the number of sampling rows and columns. After sampling, the sampling points outside the circular ring are removed to obtain the above-mentioned first initial navigation pose points.

[0132] Optionally, the arc sampling method includes: generating an arbitrary number of concentric circles within a circular ring centered on PW. The radii of the concentric circles form an arithmetic progression with thr1 and thr2. Sampling is performed at equal arcs on all concentric circles. Taking 3 concentric circles as an example, the arc relationship formula is deduced as follows:

[0133] Among them, L is the total arc length, and rad is the radian value between two sampling points on the circle.

[0134] Optionally, the star sampling method includes: sending rays outward with PW as the endpoint, sampling on the rays within the circular ring range, with the distance d between any two adjacent points on the same ray being equal, and the angle between any two adjacent rays being equal and divisible by 180. Setting the angle as θ, the relationship formula for the sampling point distance d is as follows:

[0135] Among them, N is the number of rays, and M is the number of sampling points on each ray.

[0136] It should be noted that the pose is a general term for the position and attitude of the robot, where the position is the position of the sampling point (coordinate values on the map plane), and the attitude is related to the installation angle of the camera on the robot. To make the camera face the target object, that is, the projection of the camera's z-axis on the map is parallel to the vector from the camera to the grasping point.

[0137] In an alternative embodiment, determining the target navigation point from the multiple first initial navigation pose points includes: identifying obstacle information from the monitoring information; removing the initial grasping points that cannot be navigated from the multiple first initial navigation pose points based on the obstacle information to obtain multiple second initial navigation pose points; calculating the travel distances between the multiple second initial navigation pose points and the current position of the grasping robot; and taking the second initial navigation pose point with the minimum travel distance as the target navigation point.

[0138] As an alternative embodiment, identify obstacle information from the monitoring information. According to the obstacle information, eliminate the pose points that cannot be navigated among the first initial navigation pose points. If there are navigable pose points, continue with the grasping operation; if there are no navigable points, it is determined that there are too many obstacles around the target object to perform the grasping, the target grasping object at the PW position is blocked, and no operation is performed on it later. At the same time, the grasping robot enters the autonomous cruising state.

[0139] Optionally, calculate the travel distances between the multiple second initial navigation pose points and the current position of the grasping robot; take the second initial navigation pose point with the minimum travel distance as the target navigation point, and use move_base to generate a navigation trajectory and perform autonomous movement to approach the target object for grasping.

[0140] In an alternative embodiment, after controlling the grasping robot to move to the target navigation point for grasping, the method further includes: determining whether a stop instruction is received; if no stop instruction is received, continue to control the grasping robot to perform cruise monitoring processing on the target scene.

[0141] As an alternative embodiment, after grasping, the robot determines whether a stop operation signal is received. If it is received, the robot stops; if the stop signal is not received, it continues with the cruise monitoring processing.

[0142] Through the above steps, the grasping robot can autonomously approach the grasping object based on visual recognition and position conversion, without human intervention, with a fast response speed and high control accuracy.

[0143] Based on the same inventive concept, the present invention also discloses a steel spline grasping system, including:

[0144] An acquisition module, configured to acquire the original image of the steel spline;

[0145] A grasping module for grasping steel splines according to the method described in any one of the above.

[0146] The system embodiments can be implemented corresponding to the foregoing method embodiments one by one, and will not be elaborated herein.

[0147] Based on the same inventive concept, the present invention also discloses a computer storage medium, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the method described in any one of the above is implemented.

[0148] Based on the same inventive concept, the present invention also discloses an electronic device, which may include: a processor, a communication interface, a memory, and a communication bus. Among them, the processor, the communication interface, and the memory complete communication with each other through the communication bus. The processor can call the logical instructions in the memory to execute a method for grasping steel splines, including:

[0149] After performing color space conversion on a specified area of the original steel spline image, output a grayscale image of a specified color channel;

[0150] Find the target position in the grayscale image of the specified color channel through a target detection algorithm and perform target classification;

[0151] Use an image segmentation model to detect and segment the result of the target classification to obtain a steel spline image;

[0152] Perform BLOB analysis on the steel spline image to obtain the camera coordinates of the steel spline;

[0153] Calibrate and convert the camera coordinates of the steel spline to obtain the robotic arm world coordinates of the large spline gripper;

[0154] Grasp the steel spline according to the robotic arm world coordinates.

[0155] In addition, when the logical instructions in the above-mentioned memory are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of this technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes: various media that can store program codes such as USB flash drives, mobile hard disks, read-only memories (ROM, Read-Only Memory), random access memories (RAM, Random Access Memory), magnetic disks, or optical discs.

[0156] On the other hand, an embodiment of the present invention also provides a computer program product. The computer program product includes a computer program stored on a non-transitory computer-readable storage medium. The computer program includes program instructions. When the program instructions are executed by a computer, the computer can execute a steel spline grasping method provided by each of the above method embodiments, including:

[0157] After performing color space conversion on a specified area of the original steel spline image, output a grayscale image of a specified color channel;

[0158] Find the target position in the grayscale image of the specified color channel through a target detection algorithm and perform target classification;

[0159] Use an image segmentation model to detect and segment the result of target classification to obtain a steel spline image;

[0160] Perform BLOB analysis on the steel spline image to obtain the camera coordinates of the steel spline;

[0161] Calibrate and convert the camera coordinates of the steel spline to obtain the robotic arm world coordinates of the large spline grasping robot;

[0162] Grasp the steel spline according to the robotic arm world coordinates.

[0163] It should be understood that although the steps in the flowchart of the accompanying drawings are shown sequentially in the direction of the arrows, these steps are not necessarily executed sequentially in the order indicated by the arrows. Unless there is a clear indication in this text, there is no strict order restriction for the execution of these steps, and they can be executed in other orders. Moreover, at least a part of the steps in the flowchart of the accompanying drawings may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed alternately or in turn with at least a part of other steps or sub-steps or stages of other steps.

[0164] The above are only the preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present invention shall be included within the protection scope of the present invention.

Claims

1. A method for grasping a steel spline, characterized in that, Including: After performing color space conversion on a specified area of the original image of the steel spline, output a grayscale image of the specified color channel; Find the target position in the grayscale image of the specified color channel through a target detection algorithm and perform target classification; Use an image segmentation model to detect and segment the result of target classification to obtain a steel spline image; Perform BLOB analysis on the steel spline image to obtain the camera coordinates of the steel spline; Calibrate and convert the camera coordinates of the steel spline to obtain the robotic arm world coordinates of the large spline gripper; Grasp the steel spline according to the robotic arm world coordinates; 2. The method for grasping a steel spline according to claim 1, wherein, The original image of the steel spline is taken from any one of the camera memory card, local memory, and software development kit; 3. The steel spline grabbing method according to claim 1, characterized in that, The content of the color space conversion is at least one of grayscale, HSV, HSI, and YUV spaces; 4. The steel spline grasping method according to claim 1, characterized in that, The target detection algorithm is any one of the YOLO series and the RCNN series; 5. The method for grasping a steel spline according to claim 1, wherein The image segmentation model is any one of SegNet, DeepLab, Mask R-CNN, U-Net, and Gated SCNN; 6. The steel spline grasping method according to claim 5, wherein The detecting and segmenting the result of target classification by using the image segmentation model to obtain the foreground target includes: using a position correction tool to correct the target motion offset and assist in precise positioning of the region of interest in the target classification result, and then using the image segmentation model to detect and segment the region of interest to obtain the foreground target; 7. The method for grasping a steel spline according to claim 1, characterized in that, The calibrating and converting the camera coordinates of the steel spline to obtain the robotic arm world coordinates of the large spline gripper includes: selecting N points on the steel spline image for calibration to obtain the camera coordinates of the steel spline, and converting the camera coordinates of the N points to the robotic arm world coordinates of the large spline gripper, where N is a positive integer greater than or equal to 4; 8. The steel spline grasping method according to claim 1, characterized in that, The grasping the steel spline according to the robotic arm world coordinate system includes: analyzing and calculating the center point coordinates and rotation angle of the steel spline according to the robotic arm world coordinate system, and then grasping the steel spline according to the center point coordinates and rotation angle of the steel spline; 9. A steel spline grasping system, characterized in that, Including: An acquisition module for acquiring the original image of the steel spline; A grasping module for grasping the steel spline according to the method described in any one of claims 1-8; 10. A computer storage medium, characterized in that, Computer program instructions are stored on the computer storage medium, and when the computer program instructions are executed by a processor, the method described in any one of claims 1-8 is implemented.

Images