Method, system and electronic equipment for identifying and grabbing scattered steel bars

Abstract

The embodiments of the present invention disclose a method, system and electronic device for identifying and grabbing scattered steel bars. The method includes: obtaining an image taken by an industrial camera overlooking the entire working area; identifying steel bars and a grabbing robot arm, calculating the distance between the steel bars and the grabbing robot arm, and moving the grabbing robot arm above the steel bars; obtaining an image taken by a depth camera set on the grabbing robot arm; screening complete and non-overlapping steel bar target feature images; calculating the average depth value of qualified steel bar targets in combination with the depth position information obtained by the depth camera; determining the steel bar closest to the depth camera, and determining the corresponding position and posture to obtain the steel bar information to be grabbed; according to the steel bar information to be grabbed, the robot arm grabs the corresponding steel bar and places it at the designated position. By implementing the method of the embodiments of the present invention, accurate target recognition and positioning can be achieved under complex backgrounds and in environments of different heights, improving detection speed and work efficiency, and reducing missed detection rate.

Description

Technical Field

[0001] The invention relates to a method for identifying and grasping steel bars, and more specifically to a method, system and electronic equipment for identifying and grasping scattered steel bars. Background Art

[0002] Traditional methods of grabbing rebar usually rely on manual operation, which is not only inefficient but also labor-intensive and difficult to meet the needs of large-scale production. In addition, handling and grabbing rebar is a heavy and dangerous task, and manual operation poses a great safety hazard. Grasping rebar through automated equipment can effectively reduce workers' contact with dangerous environments, thereby reducing the probability of safety accidents. Automated grabbing also greatly reduces workers' physical labor and improves work efficiency and working environment.

[0003] With the development of automation technology, the level of intelligence and precision of robotic arms, as important equipment in production lines, has been continuously improved; the progress of machine vision and sensor technology has provided the robotic arms with powerful perception capabilities, enabling them to accurately identify the position, shape and size of steel bars. However, the existing machine vision recognition systems are mostly aimed at single workpieces or neatly placed workpieces, while in actual construction sites, workpieces such as steel bars are usually placed in a scattered manner and may be mixed with other workpieces, which poses a higher challenge to the recognition ability of machine vision.

[0004] Therefore, it is necessary to design a new method to achieve accurate target recognition and positioning in complex backgrounds and at different heights, improve detection speed and work efficiency, and reduce missed detection rate. Summary of the invention

[0005] The purpose of the present invention is to overcome the defects of the prior art and provide a method, system and electronic equipment for identifying and grasping scattered steel bars.

[0006] To achieve the above object, the present invention adopts the following technical solution: a method for identifying and grabbing scattered steel bars, comprising:

[0007] Acquire an image taken by an industrial camera overlooking the entire work area to obtain an image of the work area;

[0008] Recognize the steel bars and the grabbing robot arm on the work area image, calculate the distance between the steel bars and the grabbing robot arm, and send the distance so that the grabbing robot arm moves above the steel bars;

[0009] Acquire an image taken by a depth camera disposed on a grasping robot arm to obtain an initial image;

[0010] Screening complete and non-overlapping steel bar target feature images in the initial image to obtain qualified steel bar target feature images;

[0011] Calculating an average depth value of a qualified steel bar target according to the qualified steel bar target feature image combined with the depth position information acquired by the depth camera;

[0012] Sorting according to the average depth value to determine the steel bar closest to the depth camera, and determining the corresponding position and posture to obtain the steel bar information to be grasped;

[0013] According to the information of the steel bars to be grasped, the robot arm grasps the corresponding steel bars and places them at the designated position.

[0014] A further technical solution is: the image of the working area is used to identify the steel bars and the grabbing mechanical arm, the distance between the steel bars and the grabbing mechanical arm is calculated, and the distance is sent so that the grabbing mechanical arm moves above the steel bars, including:

[0015] Using a convolutional neural network detection algorithm to identify steel bars in the image of the work area to obtain the position coordinates of the steel bars in the industrial camera view;

[0016] Determine the position coordinates of the grabbing robot arm in the industrial camera view through the QR code marked on the grabbing robot arm;

[0017] The distance between the steel bar and the grabbing robot arm is calculated according to the position coordinates of the steel bar in the industrial camera view and the position coordinates of the grabbing robot arm in the industrial camera view, so as to obtain the distance between the steel bar and the grabbing robot arm;

[0018] The distance is sent so that the grabbing robot moves over the steel bar.

[0019] A further technical solution is: screening the complete and non-overlapping steel bar target feature images in the initial image to obtain a qualified steel bar target feature image includes:

[0020] Extracting a steel bar target feature image in the initial image;

[0021] Preliminarily screening out incomplete and overlapping target feature images in the steel bar target feature image to obtain a preliminary qualified feature image;

[0022] Performing image preprocessing on the preliminary qualified feature image to obtain a preprocessing result;

[0023] Extracting edge contour information of the image from the preprocessing result, and calculating the length and area of ​​the contour feature;

[0024] Match the contour features with the workpiece template in the system template library for geometric features and calculate the matching degree;

[0025] Complete workpiece features are screened out according to the matching degree to obtain a qualified steel bar target feature image.

[0026] A further technical solution is: the preliminary screening of incomplete and overlapping target feature images in the steel bar target feature image to obtain a preliminary qualified feature image includes:

[0027] Searching for regional features in the steel bar target feature image that are similar to the given original working template through a correlation matching CC algorithm to obtain similarity matching features;

[0028] The features corresponding to the stacked and incomplete workpieces in the similarity matching features are filtered out to obtain a preliminary qualified feature image.

[0029] A further technical solution is: according to the information of the steel bar to be grabbed, the mechanical arm grabs the corresponding steel bar and places it at a designated position, including:

[0030] Receive construction drawings or parameters, construct a coordinate system, and calculate the placement position of the steel bars to be grabbed based on the coordinate system; wherein the coordinate system takes the upper left corner of the control platform as the origin, the ground plane as the horizontal plane, the x-axis is parallel to the gantry, the y-axis is perpendicular to the gantry, and the z-axis is perpendicular to the ground;

[0031] The steel bars to be grabbed are grabbed according to the information of the steel bars to be grabbed, and the steel bars to be grabbed are placed at a placement position.

[0032] A further technical solution is: receiving construction drawings or parameters, constructing a coordinate system, and calculating the placement of steel bars based on the coordinate system, including:

[0033] Receive construction drawings or parameters and extract the length, width, shape, spacing and quantity of steel bars to obtain steel bar related information;

[0034] The control unit analyzes the relevant information of the steel bars, constructs a coordinate system, and calculates the placement coordinates of each steel bar and the coordinates of the cross points based on the coordinate system;

[0035] The step of analyzing the steel bar related information and calculating the placement coordinates of each steel bar and the coordinates of the cross intersections includes:

[0036] Establish a coordinate system;

[0037] According to the length and spacing of the horizontal and vertical bars in the relevant information of the steel bars, the coordinates of the two ends of each horizontal and vertical bar are calculated to determine the position of the cross intersection.

[0038] A further technical solution is: grabbing the steel bar to be grabbed according to the steel bar information to be grabbed, and placing the steel bar to be grabbed at a placement position, including:

[0039] Calculating the three-dimensional coordinates of the grabbing point corresponding to the steel bar to be grabbed;

[0040] Adjusting the movement and rotation of the robot arm according to the three-dimensional coordinates of the grasping point to accurately grasp the steel bar to be grasped;

[0041] After the robot arm completes grabbing the steel bar to be grabbed, the steel bar to be grabbed is placed in a placement position.

[0042] A further technical solution is: the calculation of the three-dimensional coordinates of the grabbing point corresponding to the steel bar to be grabbed includes:

[0043] Setting the reference coordinates of the grasping subsystem, the coordinate system of the lower arm of the grasping robot arm, the coordinate system of the upper arm of the grasping robot arm, and the coordinate system of the first depth camera; wherein the first depth camera is arranged on the grasping robot arm; the grasping subsystem includes the grasping robot arm;

[0044] The grasping position in the coordinate system of the upper arm of the grasping robot is determined according to the transformation relationship between the reference coordinates of the grasping subsystem and the coordinate system of the lower arm of the grasping robot, the coordinate system of the upper arm of the grasping robot and the coordinate system of the lower arm of the grasping robot, and the coordinate system of the first depth camera and the coordinate system of the upper arm of the grasping robot to obtain the three-dimensional coordinates of the grasping point corresponding to the steel bar to be grasped.

[0045] The present invention also provides a scattered steel bar identification and grabbing system, comprising:

[0046] An industrial image acquisition unit, used to acquire an image taken by an industrial camera overlooking the entire working area to obtain an image of the working area;

[0047] A detection unit, used for identifying the steel bars and the grabbing mechanical arm in the working area image, calculating the distance between the steel bars and the grabbing mechanical arm, and sending the distance so that the grabbing mechanical arm moves above the steel bars;

[0048] An initial image acquisition unit, used to acquire an image taken by a depth camera arranged on the grasping mechanical arm to obtain an initial image;

[0049] A screening unit, used for screening complete and non-overlapping steel bar target feature images in the initial image to obtain qualified steel bar target feature images;

[0050] A depth value calculation unit, used to calculate the average depth value of the qualified steel bar target according to the qualified steel bar target feature image combined with the depth position information obtained by the depth camera;

[0051] A unit for determining information of steel bars to be grasped, used for sorting according to the average depth value to determine the steel bars closest to the depth camera, and determining the corresponding position and posture to obtain information of the steel bars to be grasped;

[0052] The grabbing unit is used to grab the corresponding steel bars by a mechanical arm according to the information of the steel bars to be grabbed, and place them at a designated position.

[0053] The present invention further provides an electronic device, comprising a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the above method when executing the computer program.

[0054] The beneficial effects of the present invention compared with the prior art are as follows: the present invention obtains the image of the working area through an industrial camera, calculates the distance between the steel bar and the robotic arm, moves the robotic arm above the steel bar, then uses a depth camera to shoot and screen qualified steel bar images, calculates the average depth value of the steel bar, and finally determines the nearest steel bar according to the depth value, and the robotic arm grabs and places it; it realizes accurate target recognition and positioning in complex backgrounds and at different heights, improves detection speed and work efficiency, and reduces missed detection rate.

[0055] The present invention is further described below in conjunction with the accompanying drawings and specific embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying any creative work.

[0057] Figure 1 A schematic diagram of a process flow of a method for identifying and grabbing scattered steel bars provided in an embodiment of the present invention;

[0058] Figure 2 A schematic diagram of a sub-process of a method for identifying and grabbing scattered steel bars provided in an embodiment of the present invention;

[0059] Figure 3 A schematic diagram of a sub-process of a method for identifying and grabbing scattered steel bars provided in an embodiment of the present invention;

[0060] Figure 4 A schematic diagram of a sub-process of a method for identifying and grabbing scattered steel bars provided in an embodiment of the present invention;

[0061] Figure 5 A schematic diagram of a sub-process of a method for identifying and grabbing scattered steel bars provided in an embodiment of the present invention;

[0062] Figure 6 A schematic diagram of a sub-process of a method for identifying and grabbing scattered steel bars provided in an embodiment of the present invention;

[0063] Figure 7 A schematic diagram of a sub-process of a method for identifying and grabbing scattered steel bars provided in an embodiment of the present invention;

[0064] Figure 8 A schematic diagram of a sub-process of a method for identifying and grabbing scattered steel bars provided in an embodiment of the present invention;

[0065] Fig. 9 A schematic diagram of a coordinate system provided by an embodiment of the present invention;

[0066] Fig.10 A schematic diagram of the reference coordinates of the grasping subsystem, the coordinate system of the lower arm of the grasping robot, the coordinate system of the upper arm of the grasping robot, and the coordinate system of the first depth camera provided in an embodiment of the present invention;

[0067] Fig.11 A schematic diagram of an operating area image provided by an embodiment of the present invention;

[0068] Fig.12 A schematic diagram of determining a qualified steel bar target feature image provided by an embodiment of the present invention;

[0069] Fig.13 A schematic block diagram of a system for identifying and grasping scattered steel bars provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0070] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0071] It should be understood that when used in this specification and the appended claims, the terms "include" and "comprises" indicate the presence of described features, integers, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or combinations thereof.

[0072] It should also be understood that the terms used in this specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. As used in the specification of the present invention and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.

[0073] It should be further understood that the term "and / or" used in the present description and the appended claims refers to and includes any and all possible combinations of one or more of the associated listed items.

[0074] The method for identifying and grabbing scattered steel bars mentioned in the present embodiment of the invention is applied to the control process of the fully automatic steel bar batch grabbing and tying system, wherein the fully automatic steel bar batch grabbing and tying system includes at least two groups of gantries, each of which is composed of a horizontal beam, a vertical beam and a moving device. The vertical beam moves up and down through a lifting device to adjust the operating height of the horizontal beam; the horizontal beam of the gantry is provided with a grabbing mechanical arm and a tying mechanical arm, the grabbing mechanical arm is used to grab the steel bars scattered on the side, and the tying mechanical arm is used to tie the cross points; the number of grabbing mechanical arms and tying mechanical arms depends on the specific situation, and there are at least 2 grabbing and 1 tying. That is, each gantry is equipped with at least 2 grabbing robotic arms and 1 tying robotic arm, and the specific number can be adjusted according to demand; each gantry has 4 grabbing robotic arms (No. 1, No. 2, No. 3, No. 4 from left to right) and 2 tying robotic arms (No. 1 and No. 2 from left to right); and the interval between the gantries is set to the length of half a steel bar to ensure that the robotic arms of the front and rear gantries can be accurately positioned.

[0075] In addition, the method corresponding to this embodiment is mainly used for steel bar positioning before steel bar grabbing and steel bar identification and positioning during steel bar grabbing, and the grabbing subsystem is used to grab steel bars, including a grabbing mechanical arm set on the horizontal reinforcement gantry and a grabbing mechanical arm set on the vertical reinforcement gantry; before executing steel bar grabbing and binding, the vertical reinforcement should be stacked on the side of the platform, perpendicular to the gantry; the horizontal reinforcement should be stacked in front of the platform, parallel to the gantry; when grabbing steel bars, whether it is horizontally grabbing horizontal reinforcement or vertical reinforcement, the specific position of the reinforcement is located by the grabbing sensing subsystem; the grabbing subsystem grabs the reinforcement through the feedback system; the grabbing subsystem places the reinforcement to the predetermined target position according to the placement position of the control unit. The grabbing subsystem can ensure that the vertical and horizontal reinforcements are stacked in the accurate position required by the design, thereby improving the construction accuracy; the systematic grabbing and placement process reduces manual intervention and speeds up the construction speed; the automated grabbing and placement process reduces human errors and improves the consistency and reliability of construction; the correct stacking position ensures the effective use of the platform space and improves the overall construction efficiency.

[0076] In the process of grabbing steel bars, a grabbing sensing subsystem is also required. The grabbing sensing subsystem includes a first plane camera arranged on the vertical steel bar gantry and the horizontal steel bar gantry and a first depth camera arranged on the grabbing mechanical arm.

[0077] Among them, the first plane camera is used to capture and identify the steel bars to be grasped, determine the position of the steel bars to be grasped, and send the position of the steel bars to be grasped to the grasping subsystem, so that the grasping subsystem moves to the position of the steel bars to be grasped; the first depth camera is used to analyze and determine the grasping point of the steel bars to be grasped when the grasping subsystem moves to the position of the steel bars to be grasped, and then determine the three-dimensional coordinates of the grasping point, and send it to the grasping subsystem, and when the grasping subsystem moves to the three-dimensional coordinates, the three-dimensional coordinates of the grasping point are fed back in real time until the grasping subsystem grasps the steel bars;

[0078] The three-dimensional coordinates of the grasping point are determined according to the transformation relationship between the reference coordinates of the grasping subsystem and the coordinate system of the lower arm of the grasping robot, the coordinate system of the upper arm of the grasping robot and the coordinate system of the lower arm of the grasping robot, and the coordinate system of the first depth camera and the coordinate system of the upper arm of the grasping robot. Among them, the reference coordinates of the grasping subsystem take the direction perpendicular to the chassis as the z-axis, and parallel to the chassis as the x and y axes; the coordinate system of the lower arm of the grasping robot takes the z-axis parallel to the lower arm, the x-axis parallel to the degree of freedom plane, and the y-axis perpendicular to the degree of freedom plane; the coordinate system of the upper arm of the grasping robot takes the z-axis parallel to the lower arm, the x-axis parallel to the degree of freedom plane, and the y-axis perpendicular to the degree of freedom plane; the coordinate system of the first depth camera takes the z-axis parallel to the lower arm, the x-axis parallel to the degree of freedom plane, and the y-axis perpendicular to the degree of freedom plane.

[0079] In this embodiment, the grasping perception subsystem first determines the distribution of steel bars and the grasping target through the first plane camera, and then notifies the grasping subsystem to reach the initial position. After reaching the initial position, the first depth camera accurately measures the relative position between the manipulator and the steel bar, and helps the grasping subsystem 50 to grasp the steel bar accurately through continuous feedback.

[0080] Specifically, the first plane camera first identifies and locates the position of the steel bar to ensure that the grasping subsystem can accurately reach the target area; the first depth camera further provides precise three-dimensional coordinates and grasping point information, significantly improving the accuracy of the grasping operation; the first depth camera provides real-time three-dimensional coordinate feedback, so that the grasping subsystem can dynamically adjust the grasping strategy to ensure high-precision operation in complex environments; the first plane camera helps to determine the approximate position of the steel bar, thereby reducing the time and movement path of the grasping subsystem when searching for the target; after the grasping subsystem approaches the target, the first depth camera further analyzes the specific grasping point, optimizes the grasping action path, and reduces the probability of failure caused by position error; the system realizes precise control between different devices through the conversion relationship of multiple coordinate systems. Accurate docking allows for flexible adjustment of position and posture in complex grasping tasks; the coordinate systems between the grasping robot and the depth camera are compatible, supporting complex spatial calculations and adjustments, allowing the system to adapt to various shapes and positions of steel bar grasping tasks; by automatically identifying and positioning steel bars, manual intervention and adjustments are reduced, the level of automation is improved, and work efficiency is improved; real-time feedback and precise grasping point positioning reduce the number of repeated adjustments and trial and error, saving time and resources; the real-time feedback provided by the first depth camera can help the grasping subsystem correct the grasping action in time, improving the stability and reliability of the system in actual operation; the collaborative work of the first plane camera and the depth camera ensures the system's comprehensive perception and precise control of the environment, reducing the occurrence of errors and failures.

[0081] The method of this embodiment combines convolutional neural network technology with depth information to achieve accurate target positioning in complex backgrounds. It optimizes the sorting and extraction of workpiece position features and further improves the extraction effect of feature information through template matching. This method can detect and grasp different workpieces, improve work efficiency, and replace manual operations.

[0082] Next, the technical solution of the embodiment of the present invention is clearly and completely described in conjunction with the drawings in the embodiment of the present invention.

[0083] Figure 1 Schematic diagram of the process of the method for identifying and grabbing scattered steel bars provided by an embodiment of the present invention. Figure 1 As shown, the method includes the following steps S110 to S170.

[0084] S110, acquiring an image taken by an industrial camera overlooking the entire working area to obtain an image of the working area.

[0085] In this embodiment, the work area image refers to an image that can present the entire work area.

[0086] In the construction site environment, the industrial camera is installed at the highest point of the entire working area, overlooking the entire grasping operation area. The industrial camera can find the position of the steel workpiece and the robotic arm in the two-dimensional plane. The workpieces are stacked on the ground, and the robotic arm is installed on the gantry and is movable. The robotic arm here is only the grasping robotic arm in the grasping subsystem.

[0087] S120, identifying the steel bars and the grabbing robot arm in the work area image, calculating the distance between the steel bars and the grabbing robot arm, and sending the distance so that the grabbing robot arm moves above the steel bars.

[0088] In this embodiment, in an industrial work scenario, a convolutional neural network detection algorithm is used to identify steel workpieces, and the position of the robotic arm is located through QR code recognition technology. Through these technologies, the specific positions of the workpiece and the robotic arm in the working environment are determined, and the distance between them is calculated. Based on this information, the motion control module of the robotic arm will move the robotic arm to the top of the workpiece for the next grabbing operation.

[0089] Specifically, the grabbing robot arm is set on the gantry, and the mobile subsystem includes two vertical rib gantries, one horizontal rib gantry and a mobile platform. The vertical rib gantry and the horizontal rib gantry are respectively connected to the mobile platform, that is, the movement of the grabbing robot arm is realized by the mobile subsystem, among which the vertical rib gantry is used to move the vertical ribs; the horizontal rib gantry is used to move the horizontal ribs.

[0090] In one embodiment, see Figure 2 The above-mentioned step S120 may include steps S121~S124.

[0091] S121. Use a convolutional neural network detection algorithm to identify steel bars in the image of the working area to obtain the position coordinates of the steel bars in the industrial camera view.

[0092] In this embodiment, the image of the work area captured by the industrial camera is analyzed by a convolutional neural network (CNN). CNN is a powerful tool in deep learning that can automatically extract features from images and classify them. In this step, the CNN model is trained to recognize the characteristics of the steel bars, thereby determining the precise location of the steel bars in the image. This process generates two-dimensional coordinate data of the steel bars, reflecting the location of the steel bars in the camera view.

[0093] Using CNN for steel bar identification is highly accurate and robust. It can handle a variety of complex background and lighting conditions, ensuring that steel bar identification can maintain high reliability in different environments. This not only improves positioning accuracy, but also reduces the need for manual intervention and speeds up the detection process.

[0094] S122. Determine the position coordinates of the grasping robot arm in the industrial camera view through the QR code marked on the grasping robot arm.

[0095] In this embodiment, a QR code is installed on the grabbing robot arm, and the QR code contains the position identification information of the robot arm. After the industrial camera captures the image of the QR code, the QR code information is read through image processing technology to obtain the position coordinates of the robot arm in the camera view. These coordinates represent the current position of the robot arm and are used for subsequent calculations and operations.

[0096] QR code is a low-cost and easy-to-implement location identification solution. It allows the system to obtain the position data of the robot arm in real time, ensuring that the position of the robot arm during operation can be accurately tracked. This method increases the flexibility and accuracy of the system while simplifying the process of detecting the robot arm's position.

[0097] S123. Calculate the distance between the steel bar and the grasping robot arm according to the position coordinates of the steel bar in the industrial camera view and the position coordinates of the grasping robot arm in the industrial camera view to obtain the distance between the steel bar and the grasping robot arm.

[0098] In this embodiment, the steel bar and robot arm position coordinates obtained from steps S121 and S122 are used to determine the actual distance between them through geometric calculation. Typically, such calculation includes using the Euclidean distance formula to convert the straight-line distance between two coordinate points into the actual distance. These distance data are used for subsequent action adjustments.

[0099] By precisely calculating the distance between the rebar and the robotic arm, the system can accurately determine the distance the robotic arm needs to move, which helps ensure that the movement of the robotic arm is precise and reduces grasping failures due to position errors, thereby improving the overall efficiency and reliability of the system.

[0100] S124: Send the distance so that the grabbing robot arm moves above the steel bar.

[0101] In this embodiment, the calculated distance information is transmitted to the control system of the robot arm, and the control system instructs the grabbing robot arm to perform appropriate movement operations based on the distance data, specifically, the slideway of the robot arm drives the grabbing robot arm to slide so that it moves above the steel bar. This process usually involves adjusting the movement path and position of the grabbing robot arm so that it is aligned with the position of the steel bar.

[0102] By sending precise distance data, the robotic arm can accurately position itself above the steel bars, thereby improving the accuracy of the grabbing operation. This not only optimizes the automated operation process, but also reduces production delays or errors caused by inaccurate positioning of the robotic arm. The automation and intelligence level of the system has been significantly improved, enhancing overall operation efficiency.

[0103] S130: Acquire an image taken by a depth camera disposed on the grasping robot arm to obtain an initial image.

[0104] In this embodiment, the depth camera refers to a first depth camera, and the initial image refers to a depth image captured by the robotic arm.

[0105] When the base of the robotic arm moves under the steel workpiece, the robotic arm and the depth camera are started to begin detection.

[0106] S140, screening complete and non-overlapping steel bar target feature images in the initial image to obtain qualified steel bar target feature images.

[0107] In this embodiment, a qualified steel bar target feature image refers to a steel bar target feature image that displays the steel bars completely and does not overlap with other steel bar images.

[0108] like Fig.11 As shown in the figure, the improved EffiNet-Yolo model and depth camera are used for workpiece detection. After image preprocessing, feature extraction and template matching optimization, the workpieces are accurately identified and screened. Then, the depth information of the workpieces is calculated to sort them, ensuring that the robotic arm can grab the nearest workpiece first.

[0109] This step removes non-compliant rebar images, ensuring that subsequent analysis and processing are based only on accurate rebar data; it improves the overall accuracy of the system, reduces the risk of misidentification and processing, and thus optimizes the rebar detection and processing process.

[0110] In one embodiment, see Figure 3 The above-mentioned step S140 may include steps S141~S146.

[0111] S141, extracting a steel bar target feature image in the initial image.

[0112] In this embodiment, feature extraction is performed through a trained improved EfficientNet-Yolo convolutional neural network model.

[0113] Specifically, the EfficientNetv2 backbone network is known for its efficient feature extraction capabilities and accuracy. Combined with Yolov8's advanced detection algorithm, it achieves fast and high-precision object detection. The training data set contains two thousand images of steel workpieces, 80% of which are used for training and 20% for testing. The detection accuracy is comparable to the original Yolov8s. This detection method can accurately locate the workpiece, and after detecting the target, it will crop the feature frame image for subsequent processing.

[0114] S142, preliminarily screening out incomplete and overlapping target feature images in the steel bar target feature image to obtain a preliminarily qualified feature image.

[0115] In this embodiment, the preliminary qualified feature image refers to the image remaining after the incomplete and overlapping target feature images in the steel bar target feature image are preliminarily screened out.

[0116] In one embodiment, see Figure 3 The above-mentioned step S142 may include steps S1421~S1422.

[0117] S1421. Searching for regional features in the steel bar target feature image that are similar to the given original working template through a correlation matching CC algorithm to obtain similarity matching features.

[0118] In this embodiment, the similarity matching features refer to features whose feature matching degree meets the requirements.

[0119] Specifically, the rebar target feature image is analyzed using a correlation matching (CC) algorithm to find regional features similar to a given original working template. This step identifies potential rebar target regions by comparing the similarity between the rebar image and the predefined template.

[0120] The correlation matching algorithm helps locate the areas in the image that match the expected features, improving the recognition accuracy of the rebar targets. This allows subsequent processing to be optimized more accurately for the relevant features.

[0121] S1422: Filter out features corresponding to stacked and incomplete workpieces from the similarity matching features to obtain a preliminary qualified feature image.

[0122] In this embodiment, among the similarity matching features screened out in step S1421, those workpiece features that are overlapped or incomplete are further filtered out, so that only those image features that are complete and non-overlapping are retained.

[0123] By eliminating features that do not meet the requirements (such as stacking and incompleteness), the selected image features are ensured to be more representative and accurate. This process reduces the interference of false detection and repeated features and improves the quality of the final qualified image features.

[0124] The preliminary screening process improves the integrity and accuracy of the steel bar target feature image through precise correlation matching and feature filtering, laying a solid foundation for subsequent processing and ensuring the reliability and accuracy of the final detection results.

[0125] S143, performing image preprocessing on the preliminary qualified feature image to obtain a preprocessing result.

[0126] In this embodiment, the preprocessing result refers to an image formed after the preliminary qualified image is subjected to image preprocessing.

[0127] Specifically, the following preprocessing operations are performed on the preliminary qualified feature images to improve image quality and processing efficiency:

[0128] Weighted Average Grayscale: Converts the image to grayscale and applies a weighted average method to reduce the influence of color information and emphasize important image details.

[0129] Bilateral filtering: Use bilateral filtering to remove noise from an image while preserving edge information. This filtering method is effective in smoothing out noise in an image without blurring important details in the image.

[0130] Non-local Means Denoising: Apply the Non-local Means Denoising algorithm to further clean up the noise in the image. This algorithm provides more effective noise suppression by considering pixel values ​​in similar regions of the image.

[0131] Global histogram brightness equalization: Perform brightness equalization on the image to enhance the contrast of the image. This process makes the details in the image more obvious and easier to identify by adjusting the image brightness distribution.

[0132] Edge sharpening: sharpen the edges of an image to improve its clarity and make the edges more obvious, which is helpful for subsequent image analysis.

[0133] Through these preprocessing steps, the number of image channels can be effectively reduced, the processing speed can be accelerated, noise can be removed, the contrast can be enhanced, and the overall clarity of the image can be improved, thereby providing higher quality input for subsequent image processing and analysis.

[0134] S144, extracting edge contour information of the image from the preprocessing result, and calculating the length and area of ​​the contour feature.

[0135] In this embodiment, the Canny edge detection algorithm is used to extract edge contour information in the image, so that the contours of the objects in the image can be clearly identified. Subsequently, the length and area of ​​the extracted contours are calculated to obtain detailed contour features.

[0136] S145, geometrically matching the contour feature with the workpiece template in the system template library, and calculating the matching degree.

[0137] In this embodiment, these contour features are compared with the template features in the workpiece template library stored in the input system acquired by the camera, and a matching algorithm based on geometric features is used for comparison to calculate the matching degree.

[0138] S146. Filter out complete workpiece features according to the matching degree to obtain a qualified steel bar target feature image.

[0139] In this embodiment, the matching degree of each workpiece is calculated according to the matching results, and workpiece features with low matching degree and incompleteness are screened out.

[0140] In the case of unsuccessful matching, the camera angle needs to be adjusted and the detection needs to be repeated. The optimization goal of this detection algorithm is to identify and find those relatively complete workpiece features, so as to obtain the most similar, non-overlapping and relatively complete workpiece information to the workpiece in the template library. This process ensures high-precision workpiece recognition and matching.

[0141] S150, calculating an average depth value of the qualified steel bar target according to the qualified steel bar target feature image in combination with the depth position information acquired by the depth camera.

[0142] In this embodiment, a depth camera is used to capture target feature images of qualified steel bars, and depth position information is obtained therefrom; an average depth value of the steel bar target is calculated, and this value reflects the overall distance of the steel bar relative to the depth camera.

[0143] S160 , sorting according to the average depth value to determine the steel bar closest to the depth camera, and determining the corresponding position and posture to obtain information of the steel bar to be captured.

[0144] In this embodiment, the steel bar information to be captured refers to the position and posture of the steel bar closest to the depth camera, and the position refers to the coordinates relative to the coordinate system of the depth camera. This coordinate system defines the position and posture of the steel bar in the camera's perspective, including the depth, horizontal and vertical position of the steel bar in front of the camera, and the angle and orientation of the steel bar.

[0145] First, sort the average depth values ​​of all qualified steel bars. Based on the sorting results, determine the steel bar closest to the depth camera. Accurately measure and determine the specific position and posture of the nearest steel bar, including its spatial coordinates and angle. Generate detailed data on the steel bar to be captured based on this information.

[0146] In this embodiment, by calculating and sorting the depth values, the steel bars closest to the camera can be accurately identified, reducing the risk of misgrasping or missing grasping; clarifying the position and posture of the target steel bars helps the grasping system to perform operations quickly and accurately, thereby improving work efficiency; by locating the closest steel bars, interference from other steel bars can be reduced, thereby reducing the complexity of positioning and grasping.

[0147] S170. According to the information of the steel bars to be grabbed, the robot arm grabs the corresponding steel bars and places them at a designated position.

[0148] In this embodiment, the robot grabs the steel bar and moves to the specified position. The robot grabs the steel bar and moves to the specified position. The robot grabs the steel bar according to the position and posture of the workpiece. The robot force feedback determines whether the workpiece is successfully grabbed. If the grabbing is not completed, the camera angle needs to be adjusted to re-detect. After the grabbing is successful, the robot moves to the specified position to complete the grabbing task of the workpiece.

[0149] When the robot arm completes the grabbing action and sends it to the given position, it returns to the initial state; it repeats the process action again to complete the identification and grabbing of other steel bars until all workpieces are grabbed and the operation ends when no workpieces are detected.

[0150] In this embodiment, before grabbing the steel bars, in addition to determining the position of the steel bars, it is also necessary to know the placement position of the steel bars.

[0151] In one embodiment, see Figure 4 The above-mentioned step S170 may include steps S171~S172.

[0152] S171. Receive construction drawings or parameters, construct a coordinate system, and calculate the placement position of the steel bars to be grabbed based on the coordinate system; wherein the coordinate system takes the upper left corner of the control platform as the origin, the ground plane as the horizontal plane, the x-axis is parallel to the gantry, the y-axis is perpendicular to the gantry, and the z-axis is perpendicular to the ground.

[0153] In one embodiment, see Figure 5 , the above-mentioned step S171 may include steps S1711~S1712.

[0154] S1711. Receive construction drawings or parameters, and extract the length, width, shape, placement spacing and quantity of steel bars to obtain steel bar related information;

[0155] S1712, the control unit parses the steel bar related information, constructs a coordinate system, and calculates the placement coordinates of each steel bar and the coordinates of the cross intersection based on the coordinate system;

[0156] The step of analyzing the steel bar related information and calculating the placement coordinates of each steel bar and the coordinates of the cross intersections includes:

[0157] Establish a coordinate system;

[0158] According to the length and spacing of the horizontal and vertical bars in the relevant information of the steel bars, the coordinates of the two ends of each horizontal and vertical bar are calculated to determine the position of the cross intersection.

[0159] Specifically, Fig. 9As shown in the figure, a coordinate system is established with the ground plane as the horizontal plane, the direction parallel to the gantry as the x-axis, the direction perpendicular to the gantry as the y-axis, and the direction perpendicular to the ground as the z-axis. The steel bars include vertically arranged vertical bars and horizontally arranged horizontal bars; the placement positions include the placement positions of each horizontal bar, the placement positions of the vertical bars, and the coordinates of the cross points; the steel bar to be grasped is a horizontal bar or a vertical bar.

[0160] Assume that both the horizontal bars and the vertical bars are straight steel bars with a long shape. The length of the horizontal bar is L1, the length of the vertical bar is L2, and the height of the fixed groove is h above the ground plane. The number of horizontal bars to be placed is N1, then the spacing is L2 / N1, the number of vertical bars to be placed is N2, and the spacing is x. Then the coordinates of the left end and the right end of each horizontal bar are (0, n*L2 / N1, h), (L1, n*L2 / N1, h), where 0 ≤ n < N1, and n represents the number of the horizontal bar.

[0161] The coordinates of the cross point refer to the coordinates of the intersection point of the cross knot.

[0162] The coordinates of the left end and the right end of each vertical bar are (m*x, 0, h), (m*x, L2, h), where 0 ≤ m < N2, and m represents the number of the vertical bar.

[0163] The coordinates of the cross knot are (m*x, n*L2 / N1, h), where 0 < n < N1, 0 < m < N2, and m and n respectively represent the IDs of the horizontal bar and the vertical bar.

[0164] In another embodiment, when the steel bars are of other shapes, the corresponding coordinates can also be obtained in the above manner.

[0165] S172. Grasp the steel bar to be grasped according to the information of the steel bar to be grasped, and place the steel bar to be grasped at the placement position.

[0166] In this embodiment, after determining the placement position, it is necessary to convert the information of the steel bar to be grasped into coordinates in the coordinate system with the grasping robotic arm as the reference system, that is, coordinate transformation is required.

[0167] In one embodiment, please refer to Figure 7 , the above step S172 may include steps S1721 to S1723.

[0168] S1721. Calculate the three-dimensional coordinates of the grasping point corresponding to the steel bar to be grasped.

[0169] In one embodiment, please refer to Figure 8 , the above step S1721 may include steps S17211 to S17212.

[0170] S17211. Set the reference coordinates of the grasping subsystem, the coordinate system of the lower arm of the grasping robot, the coordinate system of the upper arm of the grasping robot, and the coordinate system of the first depth camera; wherein the first depth camera is set on the grasping robot.

[0171] In this embodiment, the first depth camera refers to a depth camera.

[0172] S17212. Determine the grasping position in the coordinate system of the upper arm of the grasping robot according to the transformation relationship between the reference coordinates of the grasping subsystem and the coordinate system of the lower arm of the grasping robot, the coordinate system of the upper arm of the grasping robot and the coordinate system of the lower arm of the grasping robot, and the coordinate system of the first depth camera and the coordinate system of the upper arm of the grasping robot, so as to obtain the three-dimensional coordinates of the grasping point corresponding to the steel bar to be grasped.

[0173] Among them, the reference coordinates of the grasping subsystem have the z-axis perpendicular to the chassis, and the x and y axes parallel to the chassis; the coordinate system of the lower arm of the grasping robot has the z-axis parallel to the lower arm, the x-axis parallel to the degree of freedom plane, and the y-axis perpendicular to the degree of freedom plane; the coordinate system of the upper arm of the grasping robot has the z-axis parallel to the lower arm, the x-axis parallel to the degree of freedom plane, and the y-axis perpendicular to the degree of freedom plane; the coordinate system of the first depth camera has the z-axis parallel to the lower arm, the x-axis parallel to the degree of freedom plane, and the y-axis perpendicular to the degree of freedom plane.

[0174] The grasping perception subsystem is composed of a first depth camera and a first plane camera on the grasping robot arm.

[0175] In this embodiment, the grasping perception subsystem first determines the distribution of steel bars and the grasping target through the first plane camera, and then notifies the grasping subsystem to reach the initial position. After reaching the initial position, the first depth camera accurately measures the relative position between the manipulator and the steel bar, and helps the grasping subsystem 50 to grasp the steel bar accurately through continuous feedback.

[0176] Specifically, the first plane camera first identifies and locates the position of the steel bar to ensure that the grasping subsystem 50 can accurately reach the target area; the first depth camera further provides accurate three-dimensional coordinates and grasping point information, significantly improving the accuracy of the grasping operation; the first depth camera provides real-time three-dimensional coordinate feedback, so that the grasping subsystem can dynamically adjust the grasping strategy to ensure high-precision operation in complex environments; the first plane camera helps determine the approximate position of the steel bar, thereby reducing the time and movement path of the grasping subsystem when searching for the target; after the grasping subsystem approaches the target, the first depth camera further analyzes the specific grasping point, optimizes the grasping action path, and reduces the probability of failure caused by position error; the system realizes the conversion relationship between different devices through multiple coordinate systems. Precise docking allows flexible adjustment of position and posture in complex grasping tasks; the coordinate systems between the grasping robot and the depth camera are compatible, supporting complex spatial calculations and adjustments, allowing the system to adapt to various steel bar grasping tasks in different shapes and positions; automatic identification and positioning of steel bars reduces manual intervention and adjustments, improves the level of automation, and improves work efficiency; real-time feedback and precise grasping point positioning reduce the number of repeated adjustments and trial and error, saving time and resources; the real-time feedback provided by the first depth camera can help the grasping subsystem correct the grasping action in time, improving the stability and reliability of the system in actual operation. The coordinated work of the first plane camera and the depth camera ensures the system's comprehensive perception and precise control of the environment, reducing the occurrence of errors and failures.

[0177] Specifically, the three-dimensional coordinates of the grasping point are determined by the following process:

[0178] Set the reference coordinates of the grasping subsystem 50, the coordinate system of the grasping robot lower arm, the coordinate system of the grasping robot upper arm, and the coordinate system of the first depth camera, such as Fig.10As shown, the ellipse in the figure is the base of the robot arm, the rectangle is the upper and lower arms of the robot arm, the upper one is the lower arm, and the square below the upper arm represents the first depth camera; let O be the reference coordinate of the grasping subsystem 50, with the z axis perpendicular to the chassis, and the x and y axes parallel to the chassis; O1 is the coordinate system of the lower arm of the grasping robot arm, with the z axis parallel to the lower arm, the x axis parallel to the degree of freedom plane, and the y axis perpendicular to the degree of freedom plane; O2 is the coordinate system of the upper arm of the grasping robot arm, with the z axis parallel to the lower arm, the x axis parallel to the degree of freedom plane, and the y axis perpendicular to the degree of freedom plane; O3 is the coordinate system of the first depth camera, and the xyz three-axis direction is consistent with the grasping robot The coordinate system O2 of the upper arm of the grasping robot arm is consistent; the transformation from the coordinate system O1 to the coordinate system O includes translation along the z axis and rotation around the three axes xyz. According to the known rotation angle and translation, let the rotation matrix be R1, the translation amount be t1, and let the coordinates of the point under the coordinate system O1 be a1=(x1, y1, z1), then the coordinate transformation formula from O1 to O can be obtained; according to the transformation relationship between the reference coordinates of the grasping subsystem 50 and the coordinate system of the lower arm of the grasping robot arm, the coordinate system of the upper arm of the grasping robot arm and the coordinate system of the lower arm of the grasping robot arm, and the coordinate system of the first depth camera and the coordinate system of the upper arm of the grasping robot arm, the grasping position coordinates in the coordinate system of the grasping robot arm are determined. Specifically, the coordinate transformation formula from O1 to O is: , where a is the vector representation of the coordinates in the O coordinate system, a = (x, y, z); similarly, the coordinate transformation from O2 to O1 can be obtained as ; Where R2 is the rotation matrix, t2 is the translation vector, and a2=(x2, y2, z2) is the point coordinate in the coordinate system O2; Similarly, the coordinate transformation from O3 to O2 can be obtained as , where e=[1, 0; 0, 1], t3 is the translation vector, and a3=(x3, y3, z3) is the coordinate in the coordinate system O3. Combining the above transformation formula, the three-dimensional coordinates of the grasping point can be calculated.

[0179] S1722, adjusting the movement and rotation of the robot arm according to the three-dimensional coordinates of the grasping point to accurately grasp the steel bar to be grasped;

[0180] S1723. After the robot arm finishes grabbing the steel bar to be grabbed, it places the steel bar to be grabbed in a placement position.

[0181] In this embodiment, when the transverse reinforcement is just being grabbed, the slot needs to be configured. Specifically, the fixed slot subsystem provides a fixed position for the transverse reinforcement to ensure that its spacing and posture meet the design requirements. Specifically, the fixed slot system adjusts the fixed spacing and posture of the transverse reinforcement according to the parameters calculated by the control unit. During the binding process, first, a group of transverse reinforcements are fixed at a predetermined spacing and posture according to the design requirements. After all the transverse reinforcements are fixed in the slots, the vertical reinforcements are placed in sequence for binding.

[0182] The slots provided by the fixed slot subsystem can ensure that the transverse reinforcement is accurately fixed according to the designed spacing and posture, which contributes to the stability of the structure; the process of fixing the transverse reinforcement first and then tying the vertical reinforcement optimizes the construction steps and reduces the adjustment time; the fixed slot subsystem ensures the consistency of each construction and reduces human errors; the position of the fixed slot is automatically adjusted according to the design parameters, which improves the automation level and accuracy of the construction.

[0183] First, identify and grab the corresponding horizontal reinforcement in sequence according to the above steps, and then identify and grab the vertical reinforcement according to the above steps. When a vertical reinforcement is placed at the placement coordinate position of the vertical reinforcement in the calculated placement position, the vertical reinforcement and the horizontal reinforcement form N1-1 cross intersections, that is, cross knots. According to the positions of the pre-calculated cross intersections, the second plane camera is used to quickly locate the cross points.

[0184] After each vertical reinforcement is placed, a cross knot needs to be tied. At this time, the tying sensing subsystem and the tying subsystem are needed to complete. The tying sensing subsystem is used to identify the cross knot and determine the position of the cross knot after each vertical reinforcement is placed; the tying subsystem is used to tie the steel bars at the cross knot position according to the cross knot position and the cross intersection coordinates determined by the tying sensing subsystem.

[0185] Specifically, the tying subsystem includes a tying mechanical arm arranged on a transverse reinforcement gantry.

[0186] In this embodiment, after a vertical reinforcement is placed at a determined position, the binding sensing subsystem provides the binding subsystem with the position of the cross knot that needs to be tied; the binding subsystem continuously adjusts its position through sensing feedback and reaches the binding position, and binds the steel bars at the cross knot.

[0187] In one embodiment, the above-mentioned lashing sensing subsystem includes a second plane camera disposed on the cross-bar gantry and a second depth camera disposed on the lashing mechanical arm;

[0188] wherein the second depth camera is used to identify and preliminarily locate the position of the cross knot, and send the preliminarily determined position to the lashing subsystem, so that the lashing subsystem moves to the cross knot according to the preliminarily determined position and the coordinates of the cross intersection;

[0189] The second depth camera is used to analyze the position of the cross knot to the local coordinates of the depth camera when the binding subsystem moves to the preliminarily determined position to determine the position of the cross knot, and send the position of the cross knot to the binding subsystem, so that the binding subsystem moves to the position of the cross knot and binds the steel bars at the position of the cross knot.

[0190] In this embodiment, the tying sensing subsystem first determines the cross knot position of the steel bar through the second plane camera, and then notifies the tying subsystem to reach the preliminary position. The tying subsystem 70 will combine the cross intersection coordinates and the second plane camera to determine the cross knot position of the steel bar and move to the approximate position of the cross knot. After arriving, the second depth camera will accurately measure the distance information between the robot arm and the cross knot, and help the tying subsystem to accurately tie the steel bar through continuous feedback.

[0191] The joint use of the second plane camera and the second depth camera ensures the accurate recognition and preliminary positioning of the cross knot position, reducing positioning errors; after the second depth camera preliminarily determines the position, it automatically transmits the information to the tying subsystem, reducing manual intervention and improving operational efficiency; when the tying subsystem moves to the preliminarily determined position, the second depth camera further analyzes the specific position of the cross knot to achieve high-precision tying operations; through the dual steps of preliminary positioning and precise analysis, tying errors caused by positioning errors are reduced, and the overall tying quality and reliability are improved.

[0192] When tying, when the second depth camera on the tying robot captures the cross knot, the tying perception subsystem identifies and calculates the position of the cross knot in the local coordinate system of the second depth camera, and then notifies the tying robot to approach the cross knot. Through continuous feedback and adjustment, until the tying robot reaches the appropriate position, the tying of the cross knot is finally completed.

[0193] After all the vertical bars are placed and all the cross ties are tied, the mobile subsystem can be used to move the tied steel bars.

[0194] The above-mentioned scattered steel bar identification and grasping method obtains the image of the working area through the industrial camera, calculates the distance between the steel bar and the robotic arm, moves the robotic arm above the steel bar, then uses the depth camera to shoot and screen qualified steel bar images, calculates the average depth value of the steel bar, and finally, determines the nearest steel bar according to the depth value, and the robotic arm grasps and places it; it realizes accurate target identification and positioning in complex backgrounds and at different heights, improves detection speed and work efficiency, and reduces missed detection rate.

[0195] Fig.13 is a schematic block diagram of a scattered steel bar identification and grabbing system 300 provided in an embodiment of the present invention. Fig.13 As shown, corresponding to the above scattered steel bar identification and grasping method, the present invention also provides a scattered steel bar identification and grasping system 300. The scattered steel bar identification and grasping system 300 includes a unit for executing the above scattered steel bar identification and grasping method. Fig.13The scattered steel bar identification and grasping system 300 includes an industrial image acquisition unit 301, a detection unit 302, an initial image acquisition unit 303, a screening unit 304, a depth value calculation unit 305, a steel bar information determination unit 306 to be grasped, and a grasping unit 307.

[0196] The industrial image acquisition unit 301 is used to acquire an image taken by an industrial camera overlooking the entire working area to obtain an image of the working area; the detection unit 302 is used to identify the steel bars and the grabbing robot arm in the working area image, calculate the distance between the steel bars and the grabbing robot arm, and send the distance so that the grabbing robot arm moves above the steel bars; the initial image acquisition unit 303 is used to acquire an image taken by a depth camera set on the grabbing robot arm to obtain an initial image; the screening unit 304 is used to screen the complete and non-overlapping steel bar target feature images in the initial image to obtain a qualified steel bar target feature image; the depth value calculation unit 305 is used to calculate the average depth value of the qualified steel bar target based on the qualified steel bar target feature image combined with the depth position information obtained by the depth camera; the steel bar information determination unit 306 is used to sort according to the average depth value to determine the steel bar closest to the depth camera, and determine the corresponding position and posture to obtain the steel bar information to be grabbed; the grabbing unit 307 is used to grab the corresponding steel bar by the robot arm according to the steel bar information to be grabbed, and place it at a specified position.

[0197] In one embodiment, the above-mentioned grasping unit 307 includes the grasping subsystem, grasping sensing subsystem, etc. mentioned in the method embodiment.

[0198] In addition, the above-mentioned scattered steel bar identification and grasping system 300 also includes a moving subsystem, a binding sensing subsystem and a corresponding binding subsystem.

[0199] In one embodiment, the detection unit 302 includes:

[0200] A steel bar identification subunit is used to use a convolutional neural network detection algorithm to identify the steel bars in the image of the working area to obtain the position coordinates of the steel bars in the industrial camera view; a coordinate determination subunit is used to determine the position coordinates of the grasping robot arm in the industrial camera view by using a QR code marked on the grasping robot arm; a distance calculation subunit is used to calculate the distance between the two according to the position coordinates of the steel bars in the industrial camera view and the position coordinates of the grasping robot arm in the industrial camera view to obtain the distance between the steel bars and the grasping robot arm; a sending subunit is used to send the distance so that the grasping robot arm moves above the steel bars.

[0201] In one embodiment, the screening unit 304 includes:

[0202] A feature extraction subunit is used to extract the steel bar target feature image in the initial image; a preliminary screening subunit is used to preliminarily screen out incomplete and overlapping target feature images in the steel bar target feature image to obtain a preliminary qualified feature image; a preprocessing subunit is used to perform image preprocessing on the preliminary qualified feature image to obtain a preprocessing result; a contour calculation subunit is used to extract edge contour information of the image from the preprocessing result and calculate the length and area of ​​the contour feature; a matching subunit is used to perform geometric feature matching between the contour feature and the workpiece template in the system template library and calculate the matching degree; a re-screening subunit is used to screen out complete workpiece features according to the matching degree to obtain a qualified steel bar target feature image.

[0203] In one embodiment, the preliminary screening subunit comprises:

[0204] The template matching module is used to find the regional features in the steel bar target feature image that are similar to the given original working template through the correlation matching CC algorithm to obtain the similarity matching features; the filtering module is used to filter out the features corresponding to the stacked and incomplete workpieces in the similarity matching features to obtain a preliminary qualified feature image.

[0205] In one embodiment, the grabbing unit is used to receive construction drawings or parameters, construct a coordinate system, and calculate the placement position of the steel bars to be grabbed based on the coordinate system; wherein the coordinate system takes the upper left corner of the control platform as the origin, the ground plane as the horizontal plane, the x-axis is parallel to the gantry, the y-axis is perpendicular to the gantry, and the z-axis is perpendicular to the ground; the steel bars to be grabbed are grabbed according to the information of the steel bars to be grabbed, and the steel bars to be grabbed are placed in the placement position.

[0206] The receiving of construction drawings or parameters, constructing a coordinate system, and calculating the placement of the steel bars based on the coordinate system includes:

[0207] Receive construction drawings or parameters and extract the length, width, shape, spacing and quantity of steel bars to obtain steel bar related information;

[0208] The control unit analyzes the relevant information of the steel bars, constructs a coordinate system, and calculates the placement coordinates of each steel bar and the coordinates of the cross points based on the coordinate system;

[0209] The step of analyzing the steel bar related information and calculating the placement coordinates of each steel bar and the coordinates of the cross intersections includes:

[0210] Establish a coordinate system;

[0211] According to the length and spacing of the horizontal and vertical bars in the relevant information of the steel bars, the coordinates of the two ends of each horizontal and vertical bar are calculated to determine the position of the cross intersection.

[0212] Wherein, the step of grabbing the steel bar to be grabbed according to the steel bar information to be grabbed and placing the steel bar to be grabbed at a placement position includes:

[0213] Calculating the three-dimensional coordinates of the grabbing point corresponding to the steel bar to be grabbed;

[0214] Adjusting the movement and rotation of the robot arm according to the three-dimensional coordinates of the grasping point to accurately grasp the steel bar to be grasped;

[0215] After the robot arm completes grabbing the steel bar to be grabbed, the steel bar to be grabbed is placed in a placement position.

[0216] The step of calculating the three-dimensional coordinates of the grabbing point corresponding to the steel bar to be grabbed includes:

[0217] Setting the reference coordinates of the grasping subsystem, the coordinate system of the lower arm of the grasping robot arm, the coordinate system of the upper arm of the grasping robot arm, and the coordinate system of the first depth camera; wherein the first depth camera is arranged on the grasping robot arm; the grasping subsystem includes the grasping robot arm;

[0218] The grasping position in the coordinate system of the upper arm of the grasping robot is determined according to the transformation relationship between the reference coordinates of the grasping subsystem and the coordinate system of the lower arm of the grasping robot, the coordinate system of the upper arm of the grasping robot and the coordinate system of the lower arm of the grasping robot, and the coordinate system of the first depth camera and the coordinate system of the upper arm of the grasping robot to obtain the three-dimensional coordinates of the grasping point corresponding to the steel bar to be grasped.

[0219] It should be noted that technicians in the relevant field can clearly understand that the specific implementation process of the above-mentioned scattered steel bar identification and grasping system 300 and each unit can refer to the corresponding description in the aforementioned method embodiment, and for the convenience and brevity of description, it will not be repeated here.

[0220] The above-mentioned scattered steel bar identification and grasping system 300 can be implemented in the form of a program of an electronic device, and the electronic device program can be run on an electronic device.

[0221] The electronic device includes a processor, a memory and a network interface connected via a system bus, wherein the memory may include a non-volatile storage medium and an internal memory.

[0222] The non-volatile storage medium can store an operating system and a computer program. The computer program includes program instructions, and when the program instructions are executed, the processor can execute a method for identifying and grabbing scattered steel bars.

[0223] The processor is used to provide computing and control capabilities to support the operation of the entire computer equipment.

[0224] The internal memory provides an environment for the operation of a computer program in a non-volatile storage medium. When the computer program is executed by a processor, the processor can execute a method for identifying and grasping scattered steel bars.

[0225] The network interface is used for network communication with other devices. It will be appreciated by those skilled in the art that a specific computer device may include more or fewer components than those shown in the figure, or may combine certain components, or may have a different arrangement of components.

[0226] The processor is used to run a computer program stored in the memory to implement the following steps:

[0227] Acquire an image taken by an industrial camera overlooking the entire working area to obtain an image of the working area; perform steel bar and grabbing robot arm identification on the working area image, calculate the distance between the steel bar and the grabbing robot arm, and send the distance so that the grabbing robot arm moves above the steel bar; acquire an image taken by a depth camera set on the grabbing robot arm to obtain an initial image; filter the complete and non-overlapping steel bar target feature image in the initial image to obtain a qualified steel bar target feature image; calculate the average depth value of the qualified steel bar target based on the qualified steel bar target feature image combined with the depth position information obtained by the depth camera; sort according to the average depth value to determine the steel bar closest to the depth camera, and determine the corresponding position and posture to obtain the steel bar information to be grabbed; according to the steel bar information to be grabbed, the robot arm grabs the corresponding steel bar and places it at the specified position.

[0228] In one embodiment, when the processor implements the step of identifying the steel bar and the grabbing robot arm on the working area image, calculating the distance between the steel bar and the grabbing robot arm, and sending the distance so that the grabbing robot arm moves above the steel bar, the processor specifically implements the following steps:

[0229] A convolutional neural network detection algorithm is used to identify the steel bars in the image of the working area to obtain the position coordinates of the steel bars in the industrial camera view; the position coordinates of the grabbing robot arm in the industrial camera view are determined by a QR code marked on the grabbing robot arm; the distance between the position coordinates of the steel bars in the industrial camera view and the position coordinates of the grabbing robot arm in the industrial camera view is calculated to obtain the distance between the steel bars and the grabbing robot arm; the distance is sent to enable the grabbing robot arm to move above the steel bars.

[0230] In one embodiment, when the processor implements the step of screening the complete and non-overlapping steel bar target feature images in the initial image to obtain the qualified steel bar target feature images, the processor specifically implements the following steps:

[0231] Extract the steel bar target feature image in the initial image; preliminarily screen out incomplete and overlapping target feature images in the steel bar target feature image to obtain a preliminary qualified feature image; perform image preprocessing on the preliminary qualified feature image to obtain a preprocessing result; extract edge contour information of the image from the preprocessing result, and calculate the length and area of ​​the contour feature; perform geometric feature matching between the contour feature and the workpiece template in the system template library, and calculate the matching degree; screen out complete workpiece features according to the matching degree to obtain a qualified steel bar target feature image.

[0232] In one embodiment, when the processor implements the step of preliminarily screening out incomplete and overlapping target feature images in the steel bar target feature image to obtain a preliminary qualified feature image, the processor specifically implements the following steps:

[0233] The regional features similar to the given original working template in the target feature image of the steel bar are searched through the correlation matching CC algorithm to obtain the features of similarity matching; the features corresponding to the stacked and incomplete workpieces in the similarity matching features are filtered out to obtain a preliminary qualified feature image.

[0234] In one embodiment, when the processor implements the step of grabbing the corresponding steel bar by the robot arm according to the steel bar information to be grabbed and placing it at the designated position, the processor specifically implements the following steps:

[0235] Receive construction drawings or parameters, construct a coordinate system, and calculate the placement position of the steel bars to be grabbed based on the coordinate system; wherein the coordinate system takes the upper left corner of the control platform as the origin, the ground plane as the horizontal plane, the x-axis is parallel to the gantry, the y-axis is perpendicular to the gantry, and the z-axis is perpendicular to the ground; grab the steel bars to be grabbed according to the information of the steel bars to be grabbed, and place the steel bars to be grabbed at the placement position.

[0236] In one embodiment, when the processor implements the steps of receiving the construction drawings or parameters, constructing a coordinate system, and calculating the placement position of the steel bars based on the coordinate system, the processor specifically implements the following steps:

[0237] Receive construction drawings or parameters, and extract the length, width, shape, placement spacing and quantity of steel bars to obtain steel bar related information; the control unit parses the steel bar related information, constructs a coordinate system, and calculates the placement coordinates of each steel bar and the coordinates of the cross intersection based on the coordinate system;

[0238] The step of analyzing the steel bar related information and calculating the placement coordinates of each steel bar and the coordinates of the cross intersections includes:

[0239] Establish a coordinate system; calculate the coordinates of both ends of each horizontal and vertical reinforcement according to the length and spacing of the horizontal and vertical reinforcement in the relevant information of the reinforcement, and determine the position of the cross intersection.

[0240] In one embodiment, when the processor implements the step of grabbing the steel bar to be grabbed according to the steel bar information to be grabbed and placing the steel bar to be grabbed at the placement position, the processor specifically implements the following steps:

[0241] Calculate the three-dimensional coordinates of the grasping point corresponding to the steel bar to be grasped; adjust the movement and rotation of the robot arm according to the three-dimensional coordinates of the grasping point to accurately grasp the steel bar to be grasped; when the robot arm completes grasping the steel bar to be grasped, place the steel bar to be grasped in a placement position.

[0242] In one embodiment, when the processor implements the step of calculating the three-dimensional coordinates of the grabbing point corresponding to the steel bar to be grabbed, the processor specifically implements the following steps:

[0243] Set the reference coordinates of the grasping subsystem, the coordinate system of the lower arm of the grasping robot, the coordinate system of the upper arm of the grasping robot, and the coordinate system of the first depth camera; wherein the first depth camera is arranged on the grasping robot; the grasping subsystem includes the grasping robot; according to the transformation relationship between the reference coordinates of the grasping subsystem and the coordinate system of the lower arm of the grasping robot, the coordinate system of the upper arm of the grasping robot and the coordinate system of the lower arm of the grasping robot, and the coordinate system of the first depth camera and the coordinate system of the upper arm of the grasping robot, the grasping position in the coordinate system of the upper arm of the grasping robot is determined to obtain the three-dimensional coordinates of the grasping point corresponding to the steel bar to be grasped.

[0244] It should be understood that in the embodiments of the present application, the processor may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0245] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0246] In the several embodiments provided by the present invention, it should be understood that the disclosed systems and methods can be implemented in other ways. For example, the system embodiments described above are only schematic. For example, the division of each unit is only a logical function division, and there may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed.

[0247] The steps in the method of the embodiment of the present invention can be adjusted in order, combined and deleted according to actual needs. The units in the system of the embodiment of the present invention can be combined, divided and deleted according to actual needs. In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0248] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a storage medium. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, terminal, or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention.

[0249] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed by the present invention, and these modifications or replacements should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. A method for identifying and grabbing scattered steel bars, characterized in that: include: Acquire an image taken by an industrial camera overlooking the entire work area to obtain an image of the work area; Recognize the steel bars and the grabbing robot arm on the work area image, calculate the distance between the steel bars and the grabbing robot arm, and send the distance so that the grabbing robot arm moves above the steel bars; Acquire an image taken by a depth camera disposed on a grasping robot arm to obtain an initial image; Screening complete and non-overlapping steel bar target feature images in the initial image to obtain qualified steel bar target feature images; Calculating an average depth value of a qualified steel bar target according to the qualified steel bar target feature image combined with the depth position information acquired by the depth camera; Sorting according to the average depth value to determine the steel bar closest to the depth camera, and determining the corresponding position and posture to obtain the steel bar information to be grasped; According to the information of the steel bars to be grabbed, the robot arm grabs the corresponding steel bars and places them at the designated position; According to the information of the steel bar to be grabbed, the robot arm grabs the corresponding steel bar and places it at a designated position, including: Receive construction drawings or parameters, construct a coordinate system, and calculate the placement position of the steel bars to be grabbed based on the coordinate system; wherein the coordinate system takes the upper left corner of the control platform as the origin, the ground plane as the horizontal plane, the x-axis is parallel to the gantry, the y-axis is perpendicular to the gantry, and the z-axis is perpendicular to the ground; Grab the steel bar to be grabbed according to the steel bar information to be grabbed, and place the steel bar to be grabbed at a placement position; The method of grabbing the steel bar to be grabbed according to the steel bar information to be grabbed and placing the steel bar to be grabbed at a placement position includes: Calculating the three-dimensional coordinates of the grabbing point corresponding to the steel bar to be grabbed; Adjusting the movement and rotation of the robot arm according to the three-dimensional coordinates of the grasping point to accurately grasp the steel bar to be grasped; When the robot arm finishes grabbing the steel bar to be grabbed, the steel bar to be grabbed is placed in a placement position; The calculating of the three-dimensional coordinates of the grabbing point corresponding to the steel bar to be grabbed includes: Setting the reference coordinates of the grasping subsystem, the coordinate system of the lower arm of the grasping robot arm, the coordinate system of the upper arm of the grasping robot arm, and the coordinate system of the first depth camera; wherein the first depth camera is arranged on the grasping robot arm; the grasping subsystem includes the grasping robot arm; The grasping position in the coordinate system of the upper arm of the grasping robot is determined according to the transformation relationship between the reference coordinates of the grasping subsystem and the coordinate system of the lower arm of the grasping robot, the coordinate system of the upper arm of the grasping robot and the coordinate system of the lower arm of the grasping robot, and the coordinate system of the first depth camera and the coordinate system of the upper arm of the grasping robot to obtain the three-dimensional coordinates of the grasping point corresponding to the steel bar to be grasped.

2. The method for identifying and grabbing scattered steel bars according to claim 1, characterized in that: The step of identifying the steel bar and the grabbing mechanical arm on the working area image, calculating the distance between the steel bar and the grabbing mechanical arm, and sending the distance so that the grabbing mechanical arm moves above the steel bar includes: Using a convolutional neural network detection algorithm to identify steel bars in the image of the work area to obtain the position coordinates of the steel bars in the industrial camera view; Determine the position coordinates of the grabbing robot arm in the industrial camera view through the QR code marked on the grabbing robot arm; The distance between the steel bar and the grabbing robot arm is calculated according to the position coordinates of the steel bar in the industrial camera view and the position coordinates of the grabbing robot arm in the industrial camera view, so as to obtain the distance between the steel bar and the grabbing robot arm; The distance is sent so that the grabbing robot moves over the steel bar.

3. The method for identifying and grabbing scattered steel bars according to claim 1, characterized in that: The screening of the complete and non-overlapping steel bar target feature images in the initial image to obtain a qualified steel bar target feature image includes: Extracting a steel bar target feature image in the initial image; Preliminarily screening out incomplete and overlapping target feature images in the steel bar target feature image to obtain a preliminary qualified feature image; Performing image preprocessing on the preliminary qualified feature image to obtain a preprocessing result; Extracting edge contour information of the image from the preprocessing result, and calculating the length and area of ​​the contour feature; Match the contour features with the workpiece template in the system template library for geometric features and calculate the matching degree; Complete workpiece features are screened out according to the matching degree to obtain a qualified steel bar target feature image.

4. The method for identifying and grabbing scattered steel bars according to claim 3 is characterized in that: The preliminary screening of incomplete and overlapping target feature images in the steel bar target feature image to obtain a preliminary qualified feature image includes: Searching for regional features in the steel bar target feature image that are similar to the given original working template through a correlation matching CC algorithm to obtain similarity matching features; The features corresponding to the stacked and incomplete workpieces in the similarity matching features are filtered out to obtain a preliminary qualified feature image.

5. The method for identifying and grabbing scattered steel bars according to claim 4, characterized in that: The receiving of construction drawings or parameters, constructing a coordinate system, and calculating the placement of steel bars based on the coordinate system includes: Receive construction drawings or parameters and extract the length, width, shape, spacing and quantity of steel bars to obtain steel bar related information; The control unit analyzes the relevant information of the steel bars, constructs a coordinate system, and calculates the placement coordinates of each steel bar and the coordinates of the cross points based on the coordinate system; The step of analyzing the steel bar related information and calculating the placement coordinates of each steel bar and the coordinates of the cross intersections includes: Establish a coordinate system; According to the length and spacing of the horizontal and vertical bars in the relevant information of the steel bars, the coordinates of the two ends of each horizontal and vertical bar are calculated to determine the position of the cross intersection.

6. The scattered steel bar identification and grabbing system is characterized by: include: An industrial image acquisition unit, used to acquire an image taken by an industrial camera overlooking the entire working area to obtain an image of the working area; A detection unit, used for identifying the steel bars and the grabbing mechanical arm in the working area image, calculating the distance between the steel bars and the grabbing mechanical arm, and sending the distance so that the grabbing mechanical arm moves above the steel bars; An initial image acquisition unit, used to acquire an image taken by a depth camera arranged on the grasping mechanical arm to obtain an initial image; A screening unit, used for screening complete and non-overlapping steel bar target feature images in the initial image to obtain qualified steel bar target feature images; A depth value calculation unit, used to calculate the average depth value of the qualified steel bar target according to the qualified steel bar target feature image combined with the depth position information obtained by the depth camera; A unit for determining information of steel bars to be grasped, used for sorting according to the average depth value to determine the steel bars closest to the depth camera, and determining the corresponding position and posture to obtain information of the steel bars to be grasped; A grabbing unit, used to grab the corresponding steel bar by a mechanical arm according to the information of the steel bar to be grabbed, and place it at a designated position; The grabbing unit is used to receive construction drawings or parameters, construct a coordinate system, and calculate the placement position of the steel bars to be grabbed based on the coordinate system; wherein, the coordinate system takes the upper left corner of the control platform as the origin, the ground plane as the horizontal plane, the x-axis is parallel to the gantry, the y-axis is perpendicular to the gantry, and the z-axis is perpendicular to the ground; the steel bars to be grabbed are grabbed according to the information of the steel bars to be grabbed, and the steel bars to be grabbed are placed at the placement position; the grabbing of the steel bars to be grabbed according to the information of the steel bars to be grabbed, and the steel bars to be grabbed are placed at the placement position, including: calculating the three-dimensional coordinates of the grabbing point corresponding to the steel bars to be grabbed; adjusting the movement and rotation of the mechanical arm according to the three-dimensional coordinates of the grabbing point to accurately grab the steel bars to be grabbed; when the mechanical arm grabs the steel bars to be grabbed, the mechanical arm moves and rotates to accurately grab the steel bars to be grabbed; , placing the steel bar to be grasped in the placement position; wherein, the calculation of the three-dimensional coordinates of the grasping point corresponding to the steel bar to be grasped includes: setting the reference coordinates of the grasping subsystem, the coordinate system of the lower arm of the grasping robot arm, the coordinate system of the upper arm of the grasping robot arm, and the coordinate system of the first depth camera; wherein the first depth camera is arranged on the grasping robot arm; the grasping subsystem includes the grasping robot arm; according to the transformation relationship between the reference coordinates of the grasping subsystem and the coordinate system of the lower arm of the grasping robot arm, the coordinate system of the upper arm of the grasping robot arm and the coordinate system of the lower arm of the grasping robot arm, the coordinate system of the first depth camera and the coordinate system of the upper arm of the grasping robot arm, the grasping position in the coordinate system of the upper arm of the grasping robot arm is determined to obtain the three-dimensional coordinates of the grasping point corresponding to the steel bar to be grasped.

7. An electronic device, characterized in that: The electronic device comprises a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the method according to any one of claims 1 to 5 when executing the computer program.

Images