Panoramic recognition method and system for steel bar binding points and electronic equipment

Abstract

The embodiment of the invention discloses a steel bar binding point panoramic recognition method and system and electronic equipment. The method comprises the following steps: acquiring a video stream shot by an industrial camera mounted on a portal frame under a moving condition to extract images and obtain a plurality of camera view images; fusing and splicing the plurality of camera view images to obtain a scene panoramic image; performing cross intersection detection on the scene panoramic image to obtain intersection information; wherein the cross point information comprises a target frame of a bound cross point and a target frame of an unbound cross point; and the cross point information is sent to a binding mechanical arm, so that the binding mechanical arm determines the coordinates of the cross cross points which are not bound, and binding is carried out. By implementing the method provided by the embodiment of the invention, the panoramic view angle image of steel bar binding can be identified, so that all binding points can be effectively identified and detected, and potential safety risks are avoided.

Description

Technical Field

[0001] The present invention relates to intelligent manufacturing in the construction industry, and more specifically to a method, system and electronic equipment for panoramic recognition of steel bar binding points. Background Art

[0002] When identifying the cross points of the steel bar, the current automatic steel bar tying robot usually relies on the monocular camera installed on the robot to collect and analyze pictures. The main limitation of this solution is that it can only recognize a single point, so the recognition speed is slow. If a tying point is missed, it is difficult for the robot to detect this problem, which may lead to loose steel bar tying and cause safety hazards. In order to ensure the quality of tying, it is extremely important to perform panoramic recognition of the steel bar tying points.

[0003] Especially for meshes with complex shapes such as trapezoidal steel mesh, the situation is more complicated. The trapezoidal steel mesh includes not only horizontal bars but also trapezoidal vertical bars. If only one camera is used to shoot at a single perspective, this shooting method may block some binding points. The blocked binding points cannot be confirmed whether they have been bound, which may cause these points to be missed, and there are still safety hazards.

[0004] Therefore, it is necessary to design a new method to realize the recognition of panoramic perspective images of steel bar binding to ensure that all binding points can be effectively identified and detected to avoid potential safety risks. Summary of the invention

[0005] The purpose of the present invention is to overcome the defects of the prior art and provide a method, system and electronic equipment for panoramic identification of steel bar binding points.

[0006] To achieve the above object, the present invention adopts the following technical solution: a method for panoramic identification of steel bar binding points, comprising: Acquire a video stream captured by an industrial camera mounted on a gantry in a moving state to extract images and obtain a number of camera view images; Fusing and stitching several camera view images to obtain a panoramic image of the scene; Performing cross intersection detection on the scene panoramic image to obtain intersection information; wherein the intersection information includes target frames of bound cross intersections and target frames of unbound cross intersections; The intersection information is sent to the tying robot arm, so that the tying robot arm receives the intersection information and uses the depth camera data to transform the coordinate position through the coordinate system conversion to determine the coordinates of the untied cross intersection points, and the tying robot arm performs steel bar tying corresponding to the untied cross intersection point coordinates.

[0007] A further technical solution is: fusing and splicing a plurality of camera view images to obtain a panoramic image of the scene, including: Preprocessing the camera view image to obtain a preprocessing result; Identifying significant feature points of each of the preprocessing results, and determining corresponding feature point pairs in different preprocessing results; Calculate the transformation matrix between the preprocessing results based on the feature point pairs; Aligning all the preprocessing results according to the transformation matrix, and seamlessly splicing the overlapping areas to obtain a splicing result; The stitching result is cropped and scaled to obtain a panoramic image of the scene.

[0008] A further technical solution is: identifying the significant feature points of each of the preprocessing results and determining corresponding feature point pairs in different preprocessing results, including: Using ORB feature point detection algorithm to identify significant feature points of each of the preprocessing results; Feature point matching is performed based on the significant feature points of each of the preprocessing results to obtain corresponding feature point pairs in different preprocessing results.

[0009] A further technical solution is: aligning all the preprocessing results according to the transformation matrix and seamlessly splicing the overlapping areas to obtain a splicing result, including: All the pre-processing results are aligned using a transformation matrix and spliced ​​to obtain a preliminary splicing result; An image fusion algorithm is used to process the overlapping area of ​​the preliminary stitching result to ensure a smooth transition at the stitching point to obtain a stitching result.

[0010] A further technical solution is: performing cross intersection detection on the scene panoramic image to obtain intersection information includes: The cross intersection detection is performed on the panoramic image of the scene based on the target detection algorithm of Yolov8 to obtain the intersection information.

[0011] A further technical solution is: the intersection information is sent to a tying robot arm, so that the tying robot arm receives the intersection information, and uses the depth camera data to perform coordinate position transformation through coordinate system conversion to determine the coordinates of the untied cross intersection, and the tying robot arm performs steel bar tying corresponding to the coordinates of the untied cross intersection, including: Sending the intersection information to the lashing robot arm; The tying robot arm performs coordinate position transformation according to the intersection information and data captured by a depth camera installed on the tying robot arm to determine the coordinates of the untied cross intersection; The binding robot arm binds the steel bars according to the coordinates of the unbound cross points and re-checks the binding status after the binding is completed.

[0012] A further technical solution is that the tying mechanical arm performs coordinate position transformation according to the intersection information and data captured by a depth camera installed on the tying mechanical arm to determine the coordinates of the untied cross intersection, including: Set the reference coordinates of the lashing subsystem, the coordinate system of the lower arm of the lashing robot, the coordinate system of the upper arm of the lashing robot, and the coordinate system of the depth camera; According to the transformation relationship between the reference coordinates of the tying subsystem and the coordinate system of the lower arm of the tying robot arm, the coordinate system of the upper arm of the tying robot arm and the coordinate system of the lower arm of the tying robot arm, and the coordinate system of the depth camera and the coordinate system of the upper arm of the tying robot arm, the cross intersection coordinates in the coordinate system of the tying robot arm are determined in combination with the intersection information and the data taken by the depth camera installed on the tying robot arm to obtain the untied cross intersection coordinates.

[0013] Its further technical solution is: the reference coordinate of the tying subsystem takes the direction perpendicular to the chassis as the z-axis, and the x and y-axis parallel to the chassis; the coordinate system of the lower arm of the tying robot arm takes the z-axis parallel to the lower arm, the x-axis parallel to the degree of freedom plane, and the y-axis perpendicular to the degree of freedom plane; the coordinate system of the upper arm of the tying robot arm takes the z-axis parallel to the lower arm, the x-axis parallel to the degree of freedom plane, and the y-axis perpendicular to the degree of freedom plane; the coordinate system of the depth camera takes the z-axis parallel to the lower arm, the x-axis parallel to the degree of freedom plane, and the y-axis perpendicular to the degree of freedom plane; the tying subsystem includes a tying robot arm arranged on a gantry.

[0014] The present invention also provides a panoramic identification system for steel bar binding points, comprising: An image acquisition unit, used to acquire a video stream captured by an industrial camera mounted on a gantry in a moving state, to extract images and obtain a plurality of camera view images; A fusion and stitching unit, used for fusing and stitching a plurality of camera view images to obtain a panoramic image of the scene; A detection unit, configured to perform cross intersection detection on the scene panoramic image to obtain intersection information; wherein the intersection information includes a target frame of a bound cross intersection and a target frame of an unbound cross intersection; A sending unit is used to send the intersection information to the tying robot arm, so that the tying robot arm receives the intersection information and uses the depth camera data to transform the coordinate position through the coordinate system conversion to determine the coordinates of the untied cross intersection points, and the tying robot arm performs steel bar tying corresponding to the untied cross intersection point coordinates.

[0015] The present invention further provides an electronic device, comprising a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the above method when executing the computer program.

[0016] The beneficial effects of the present invention compared with the prior art are as follows: the present invention uses an industrial camera installed on a gantry to shoot a mobile video stream, extracts and splices a panoramic image of the scene; performs cross-intersection detection on the panoramic image, identifies and marks the bound and unbound intersections; transmits the intersection information to the binding robot arm to guide it to complete the binding task of the corresponding steel bars; and realizes the recognition of the panoramic perspective image of the steel bar binding to ensure that all binding points can be effectively identified and detected to avoid potential safety risks.

[0017] The present invention is further described below in conjunction with the accompanying drawings and specific embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying any creative work.

[0019] Figure 1 A schematic diagram of a flow chart of a method for panoramic identification of reinforcement bar binding points provided by an embodiment of the present invention; Figure 2 A schematic diagram of a sub-process of a method for panoramic identification of reinforcement binding points provided by an embodiment of the present invention; Figure 3 A schematic diagram of a sub-process of a method for panoramic identification of reinforcement binding points provided by an embodiment of the present invention; Figure 4 A schematic diagram of a sub-process of a method for panoramic identification of reinforcement binding points provided by an embodiment of the present invention; Figure 5 A schematic diagram of a sub-process of a method for panoramic identification of reinforcement binding points provided by an embodiment of the present invention; Figure 6 A schematic diagram of a sub-process of a method for panoramic identification of reinforcement binding points provided by an embodiment of the present invention; Figure 7 Schematic diagram of the reference coordinates of the lashing subsystem, the coordinate system of the lower arm of the lashing robot arm, the coordinate system of the upper arm of the lashing robot arm, and the coordinate system of the depth camera provided in an embodiment of the present invention Figure 8 A schematic diagram of a gantry deployment scenario provided in an embodiment of the present invention; Fig. 9 A schematic diagram of splicing and fusion provided by an embodiment of the present invention; Fig.10 A schematic diagram of a panoramic image of a scene provided by an embodiment of the present invention; Fig.11 A schematic block diagram of a panoramic identification system for reinforcement bar binding points provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0020] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0021] It should be understood that when used in this specification and the appended claims, the terms "include" and "comprises" indicate the presence of described features, integers, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or combinations thereof.

[0022] It should also be understood that the terms used in this specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. As used in the specification of the present invention and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.

[0023] It should be further understood that the term "and / or" used in the present description and the appended claims refers to and includes any and all possible combinations of one or more of the associated listed items.

[0024] See also Figure 1 , Figure 1 A schematic flow chart of a panoramic identification method for steel bar binding points provided in an embodiment of the present invention. The panoramic identification method for steel bar binding points is applied to the cross-intersection identification and binding process of an automatic steel bar batch grabbing and binding system, wherein the fully automatic steel bar batch grabbing and binding system includes at least two groups of gantries, each gantry consisting of a horizontal beam, a vertical beam and a moving device. The vertical beam moves up and down by a lifting device to adjust the operating height of the horizontal beam; the horizontal beam of the gantry is provided with a grabbing robot arm and a binding robot arm, the grabbing robot arm is used to grab the scattered steel bars on the side, and the binding robot arm is used to bind the cross-intersection points; the number of grabbing robot arms and binding robot arms depends on the specific situation, and there are at least 2 grabbing and 1 binding. The gantry includes a horizontal bar gantry for moving horizontal bars.

[0025] The cross intersection is formed by a grabbing robot arm set on the horizontal reinforcement gantry and a grabbing robot arm set on the vertical reinforcement gantry. After grabbing the horizontal reinforcement and placing it at the specified position, the vertical reinforcement is grabbed and placed at the specified position.

[0026] In addition, a first plane camera is set on the vertical reinforcement gantry and the horizontal reinforcement gantry, and a first depth camera is set on the grasping mechanical arm; the first plane camera is used to capture and identify the steel bars to be grasped, determine the location of the steel bars to be grasped, and send the location of the steel bars to be grasped to the grasping subsystem, so that the grasping subsystem moves to the location of the steel bars to be grasped; the first depth camera is used to analyze and determine the grasping point of the steel bars to be grasped when the grasping subsystem moves to the location of the steel bars to be grasped, and then determine the three-dimensional coordinates of the grasping point, and send it to the grasping subsystem, and when the grasping subsystem moves to the three-dimensional coordinates, the three-dimensional coordinates of the grasping point are fed back in real time until the grasping subsystem grasps the steel bars. The method for determining the three-dimensional coordinates of the grasping point is consistent with the method for determining the coordinates of the untied cross intersection.

[0027] In addition, a plane camera is arranged on the cross bar gantry and a depth camera is arranged on the tying mechanical arm, and the depth camera is the depth camera mentioned in this embodiment, wherein the second depth camera is used for identifying and preliminarily locating the position of the cross knot, and sending the preliminarily determined position to the tying mechanical arm, so that the tying mechanical arm moves to the cross knot according to the preliminarily determined position and the coordinates of the cross intersection; The depth camera is used to analyze the position of the cross knot to the local coordinates of the depth camera when the tying robot arm moves to the preliminarily determined position to determine the position of the cross knot, and send the position of the cross knot to the tying robot arm, so that the tying robot arm moves to the position of the cross knot and ties the steel bars at the position of the cross knot.

[0028] In this embodiment, the cross knot position of the steel bar is first determined by the second plane camera, and then the tying robot arm is notified to reach the preliminary position. The tying robot arm will move to the approximate position of the cross knot according to the cross knot position of the steel bar determined by the second plane camera. After arriving, the second depth camera will accurately measure the distance information between the robot arm and the cross knot, and help the tying robot arm to accurately tie the steel bar through continuous feedback.

[0029] When determining the position of the cross knot, that is, the cross intersection, the panoramic recognition method of the steel bar binding point mentioned in this embodiment is specifically used to recognize the cross intersection.

[0030] In one embodiment, see Figure 1 The above-mentioned method for panoramic identification of steel bar binding points includes the following steps S110 to S140.

[0031] S110 , acquiring a video stream shot by an industrial camera installed on a gantry in a moving state to extract images and obtain a plurality of camera view images.

[0032] In this embodiment, the camera view image refers to a clear scene image.

[0033] Specifically, Figure 8 As shown, the industrial camera scans the entire steel bar binding environment through a mobile system installed on a gantry, that is, a mobile platform. Before binding, the steel bars have been fixed to the entire frame through partially completed binding points, and the unbound cross points need further processing; the industrial camera moves at a certain speed, records the video stream of the scene, and extracts clear video frame images from it, which will be used to generate panoramic images.

[0034] S120: Fusing and splicing a plurality of camera view images to obtain a panoramic image of the scene.

[0035] In this embodiment, if Fig. 9 As shown, the scene panoramic image refers to a panoramic image that can present the steel bar situation.

[0036] In one embodiment, see Figure 2 The above-mentioned step S120 may include steps S121~S125.

[0037] S121. Preprocess the camera view image to obtain a preprocessing result.

[0038] In this embodiment, the preprocessing result refers to the result after the camera view image is preprocessed.

[0039] Specifically, noise reduction, brightness adjustment and contrast optimization are performed on each frame of video images. This processing can improve image quality and make feature point detection more accurate; reduce the impact of image noise and uneven lighting, and improve the accuracy of subsequent feature point recognition and stitching effects.

[0040] S122: Identify significant feature points of each of the preprocessing results, and determine corresponding feature point pairs in different preprocessing results.

[0041] In this embodiment, the feature point pair refers to a set of significant feature points that match in different preprocessing results.

[0042] Use feature point detection algorithms (such as ORB) in the preprocessed images to identify significant feature points and match corresponding feature point pairs in different images; accurate feature point matching is the key to image alignment and stitching, and can ensure accurate alignment of overlapping areas of images.

[0043] In one embodiment, see Figure 3 , the above-mentioned step S122 may include steps S1221~S1222.

[0044] S1221. Use the ORB feature point detection algorithm to identify significant feature points of each of the preprocessing results.

[0045] In this embodiment, after the image is preprocessed (such as noise removal, brightness adjustment, etc.), the ORB (Oriented FAST and Rotated BRIEF) feature point detection algorithm is used to identify significant feature points in the image. ORB is a fast and efficient feature detection algorithm that combines the FAST feature point detector and the BRIEF feature descriptor to detect points with strong features in the image.

[0046] Extract key points from the image and calculate descriptors for these key points. The descriptor is a vector representation of the feature points, which is used to describe the local features of these points.

[0047] The ORB algorithm can maintain the stability of feature points under different lighting conditions, viewing angle changes, and image rotation, and is suitable for processing images in various practical scenarios; ORB combines fast feature point detection and descriptor calculation methods to make the feature point recognition process both efficient and accurate, suitable for real-time applications; by effectively identifying significant feature points, ORB can extract key information from the image, which is crucial for subsequent feature point matching and image stitching.

[0048] S1222: Perform feature point matching based on the significant feature points of each of the preprocessing results to obtain corresponding feature point pairs in different preprocessing results.

[0049] In this embodiment, after identifying the significant feature points in each image, these feature points are paired using a feature point matching algorithm (such as brute force matching or FLANN-based matching). The matching algorithm calculates the similarity between the feature points based on their descriptors and finds the corresponding feature point pairs in different images.

[0050] The feature point pairs generated by this process contain points at the same or similar positions matched from different images, providing a basis for subsequent image alignment and stitching.

[0051] Through accurate feature point matching, the correspondence between images can be correctly established, thereby improving the accuracy of stitching and alignment; feature point matching can reduce image misalignment problems caused by factors such as changes in perspective and lighting differences, thereby reducing stitching gaps and inconsistencies; through effective feature point matching, the stitched panoramic image can be made more coherent and natural, enhancing the visual effect.

[0052] S123: Calculate a transformation matrix between the preprocessing results based on the feature point pairs.

[0053] In this embodiment, the transformation matrix defines how to transform one image into the coordinate system of another image.

[0054] The transformation matrix provides the mathematical basis for image alignment, ensures the geometric consistency of images when stitching, and reduces the stitching gaps.

[0055] S124. Align all the preprocessing results according to the transformation matrix, and seamlessly splice the overlapping areas to obtain a splicing result.

[0056] In this embodiment, the stitching result refers to an image after seamless stitching.

[0057] The images are aligned using the calculated transformation matrix, and the overlapping areas in the images are seamlessly stitched. The image fusion algorithm is used to process the overlapping areas to eliminate the seams. The resulting panoramic image is more visually coherent and natural, reducing the visible gaps and inconsistencies in image stitching, and improving the overall image quality.

[0058] In one embodiment, see Figure 5 The above-mentioned step S124 may include steps S1241~S1242.

[0059] S1241. Align all the preprocessing results using a transformation matrix, and perform a splicing operation to obtain a preliminary splicing result.

[0060] In this embodiment, first, all pre-processed images are geometrically aligned according to the corresponding transformation matrices using the transformation matrix obtained by feature point matching in step S122. These transformation matrices usually include translation, rotation, scaling and perspective transformation, etc. After alignment, these aligned images are spliced. This usually involves fusing the overlapping parts in the images so that they are visually seamlessly connected to form a preliminary splicing result.

[0061] Using the transformation matrix can ensure that all images are accurately aligned when stitching, reducing the visual differences caused by image misalignment; through alignment and stitching operations, a preliminary stitching result is generated, providing basic data for subsequent image fusion; aligning the preprocessing results and performing stitching operations is an efficient process that can quickly generate preliminary panoramic images and is suitable for real-time or large-scale image processing applications.

[0062] S1242: Use an image fusion algorithm to process the overlapping area of ​​the preliminary stitching result to ensure a smooth transition at the stitching point to obtain a stitching result.

[0063] In this embodiment, after the initial stitching result is generated, obvious seams or discontinuities may appear in the overlapping areas. The image fusion algorithm is used to process these overlapping areas, with the aim of reducing the obviousness of the seams through techniques such as smooth transition and color matching, so that the stitching result looks more natural; commonly used fusion techniques include gradient fusion (color transition in gradient areas), multi-band fusion (synthesis of image details of different frequencies), etc. These techniques can effectively handle color differences and uneven illumination in images.

[0064] The image fusion algorithm can smoothly process the transition area of ​​the stitching, eliminate visual seams, and make the stitching result more natural and coherent; through the fusion algorithm, the overlapping areas in the stitching result will be seamlessly connected with the surrounding areas, enhancing the overall visual effect and quality of the panoramic image; the more natural stitching effect improves the user's visual experience, making the final panoramic image more in line with the visual experience of the actual scene. S125: cropping and scaling the stitching result to obtain a panoramic image of the scene.

[0065] In this embodiment, the spliced ​​image is cropped to remove useless edge portions, and is scaled to meet display requirements and application scenarios.

[0066] The display effect of the final panoramic image is optimized to adapt it to specific application requirements, such as monitoring, display or analysis, ensuring the clarity and readability of the image in actual use.

[0067] S130, performing cross intersection detection on the scene panoramic image to obtain intersection information; wherein the intersection information includes target frames of bound cross intersections and target frames of unbound cross intersections.

[0068] In this embodiment, cross intersection detection is performed on the scene panoramic image based on the target detection algorithm of Yolov8 to obtain intersection information.

[0069] Since convolutional neural networks have achieved significant optimization effects in the field of detection and recognition, they are widely used in various recognition tasks. In this embodiment, the Yolov8 target detection algorithm framework is used in the cross intersection detection process for detection. In order to identify the trapezoidal state of steel bars, image data sets of different scales were collected, including 1500 high-position bundling state images and 1500 low-position images. The data set is divided into 70% training set, 15% validation set and 15% test set. The experimental results show that the detection accuracy of the model is basically the same as the original model, and a good detection effect is achieved.

[0070] S140. Send the intersection information to the tying robot arm, so that the tying robot arm receives the intersection information and uses the depth camera data to transform the coordinate position through the coordinate system conversion to determine the coordinates of the untied cross intersection points, and the tying robot arm performs steel bar tying corresponding to the untied cross intersection point coordinates.

[0071] In one embodiment, see Figure 6 The above-mentioned step S140 may include steps S141~S143.

[0072] S141, sending the intersection information to the binding mechanical arm; S142: The tying robot arm performs coordinate position transformation according to the intersection information and data captured by a depth camera installed on the tying robot arm to determine the coordinates of the untied cross intersection.

[0073] In this embodiment, the untied cross point coordinates refer to the cross point coordinates based on the tying robot arm.

[0074] In one embodiment, see Figure 7 The above-mentioned step S142 may include steps S1421~S1422.

[0075] S1421. Set the reference coordinates of the tying subsystem, the coordinate system of the lower arm of the tying robot arm, the coordinate system of the upper arm of the tying robot arm, and the coordinate system of the depth camera; the reference coordinates of the tying subsystem take the z-axis perpendicular to the chassis, and the x and y-axes parallel to the chassis; the coordinate system of the lower arm of the tying robot arm takes the z-axis parallel to the lower arm, the x-axis parallel to the degree of freedom plane, and the y-axis perpendicular to the degree of freedom plane; the coordinate system of the upper arm of the tying robot arm takes the z-axis parallel to the lower arm, the x-axis parallel to the degree of freedom plane, and the y-axis perpendicular to the degree of freedom plane; the coordinate system of the depth camera takes the z-axis parallel to the lower arm, the x-axis parallel to the degree of freedom plane, and the y-axis perpendicular to the degree of freedom plane; the tying subsystem includes a tying robot arm arranged on a gantry.

[0076] S1422. According to the transformation relationship between the reference coordinates of the tying subsystem and the coordinate system of the lower arm of the tying robot arm, the coordinate system of the upper arm of the tying robot arm and the coordinate system of the lower arm of the tying robot arm, and the coordinate system of the depth camera and the coordinate system of the upper arm of the tying robot arm, the cross intersection coordinates in the coordinate system of the tying robot arm are determined in combination with the intersection information and the data captured by the depth camera installed on the tying robot arm to obtain the untied cross intersection coordinates.

[0077] Specifically, the reference coordinates of the lashing subsystem, the coordinate system of the lower arm of the lashing subsystem robot, the coordinate system of the upper arm of the lashing subsystem robot, and the coordinate system of the depth camera are set, such as Figure 7As shown, the ellipse in the figure is the base of the robot arm, the rectangle is the upper and lower arms of the robot arm, the upper one is the lower arm, and the square below the upper arm represents the depth camera; let O be the reference coordinate of the lashing subsystem, with the z axis perpendicular to the chassis, and the x and y axes parallel to the chassis; O1 is the coordinate system of the lower arm of the lashing robot arm, with the z axis parallel to the lower arm, the x axis parallel to the degree of freedom plane, and the y axis perpendicular to the degree of freedom plane; O2 is the coordinate system of the upper arm of the lashing robot arm, with the z axis parallel to the lower arm, the x axis parallel to the degree of freedom plane, and the y axis perpendicular to the degree of freedom plane; O3 is the coordinate system of the depth camera, and the xyz three-axis direction is consistent with the upper arm of the lashing robot arm. The coordinate system O2 is consistent; the transformation from coordinate system O1 to coordinate system O includes translation along the z axis and rotation around the xyz axis. According to the known rotation angle and translation, let the rotation matrix be R1, the translation be t1, and let the coordinates of the point under coordinate system O1 be a1=(x1, y1, z1), then the coordinate transformation formula from O1 to O can be obtained; according to the transformation relationship between the reference coordinates of the tying subsystem and the coordinate system of the lower arm of the tying robot arm, the coordinate system of the upper arm of the tying robot arm and the coordinate system of the lower arm of the tying robot arm, and the coordinate system of the depth camera and the coordinate system of the upper arm of the tying robot arm, the coordinates of the untied cross point in the coordinate system of the tying robot arm are determined. Specifically, the coordinate transformation formula from O1 to O is: , where a is the vector representation of the coordinates in the O coordinate system, a = (x, y, z); similarly, the coordinate transformation from O2 to O1 can be obtained as ; Where R2 is the rotation matrix, t2 is the translation vector, and a2=(x2, y2, z2) is the point coordinate in the coordinate system O2; Similarly, the coordinate transformation from O3 to O2 can be obtained as , where e=[1, 0; 0, 1], t3 is the translation vector, and a3=(x3, y3, z3) is the coordinate in the coordinate system O3. Combining the above transformation formula, the coordinates of the untied cross intersection can be calculated.

[0078] S143, the binding robot arm binds the steel bars according to the coordinates of the unbound cross points, and re-checks the binding status after the binding is completed.

[0079] The binding robot first locates the binding position by obtaining the coordinates of the cross points of the unbound steel bars; the robot operates precisely according to these coordinates and binds the designated position. After the binding is completed, the binding robot will conduct another inspection to verify whether the binding status of the steel bars meets the requirements and ensure the accuracy and firmness of the binding.

[0080] In this embodiment, multiple binding robot arms can coordinate operations and automatically complete the rebar binding task through advanced control algorithms and communication protocols. These robot arms are equipped with sensors and visual systems to achieve precise movements and real-time task allocation; they can also identify, locate, bind and detect rebars, integrate hardware, software and control systems to form a complete automated rebar binding system; in complex environments, the system's advanced image processing and machine learning technologies perform high-precision detection. Through fast algorithms and optimized hardware, fast detection speed and high work efficiency are ensured; after the rebar binding is completed, the system performs subsequent verification through visual sensors and detection algorithms to ensure that the binding quality meets the standards. These technologies can detect whether the binding is firm and correct, so as to make necessary adjustments.

[0081] In addition, by integrating high-performance sensors, precise control systems and intelligent algorithms, the robot arm can quickly and accurately perform binding tasks. Optimized path planning and real-time adjustment algorithms further improve efficiency and accuracy; industrial cameras composed of high-resolution cameras and wide-angle lenses are used to shoot the steel bar area from multiple angles, and then a panoramic bird's-eye view is generated through image stitching technology to provide comprehensive steel bar layout information; combined with panoramic images and deep learning technology, the system can automatically detect the binding status in a large area, identify unbound or incorrectly bound areas, and achieve overall completion evaluation; using high-resolution image analysis and pattern recognition technology, the system can quickly and accurately locate the cross-intersection of the steel bars, improving the detection accuracy and processing speed.

[0082] The above-mentioned panoramic recognition method of steel bar binding points uses an industrial camera installed on a gantry to shoot a mobile video stream, extract and stitch a panoramic image of the scene; performs cross-intersection detection on the panoramic image, identifies and marks the bound and unbound intersections; transmits the intersection information to the binding robot arm to guide it to complete the binding task of the corresponding steel bars; and realizes the recognition of the panoramic perspective image of the steel bar binding to ensure that all binding points can be effectively identified and detected to avoid potential safety risks.

[0083] Fig.11 is a schematic block diagram of a panoramic identification system 300 for steel bar binding points provided by an embodiment of the present invention. Fig.11 As shown, corresponding to the above-mentioned panoramic identification method of steel bar binding points, the present invention also provides a panoramic identification system 300 of steel bar binding points. The panoramic identification system 300 of steel bar binding points includes a unit for executing the above-mentioned panoramic identification method of steel bar binding points. Specifically, please refer to Fig.11 The steel bar binding point panoramic recognition system 300 includes an image acquisition unit 301, a fusion and splicing unit 302, a detection unit 303 and a sending unit 304.

[0084] The image acquisition unit 301 is used to acquire a video stream shot by an industrial camera installed on a gantry in a moving state to extract images and obtain a plurality of camera view images; the fusion and stitching unit 302 is used to fuse and stitch a plurality of camera view images to obtain a panoramic image of the scene; the detection unit 303 is used to perform cross intersection detection on the panoramic image of the scene to obtain intersection information; wherein the intersection information includes a target frame of a bound cross intersection and a target frame of an unbound cross intersection; the sending unit 304 is used to send the intersection information to a binding robot arm, so that the binding robot arm receives the intersection information, and uses the depth camera data to perform coordinate position transformation through coordinate system transformation to determine the coordinates of the unbound cross intersection, and the binding robot arm performs steel bar binding corresponding to the coordinates of the unbound cross intersection.

[0085] In one embodiment, the fusion and splicing unit 302 includes: A preprocessing subunit is used to preprocess the camera view image to obtain a preprocessing result; a feature point determination subunit is used to identify the significant feature points of each of the preprocessing results and determine the corresponding feature point pairs in different preprocessing results; a transformation matrix determination subunit is used to calculate the transformation matrix between the preprocessing results based on the feature point pairs; a stitching subunit is used to align all the preprocessing results according to the transformation matrix and seamlessly stitch the overlapping areas to obtain a stitching result; a cropping and scaling subunit is used to crop and scale the stitching result to obtain a panoramic image of the scene.

[0086] In one embodiment, the feature point determination subunit includes: A feature point recognition module is used to use the ORB feature point detection algorithm to identify the significant feature points of each of the preprocessing results; a feature point matching module is used to perform feature point matching based on the significant feature points of each of the preprocessing results to obtain corresponding feature point pairs in different preprocessing results.

[0087] In one embodiment, the splicing subunit comprises: The preliminary stitching module is used to align all the preprocessing results by using a transformation matrix and perform a stitching operation to obtain a preliminary stitching result; the transition module is used to process the overlapping area of ​​the preliminary stitching results by using an image fusion algorithm to ensure a smooth transition at the stitching point to obtain a stitching result.

[0088] In one embodiment, the detection unit 303 is used to perform cross intersection detection on the scene panoramic image based on the target detection algorithm of Yolov8 to obtain intersection information.

[0089] In one embodiment, the sending unit 304 is used to send the intersection information to the tying robot arm; the tying robot arm performs coordinate position transformation according to the intersection information and the data captured by the depth camera installed on the tying robot arm to determine the coordinates of the untied cross intersection points; the tying robot arm ties the steel bars according to the coordinates of the untied cross intersection points, and re-detects the tying status after the tying is completed.

[0090] In one embodiment, the tying robot arm performs coordinate position transformation according to the intersection information and data captured by a depth camera installed on the tying robot arm to determine the coordinates of the untied cross intersection, including: Set the reference coordinates of the tying subsystem, the coordinate system of the lower arm of the tying robot arm, the coordinate system of the upper arm of the tying robot arm, and the coordinate system of the depth camera; according to the transformation relationship between the reference coordinates of the tying subsystem and the coordinate system of the lower arm of the tying robot arm, the coordinate system of the upper arm of the tying robot arm and the coordinate system of the lower arm of the tying robot arm, and the coordinate system of the depth camera and the coordinate system of the upper arm of the tying robot arm, determine the cross intersection coordinates in the coordinate system of the tying robot arm in combination with the intersection information and the data taken by the depth camera installed on the tying robot arm to obtain the untied cross intersection coordinates.

[0091] It should be noted that technical personnel in the relevant field can clearly understand that the specific implementation process of the above-mentioned steel bar binding point panoramic identification system 300 and each unit can refer to the corresponding description in the aforementioned method embodiment, and for the convenience and brevity of description, it will not be repeated here.

[0092] The above-mentioned panoramic identification system 300 of steel bar binding points can be implemented in the form of a program of an electronic device, and the electronic device program can be run on an electronic device.

[0093] The electronic device includes a processor, a memory and a network interface connected via a system bus, wherein the memory may include a non-volatile storage medium and an internal memory.

[0094] The non-volatile storage medium can store an operating system and a computer program. The computer program includes program instructions, and when the program instructions are executed, the processor can execute a method for identifying and grabbing scattered steel bars.

[0095] The processor is used to provide computing and control capabilities to support the operation of the entire computer equipment.

[0096] The internal memory provides an environment for the operation of a computer program in a non-volatile storage medium. When the computer program is executed by a processor, the processor can execute a method for identifying and grasping scattered steel bars.

[0097] The network interface is used for network communication with other devices. It will be appreciated by those skilled in the art that a specific computer device may include more or fewer components than those shown in the figure, or may combine certain components, or may have a different arrangement of components.

[0098] The processor is used to run a computer program stored in the memory to implement the following steps: A video stream captured by an industrial camera installed on a gantry in a moving state is obtained to extract images and obtain a plurality of camera view images; the plurality of camera view images are fused and spliced ​​to obtain a panoramic image of the scene; cross intersection detection is performed on the panoramic image of the scene to obtain intersection information; wherein the intersection information includes a target frame of a bound cross intersection and a target frame of an unbound cross intersection; the intersection information is sent to a binding robot arm so that the binding robot arm receives the intersection information and uses the depth camera data to perform a coordinate position transformation through a coordinate system transformation to determine the coordinates of the unbound cross intersection, and the binding robot arm performs steel bar binding corresponding to the coordinates of the unbound cross intersection.

[0099] In one embodiment, when the processor implements the step of fusing and stitching a plurality of camera view images to obtain a panoramic image of the scene, the processor specifically implements the following steps: The camera view image is preprocessed to obtain a preprocessing result; significant feature points of each preprocessing result are identified, and corresponding feature point pairs in different preprocessing results are determined; a transformation matrix between the preprocessing results is calculated based on the feature point pairs; all the preprocessing results are aligned according to the transformation matrix, and overlapping areas are seamlessly spliced ​​to obtain a spliced ​​result; the spliced ​​result is cropped and scaled to obtain a panoramic image of the scene.

[0100] In one embodiment, when the processor implements the step of identifying the significant feature points of each of the preprocessing results and determining the corresponding feature point pairs in different preprocessing results, the processor specifically implements the following steps: An ORB feature point detection algorithm is used to identify significant feature points of each of the preprocessing results; feature point matching is performed based on the significant feature points of each of the preprocessing results to obtain corresponding feature point pairs in different preprocessing results.

[0101] In one embodiment, when the processor implements the step of aligning all the preprocessing results according to the transformation matrix and seamlessly splicing the overlapping areas to obtain the splicing result, the processor specifically implements the following steps: All the preprocessing results are aligned using a transformation matrix and stitched together to obtain a preliminary stitching result; an image fusion algorithm is used to process the overlapping area of ​​the preliminary stitching result to ensure a smooth transition at the stitching point to obtain a stitching result.

[0102] In one embodiment, when the processor implements the step of performing cross intersection detection on the scene panoramic image to obtain intersection information, the processor specifically implements the following steps: The cross intersection detection is performed on the panoramic image of the scene based on the target detection algorithm of Yolov8 to obtain the intersection information.

[0103] In one embodiment, when the processor implements the sending of the intersection information to the tying robot arm so that the tying robot arm receives the intersection information and uses the depth camera data to perform coordinate position transformation through coordinate system transformation to determine the untied cross intersection coordinates, and the tying robot arm performs the steel bar tying step corresponding to the untied cross intersection coordinates, the following steps are specifically implemented: The intersection information is sent to the binding robot arm; the binding robot arm performs coordinate position transformation according to the intersection information and the data captured by the depth camera installed on the binding robot arm to determine the coordinates of the unbound cross intersection points; the binding robot arm binds the steel bars according to the coordinates of the unbound cross intersection points, and re-checks the binding status after the binding is completed.

[0104] In one embodiment, when the processor implements the step of performing coordinate position transformation by the tying robot arm according to the intersection information and the data captured by the depth camera installed on the tying robot arm to determine the coordinates of the untied cross intersection, the processor specifically implements the following steps: Set the reference coordinates of the tying subsystem, the coordinate system of the lower arm of the tying robot arm, the coordinate system of the upper arm of the tying robot arm, and the coordinate system of the depth camera; according to the transformation relationship between the reference coordinates of the tying subsystem and the coordinate system of the lower arm of the tying robot arm, the coordinate system of the upper arm of the tying robot arm and the coordinate system of the lower arm of the tying robot arm, and the coordinate system of the depth camera and the coordinate system of the upper arm of the tying robot arm, determine the cross intersection coordinates in the coordinate system of the tying robot arm in combination with the intersection information and the data taken by the depth camera installed on the tying robot arm to obtain the untied cross intersection coordinates.

[0105] Among them, the reference coordinates of the tying subsystem have the z-axis perpendicular to the chassis, and the x and y axes parallel to the chassis; the coordinate system of the lower arm of the tying robotic arm has the z-axis parallel to the lower arm, the x-axis parallel to the degree of freedom plane, and the y-axis perpendicular to the degree of freedom plane; the coordinate system of the upper arm of the tying robotic arm has the z-axis parallel to the lower arm, the x-axis parallel to the degree of freedom plane, and the y-axis perpendicular to the degree of freedom plane; the coordinate system of the depth camera has the z-axis parallel to the lower arm, the x-axis parallel to the degree of freedom plane, and the y-axis perpendicular to the degree of freedom plane; the tying subsystem includes a tying robotic arm arranged on a gantry.

[0106] It should be understood that in the embodiments of the present application, the processor may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0107] In the several embodiments provided by the present invention, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of each unit is only a logical function division, and there may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed.

[0108] The steps in the method of the embodiment of the present invention can be adjusted in order, combined and deleted according to actual needs. The units in the device of the embodiment of the present invention can be combined, divided and deleted according to actual needs. In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0109] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a storage medium. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, terminal, or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention.

[0110] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed by the present invention, and these modifications or replacements should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. A panoramic identification method for steel bar binding points, characterized in that: include: Acquire a video stream captured by an industrial camera mounted on a gantry in a moving state to extract images and obtain a number of camera view images; Fusing and stitching several camera view images to obtain a panoramic image of the scene; Performing cross intersection detection on the scene panoramic image to obtain intersection information; wherein the intersection information includes target frames of bound cross intersections and target frames of unbound cross intersections; The intersection information is sent to the tying robot arm, so that the tying robot arm receives the intersection information and uses the depth camera data to transform the coordinate position through the coordinate system conversion to determine the coordinates of the untied cross intersection points, and the tying robot arm performs steel bar tying corresponding to the untied cross intersection point coordinates.

2. The method for panoramic identification of steel bar binding points according to claim 1, characterized in that: The fusing and stitching of the plurality of camera view images to obtain a panoramic image of the scene includes: Preprocessing the camera view image to obtain a preprocessing result; Identifying significant feature points of each of the preprocessing results, and determining corresponding feature point pairs in different preprocessing results; Calculate the transformation matrix between the preprocessing results based on the feature point pairs; Aligning all the preprocessing results according to the transformation matrix, and seamlessly splicing the overlapping areas to obtain a splicing result; The stitching result is cropped and scaled to obtain a panoramic image of the scene.

3. The method for panoramic identification of steel bar binding points according to claim 2, characterized in that: The identifying of the significant feature points of each of the preprocessing results and determining corresponding feature point pairs in different preprocessing results includes: Using ORB feature point detection algorithm to identify significant feature points of each of the preprocessing results; Feature point matching is performed based on the significant feature points of each of the preprocessing results to obtain corresponding feature point pairs in different preprocessing results.

4. The method for panoramic identification of steel bar binding points according to claim 2, characterized in that: The step of aligning all the preprocessing results according to the transformation matrix and seamlessly splicing the overlapping areas to obtain a splicing result includes: All the pre-processing results are aligned using a transformation matrix and spliced ​​to obtain a preliminary splicing result; An image fusion algorithm is used to process the overlapping area of ​​the preliminary stitching result to ensure a smooth transition at the stitching point to obtain a stitching result.

5. The method for panoramic identification of steel bar binding points according to claim 1, characterized in that: The performing cross intersection detection on the scene panoramic image to obtain intersection information includes: The cross intersection detection is performed on the panoramic image of the scene based on the target detection algorithm of Yolov8 to obtain the intersection information.

6. The method for panoramic identification of steel bar binding points according to claim 1, characterized in that: The step of sending the intersection information to the tying robot arm so that the tying robot arm receives the intersection information and uses the depth camera data to perform coordinate position transformation through coordinate system transformation to determine the coordinates of the untied cross intersections, and the tying robot arm ties the steel bars corresponding to the coordinates of the untied cross intersections, including: Sending the intersection information to the lashing robot arm; The tying robot arm performs coordinate position transformation according to the intersection information and data captured by a depth camera installed on the tying robot arm to determine the coordinates of the untied cross intersection; The binding robot arm binds the steel bars according to the coordinates of the unbound cross points and re-checks the binding status after the binding is completed.

7. The method for panoramic identification of steel bar binding points according to claim 6, characterized in that: The tying mechanical arm performs coordinate position transformation according to the intersection information and data captured by a depth camera installed on the tying mechanical arm to determine the coordinates of the untied cross intersection, including: Set the reference coordinates of the lashing subsystem, the coordinate system of the lower arm of the lashing robot, the coordinate system of the upper arm of the lashing robot, and the coordinate system of the depth camera; According to the transformation relationship between the reference coordinates of the tying subsystem and the coordinate system of the lower arm of the tying robot arm, the coordinate system of the upper arm of the tying robot arm and the coordinate system of the lower arm of the tying robot arm, and the coordinate system of the depth camera and the coordinate system of the upper arm of the tying robot arm, the cross intersection coordinates in the coordinate system of the tying robot arm are determined in combination with the intersection information and the data taken by the depth camera installed on the tying robot arm to obtain the untied cross intersection coordinates.

8. The method for panoramic identification of steel bar binding points according to claim 7, characterized in that: The reference coordinates of the tying subsystem have the z-axis perpendicular to the chassis, and the x and y-axes parallel to the chassis; the coordinate system of the lower arm of the tying robotic arm has the z-axis parallel to the lower arm, the x-axis parallel to the degree of freedom plane, and the y-axis perpendicular to the degree of freedom plane; the coordinate system of the upper arm of the tying robotic arm has the z-axis parallel to the lower arm, the x-axis parallel to the degree of freedom plane, and the y-axis perpendicular to the degree of freedom plane; the coordinate system of the depth camera has the z-axis parallel to the lower arm, the x-axis parallel to the degree of freedom plane, and the y-axis perpendicular to the degree of freedom plane; the tying subsystem includes a tying robotic arm arranged on a gantry.

9. The panoramic identification system of steel bar binding points is characterized by: include: An image acquisition unit, used to acquire a video stream captured by an industrial camera mounted on a gantry in a moving state, to extract images and obtain a plurality of camera view images; A fusion and stitching unit, used for fusing and stitching a plurality of camera view images to obtain a panoramic image of the scene; A detection unit, configured to perform cross intersection detection on the scene panoramic image to obtain intersection information; wherein the intersection information includes a target frame of a bound cross intersection and a target frame of an unbound cross intersection; A sending unit is used to send the intersection information to the tying robot arm, so that the tying robot arm receives the intersection information and uses the depth camera data to transform the coordinate position through the coordinate system conversion to determine the coordinates of the untied cross intersection points, and the tying robot arm performs steel bar tying corresponding to the untied cross intersection point coordinates.

10. An electronic device, characterized in that: The electronic device comprises a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the method according to any one of claims 1 to 8 is implemented.

Images