Building path planning method and system based on machine vision

Abstract

The invention provides a masonry path planning method and system based on machine vision, and relates to the technical field of visual inspection, and the method comprises the steps: obtaining initial point cloud data of a to-be-smeared surface of a to-be-smeared target; analyzing the initial point cloud data by using a main plane segmentation algorithm, determining a region outer contour of the to-be-smeared surface, and determining a target space attitude of the to-be-smeared surface in a first space coordinate system through coordinate transformation; the target space attitude comprises a surface coordinate and a surface normal vector of the to-be-smeared surface; acquiring a space coordinate of a smearing part of the smearing mechanical arm under the first space coordinate system, and determining a current normal vector of the smearing part based on the space coordinate; according to the target space attitude of the to-be-smeared surface and the current normal vector of the smearing part, the smearing angle and the smearing distance of the smearing part relative to the to-be-smeared surface are adjusted, and the building path of the outer contour of the smearing part traversal area of the smearing mechanical arm is determined; wherein the smearing angle and the smearing distance are located in an expected threshold interval.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of visual detection, and particularly relates to a masonry path planning method and system based on machine vision. BACKGROUND

[0002] Due to large masonry engineering quantity of refractory bricks, the balance of the whole kiln, the consistency and uniformity of each layer of brick joints are directly related to the masonry quality, and have a key influence on whether the equipment can realize stable production, high yield and long-term safe operation. In the masonry process, the masonry quantity of each layer of refractory bricks is extremely large, how to ensure the balanced construction upwards layer by layer of the whole furnace is the core problem of the improvement of the masonry process. The traditional masonry method completely relies on manual operation: workers transport refractory materials and mortar from the ground to the masonry surface by operating lifting equipment, meanwhile, a scaffold is built on both sides of the equipment, and a wooden jump board is erected in the wall gap. With the increase of the masonry height, the height of the scaffold and the jump board needs to be adjusted gradually, and the workers perform manual work accordingly until the whole process is completed. However, this way is difficult to realize precise positioning control, the uniformity of the brick joints is poor, and then the sealing performance and service life of the overall structure are affected. Although the robot masonry technology has the advantages of high efficiency and small error, due to the narrow space in the furnace, the variety of refractory bricks, and the complex material conveying path, it is difficult for the mechanical arm to realize the requirements of precise movement, grabbing and synchronous mortar placement in the limited space. SUMMARY

[0003] In view of this, the present application provides a masonry path planning method and system based on machine vision.

[0004] The technical scheme of the present application is realized as follows: the present application provides a masonry path planning method based on machine vision in the first aspect, comprising:

[0005] obtaining initial point cloud data of a to-be-painted surface of a to-be-painted target;

[0006] analyzing the initial point cloud data by using a main plane segmentation algorithm, determining the area outer contour of the to-be-painted surface, and determining the target space posture of the to-be-painted surface in a first space coordinate system through coordinate conversion; the target space posture comprises surface coordinates and a surface normal vector of the to-be-painted surface;

[0007] obtaining the space coordinates of a painting part of a painting mechanical arm in the first space coordinate system, determining the current normal vector of the painting part based on the space coordinates, adjusting the painting angle and the painting distance of the painting part relative to the to-be-painted surface according to the target space posture of the to-be-painted surface and the current normal vector of the painting part, and determining the masonry path of the painting part of the painting mechanical arm traversing the area outer contour; wherein the painting angle and the painting distance are located in an expected threshold interval.

[0008] Preferably, after obtaining the initial point cloud data of the to-be-painted surface of the to-be-painted target, the method further comprises: preprocessing the initial point cloud data; the preprocessing comprises:

[0009] constraining the initial point cloud data by using a pass-through filter, filtering out point cloud data whose depth coordinates are outside a preset coordinate range, to obtain intermediate point cloud data;

[0010] dividing the intermediate point cloud data into a voxel grid by using a voxel filter, and retaining a center point of each voxel grid as preprocessed point cloud data.

[0011] Preferably, the analyzing the initial point cloud data by using a main plane segmentation algorithm to determine the area outer contour of the to-be-painted surface comprises:

[0012] detecting a main plane by using a random sample consensus algorithm to obtain main plane point cloud corresponding to the initial point cloud data;

[0013] performing coordinate transformation on the main plane point cloud, projecting the main plane point cloud onto a two-dimensional plane, and determining a point cloud convex hull;

[0014] obtaining a minimum bounding box based on a rotating rectangle corresponding to the point cloud convex hull, and determining the area outer contour of the to-be-painted surface according to the minimum bounding box.

[0015] Preferably, the analyzing the initial point cloud data by using a main plane segmentation algorithm to determine the area outer contour of the to-be-painted surface comprises:

[0016] analyzing the initial point cloud data by using a main plane segmentation algorithm to obtain an initial area outer contour of the to-be-painted surface;

[0017] extracting features of the initial area outer contour to obtain a set of corner point coordinates of the initial area outer contour;

[0018] sequentially arranging the corner points in the set of corner point coordinates in a clockwise or counterclockwise direction based on a spatial position relationship, to generate a final area outer contour of the to-be-painted surface.

[0019] Preferably, the determining the target spatial pose of the to-be-painted surface in the first spatial coordinate system by coordinate transformation comprises:

[0020] calibrating the shooting camera to obtain intrinsic parameters and distortion coefficients of the shooting camera;

[0021] A nine-point calibration method is used in combination with the intrinsic parameters and the distortion coefficients to determine a rigid transformation matrix between a camera coordinate system and a robot base coordinate system.

[0022] Based on the photographed point cloud data of the surface to be painted and the rigid transformation matrix, a target spatial pose of the surface to be painted in a first spatial coordinate system is determined.

[0023] On the basis of the above technical solutions, preferably, the painting angle and the painting distance of the painting part relative to the surface to be painted are adjusted according to the target spatial pose of the surface to be painted and the current normal vector of the painting part, and a masonry path of the painting part of the painting robot for traversing the outer contour of the region is determined, including:

[0024] The position coordinates of each corner point are determined from the surface coordinates of the surface to be painted.

[0025] The masonry path is determined based on the convex hull boundary corresponding to the position coordinates of the corner point, the painting angle and the painting distance.

[0026] On the basis of the above technical solutions, preferably, after the masonry path of the painting part of the painting robot for traversing the outer contour of the region is determined, the method further includes:

[0027] The joint speed of the robot is dynamically adjusted based on the trajectory curvature of the masonry path, wherein the joint speed of a straight line is greater than the joint speed of a corner.

[0028] Further preferably, the second aspect of the present application provides a masonry path planning system based on machine vision, including a data acquisition module, an analysis and conversion module and a path determination module, wherein,

[0029] The data acquisition module is configured to acquire initial point cloud data of a surface to be painted of a target to be painted.

[0030] The analysis and conversion module is configured to analyze the initial point cloud data by using a main plane segmentation algorithm, to determine the outer contour of the region of the surface to be painted, and to determine a target spatial pose of the surface to be painted in a first spatial coordinate system through coordinate conversion; the target spatial pose includes surface coordinates and a surface normal vector of the surface to be painted.

[0031] The path determination module is configured to acquire spatial coordinates of a painting part of a painting robot in the first spatial coordinate system, to determine a current normal vector of the painting part based on the spatial coordinates, to adjust the painting angle and the painting distance of the painting part relative to the surface to be painted according to the target spatial pose of the surface to be painted and the current normal vector of the painting part, and to determine a masonry path of the painting part of the painting robot for traversing the outer contour of the region; wherein the painting angle and the painting distance are located in an expected threshold interval.

[0032] Further preferably, the third aspect of the present application provides an electronic device comprising a processor and a memory; the memory has stored a computer program, wherein the computer program, when executed by the processor, implements the machine vision-based masonry path planning method of the first aspect.

[0033] Further preferably, the fourth aspect of the present application provides a computer storage medium having stored thereon a computer program, wherein the computer program, when executed by a processor, implements the machine vision-based masonry path planning method of the first aspect.

[0034] The machine vision-based masonry path planning method and system of the present application have the following beneficial effects over the prior art:

[0035] 1. The initial point cloud data of the to-be-painted surface of the to-be-painted target is analyzed by a main plane segmentation algorithm to determine the area outer contour of the to-be-painted surface, and the target space pose of the to-be-painted surface in the first space coordinate system is determined through coordinate conversion; at the same time, the space coordinates of the painting part of the painting robot in the first space coordinate system are obtained, and the current normal vector of the painting part is determined based on the space coordinates; the painting angle and the painting distance of the painting part relative to the to-be-painted surface are adjusted according to the target space pose of the to-be-painted surface and the current normal vector of the painting part, the masonry path of the painting part of the painting robot traversing the area outer contour is determined, the whole-process visual guidance of visual grabbing, visual painting, and visual placing is adopted, the masonry accuracy is greatly improved, and the masonry error is reduced.

[0036] 2. The main plane is detected by using a random sample consensus algorithm, then the main plane point cloud is subjected to coordinate transformation, projected onto a two-dimensional plane, and a convex hull is determined, and then a minimum bounding box is obtained based on the convex hull to finally determine the area outer contour of the to-be-painted surface. The main plane detection ensures accurate surface recognition, the convex hull and the minimum bounding box provide accurate boundaries, thereby optimizing the path planning of the robot arm, and by converting the three-dimensional boundary problem into a two-dimensional convex hull calculation, the computational complexity is reduced and the path planning efficiency is improved.

[0037] 3. By continuously monitoring the included angle between the current normal vector of the painting part and the surface normal vector of the to-be-painted surface, the robot arm pose is dynamically adjusted within the expected threshold to synchronously adjust the painting angle and the distance, thereby realizing high-precision masonry in the narrow space of the coke oven. BRIEF DESCRIPTION OF DRAWINGS

[0038] In order to make the technical solutions in the embodiments of the present application or the prior art clearer, the accompanying drawings needed in the embodiments or prior art description will be briefly introduced. Obviously, the accompanying drawings in the following description only only some embodiments of the present application, and all other embodiments obtained by those of ordinary skill in the art without creative work based on these drawings also belong to the protection scope of the present application.

[0039] Figure 1 A flowchart of a bricklaying path planning method based on machine vision provided by an embodiment of the present application;

[0040] Figure 2 A schematic diagram of a coke oven bricklaying robot system provided by an embodiment of the present application;

[0041] Figure 3 A schematic diagram of a refractory brick conveying device structure provided by an embodiment of the present application;

[0042] Figure 4 A schematic diagram of a truss mechanical arm structure provided by an embodiment of the present application;

[0043] Figure 5 A schematic diagram of a smearing mechanical arm structure provided by an embodiment of the present application;

[0044] Figure 6 A schematic diagram of a pumping device structure provided by an embodiment of the present application;

[0045] Figure 7 A schematic diagram of a bricklaying path planning system based on machine vision provided by an embodiment of the present application;

[0046] Figure 8 A schematic diagram of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0047] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work belong to the protection scope of the present application.

[0048] In some embodiments, as shown in Figure 1 , the method comprises the following steps. Figure 1 A flowchart of a bricklaying path planning method based on machine vision provided by an embodiment of the present application; the bricklaying path planning method based on machine vision provided by the present application comprises:

[0049] S110, obtaining initial point cloud data of a to-be-smearing surface of a to-be-smearing target.

[0050] Here, the target to be coated can be a refractory brick. A deep learning model constructed by a deep convolutional classification network is pre-trained using sample images to achieve automatic recognition and classification of the shape of the refractory brick. Based on this, images of the refractory brick are acquired in real time using a camera device, and the acquired images are input into the trained deep learning model to obtain the initial point cloud data of the surface to be coated on the target.

[0051] In some embodiments, after acquiring the initial point cloud data of the surface to be coated of the target, the method further includes: preprocessing the initial point cloud data; the preprocessing includes:

[0052] By using a pass-through filter to constrain the initial point cloud data, point cloud data whose depth coordinates are outside the preset coordinate range are filtered out to obtain intermediate point cloud data.

[0053] The intermediate point cloud data is divided into a voxel grid by using a voxel filter, and the center point of each voxel grid is retained as the preprocessed point cloud data.

[0054] In this embodiment, point cloud filtering and downsampling are used to reduce unnecessary data volume, thereby improving processing speed and reducing computational complexity. For example, a pass-through filter is applied to constrain the initial point cloud data, retaining only the point cloud data with Z-axis coordinates (i.e., point cloud depth coordinates) within a specified range, as intermediate point cloud data. Let the initial point cloud data be... Each point Coordinates are Filter by the following criteria:

[0055] ;

[0056] in, and The Z-axis range is set.

[0057] Voxel filters are used to downsample the intermediate point cloud data, further reducing the point density and computational cost. Let the voxel size be... The intermediate point cloud data is divided into a voxel grid using the following formula, while retaining the center point of each voxel:

[0058] .

[0059] S120: The initial point cloud data is analyzed using the principal plane segmentation algorithm to determine the outer contour of the area to be coated, and the target spatial pose of the surface to be coated in the first spatial coordinate system is determined through coordinate transformation; the target spatial pose includes the surface coordinates and surface normal vector of the surface to be coated.

[0060] The principal plane segmentation algorithm determines the optimal plane by randomly selecting three points to fit a plane, calculating the number of interior points, and iteratively identifying the best plane. Then, it uses convex hull calculation and boundary extraction to determine the outer contour of the region. During this process, the transformation matrix between coordinate systems can be obtained through hand-eye calibration.

[0061] In some embodiments, the initial point cloud data is analyzed using a principal plane segmentation algorithm to determine the outer contour of the area to be painted, including:

[0062] The random sampling consensus algorithm is used to detect the principal plane and obtain the principal plane point cloud corresponding to the initial point cloud data.

[0063] Perform coordinate transformation on the point cloud in the main plane, project it onto a two-dimensional plane, and determine the convex hull of the point cloud;

[0064] The minimum bounding box is obtained based on the rotating rectangle corresponding to the convex hull of the point cloud, and the outer contour of the area to be coated is determined based on the minimum bounding box.

[0065] In this embodiment, a planar model is fitted. To identify planes in the initial point cloud data, where, represents the parameters of the plane equation.

[0066] Let the convex hull of the two-dimensional projection points in the point cloud be:

[0067] ;

[0068] The convex hull is then:

[0069] ;

[0070] in, This represents the smallest convex polygon containing all points.

[0071] Calculate the angle between the plane and the Z-axis, determine the angle between the plane and the Z-axis, and ensure that the plane is close to the horizontal plane. The normal vector of the plane is... The normal vector of the Z-axis is included angle It can be obtained by calculating the dot product between normal vectors:

[0072] ;

[0073] like If the plane is less than a certain predetermined threshold, it is considered to be nearly horizontal.

[0074] The minimum bounding box is the smallest rectangular region that encloses the convex hull of a point cloud. It can be obtained by calculating the rotated rectangle of the convex hull. The convex hull of a point cloud is... The corner points of the smallest bounding box are , where each point By optimizing the rotation angle, the area of ​​the minimum bounding box is:

[0075] .

[0076] In some embodiments, the initial point cloud data is analyzed using a principal plane segmentation algorithm to determine the outer contour of the area to be painted, including:

[0077] The initial point cloud data is analyzed using the principal plane segmentation algorithm to obtain the initial outer contour of the area to be painted.

[0078] Feature extraction is performed on the outer contour of the initial region to obtain the set of corner coordinates of the outer contour of the initial region;

[0079] Based on spatial positional relationships, the corner points in the set of corner point coordinates are sorted clockwise or counterclockwise to generate the final outer contour of the area to be painted.

[0080] In this embodiment, the initial point cloud data is filtered, segmented into planes, and feature extracted to obtain a set of corner coordinates of the initial region's outer contour. Each corner point is then sorted clockwise or counterclockwise according to its spatial position to generate a polygon representing the outer contour of the smeared region. Based on this, a convex hull algorithm or polygon optimization method is used to remove redundant points, ensuring that the path is closed and without intersections.

[0081] In some embodiments, determining the target spatial pose of the surface to be coated in a first spatial coordinate system through coordinate transformation includes:

[0082] The camera is calibrated to obtain its intrinsic parameters and distortion coefficients.

[0083] The rigid transformation matrix between the camera coordinate system and the robot arm base coordinate system is determined by using the nine-point calibration method combined with intrinsic parameters and distortion coefficients.

[0084] The target spatial pose of the surface to be coated in the first spatial coordinate system is determined based on the point cloud data captured by the camera and the rigid transformation matrix.

[0085] In this embodiment, the intrinsic parameters of the camera are assumed to be: The distortion coefficient is The two-dimensional coordinates of the bricklaying robot are Camera pixel coordinates are The conversion formula is as follows:

[0086] ;

[0087] in, For rotation matrix, The above equation can be transformed into: (where the matrix is ​​a translation matrix)

[0088] ;

[0089] Therefore, we can conclude that:

[0090]

[0091] ;

[0092] Coordinates of three different points Substituting into the above formula, we get

[0093]

[0094] ;

[0095] The matrix form is as follows:

[0096]

[0097] ;

[0098] As the robot's end effector sequentially reaches nine different positions, the coordinates of these nine positions in the robot's base coordinate system are recorded, along with the corresponding pixel coordinates of the nine points on the image. Substituting this coordinate data into the formula allows for the calculation of the coordinate transformation matrix. :

[0099] .

[0100] It should be noted that in this embodiment, the target spatial pose of the surface to be coated in the first spatial coordinate system is determined by coordinate transformation, that is, the positioning of the surface to be coated is completed to achieve the final fixation of the target to be coated. This action can be performed synchronously with the acquisition of the initial point cloud data of the surface to be coated. The two can be obtained by taking pictures using different types of cameras.

[0101] S130, obtain the spatial coordinates of the smearing part of the smearing robot arm in the first spatial coordinate system, and determine the current normal vector of the smearing part based on the spatial coordinates; adjust the smearing angle and smearing distance of the smearing part relative to the surface to be smeared according to the target spatial posture of the surface to be smeared and the current normal vector of the smearing part, and determine the masonry path of the outer contour of the traversing area of ​​the smearing part of the smearing robot arm; wherein, the smearing angle and smearing distance are within the expected threshold range.

[0102] In some embodiments, adjusting the application angle and application distance of the application part relative to the surface to be applied based on the target spatial orientation of the surface to be applied and the current normal vector of the application part, and determining the masonry path of the outer contour of the area traversed by the application part of the application robot arm, includes:

[0103] Determine the coordinates of each corner point from the surface coordinates of the surface to be coated;

[0104] The masonry path is determined based on the convex hull boundary, smearing angle, and smearing distance corresponding to the corner point coordinates.

[0105] During the application process, the motion trajectory of the robotic arm and the position of the application part can be adjusted according to the convex hull boundary corresponding to the corner position coordinates, the current application angle, and the application distance. This ensures that the application part and the application surface maintain a preset ideal distance and angle, thereby achieving uniform mud application.

[0106] For example, based on the normal vector of the smeared surface The current normal vector of the smeared part The robot arm's end effector posture is adjusted in real time to maintain the angle between it and the application point at the optimal application angle. ±tolerance Within the range:

[0107] .

[0108] During execution, the application distance between the application point and the application surface is collected in real time. , and expected value If the difference between the two values ​​is greater than a preset threshold, the joint position is corrected in real time through inverse kinematics to compensate for uneven brick surface and installation deviation.

[0109] In some embodiments, after determining the masonry path of the outer contour of the area traversed by the coating section of the coating robot arm, the method further includes:

[0110] The joint speed of the robotic arm is dynamically adjusted based on the trajectory curvature of the masonry path; among which, the speed of the straight joints is greater than that of the corner joints.

[0111] Here, the speed of the robotic arm joints can be dynamically adjusted according to the curvature of the masonry path, achieving deceleration at corners and acceleration in straight lines, ensuring uniform mud thickness.

[0112] In one example, see Figure 2 , Figure 3 , Figure 4 , Figure 5 and Figure 6 , Figure 2 This is a schematic diagram of the layout of the coke oven masonry robot system provided in an embodiment of the present invention. Figure 3 This is a schematic diagram of the refractory brick conveying device provided in an embodiment of the present invention. Figure 4 This is a schematic diagram of a truss robotic arm structure provided in an embodiment of the present invention; Figure 5 This is a schematic diagram of the coating robotic arm structure provided in an embodiment of the present invention. Figure 6This is a schematic diagram of the pumping device provided in an embodiment of the present invention; the coke oven bricklaying robot system includes a lower truss guide rail 1, a refractory brick conveying device 2, a bricklaying platform 3, a bricklaying furnace body 4, an upper truss guide rail 5, a truss robotic arm 6, a truss 7, a coating robotic arm 8, and a pumping device 9. The coating robotic arm 8 can move in the X direction along the upper truss guide rail 5 via the coating robotic arm connector 85, and in the Y direction along the coating robotic arm coupling 84. Taking the completed layer of refractory bricks on the masonry platform 3 as an example, the coating robotic arm 8 first coats the bottom surface. Starting from the bottom surface coating start point, it coats the surface according to the preset coating path. Before coating begins, the pump 9 and butterfly valve 83 are turned on. The butterfly valve 83 is installed close to the nozzle to ensure real-time control of the slurry spraying and stopping. The nozzle 81 is connected to the outlet of the pump 9 through a wear-resistant pipe, and the refractory slurry is fed from the refractory slurry supply area 92. The single screw 93 is used for conveying to ensure a continuous supply of refractory slurry. When the bottom surface coating target point is reached, the bottom surface coating is completed. The pump 9 and butterfly valve 83 are turned off, and the coating robotic arm 8 smoothly switches to the side of the refractory brick for taking pictures and waiting position through the path planning algorithm.

[0113] Sensor 22 in the refractory brick conveying device 2 detects the presence of refractory bricks 61 on the brick placement platform 23. The refractory bricks 61 are conveyed via a telescopic cylinder 24 and a brick conveying guide rail 25. The manual conveying button 21 is used for equipment debugging. The truss 7 can move in the X-axis via the lower truss guide rail 1. The truss robotic arm 6 can move in the X-axis along the upper truss guide rail 5 via the truss robotic arm connector 66, in the Y-axis along the truss robotic arm coupling 65, and in the Z-axis along the linear motion component 64. When the refractory brick 61 is conveyed by the refractory brick conveying device 2 to the desired gripping position, the truss robotic arm 6 moves within the truss space according to a predetermined program to the position to be photographed and identified. The first 3D camera 62 installed on the truss robotic arm 6 takes pictures of the refractory brick 61 for identification and positioning. The clamping mechanism 67 on the truss robotic arm 6 clamps the refractory brick 61. The truss robotic arm 6 clamps the refractory brick 61 and moves it to the side of the refractory brick in the XYZ direction of the truss space in a multi-axis linkage manner according to the predetermined program.

[0114] The coating robot arm 8, whose pose has been converted to the waiting position for taking pictures of the side of the refractory brick, has its second 3D camera 82 installed on the flange 86 taking pictures of the side of the refractory brick 1 for recognition. The coordinates of each corner point of the side 1 are obtained through point cloud processing. The coordinates are sorted and transmitted to the coating robot arm 8. The coating robot arm 8 determines the starting position of the coating path through the corner point coordinates and plans the coating path trajectory of the coating robot arm 8 in real time to coat the side of the refractory brick 1.

[0115] After the real-time planning of the slurry application path is completed, before the application begins, pump 9 and butterfly valve 83 are turned on. The nozzle 81 is connected to the outlet of pump 9 via a wear-resistant pipe to ensure a continuous supply of refractory slurry. After the refractory brick side 1 is coated, pump 9 and butterfly valve 83 are turned off, and the slurry application robot arm 8 returns to the refractory brick side image waiting position. The refractory brick 61 held by the clamping mechanism 67 is rotated 180° by the rotating mechanism 63 on the truss robot arm 6. The second 3D camera 82 will take an image of the uncoated refractory brick side 2 and obtain the coordinates of each corner point of side 2 through point cloud processing. The coordinates are sorted and transmitted to the slurry application robot arm 8. The slurry application robot arm 8 determines the starting position of the slurry application path based on the corner point coordinates and plans the slurry application path trajectory in real time to apply refractory brick side 2. After the real-time planning of the slurry application path is completed, before the application begins, pump 9 and butterfly valve 83 are turned on. The nozzle 81 is connected to the outlet of pump 9 via a wear-resistant pipe to ensure a continuous supply of refractory slurry. After the coating is applied to the side of the refractory brick 2, the pump 9 and butterfly valve 83 are shut off, and the coating robot arm 8 returns to the side of the refractory brick to take a picture and wait for the coating of the next refractory brick.

[0116] The truss robotic arm 6 moves the refractory bricks 61, whose sides have been coated, to the required placement position above the bricks according to a predetermined program via multi-axis linkage. The first 3D camera 62 takes pictures of the position to be laid and performs point cloud matching to accurately locate the placement position of the refractory bricks, ensuring that the distance between the brick joints is approximately 5mm. When the truss robotic arm 6 moves downward in the Z direction along the linear motion component 64 to a certain position, the clamping mechanism 67 releases the refractory bricks 61 while applying gentle pressure, thus completing the laying of the refractory bricks 61.

[0117] See another example for further details. Figure 2 and Figure 4 The operation method and steps are as follows: The first 3D camera 62 takes pictures, identifies and positions the object, and the truss robotic arm 6 places the refractory bricks 61.

[0118] The truss robotic arm 6 moves the coated refractory bricks above the pre-placement area. The first 3D camera 62 takes a picture of the position to be laid, obtains the point cloud data of the current brick position, and matches it with the refractory brick point cloud data calculated during the recognition and grasping process to accurately locate the position where the refractory bricks should be laid, ensuring that the distance between the brick joints is approximately 5mm.

[0119] After the truss robotic arm 6 grips the refractory brick 61 that has undergone the coating process, it moves to the approximate area above the brick placement location according to a preset program. The first 3D camera 62 is then activated to acquire real-time point cloud data of the target location. Before point cloud processing, filtering (such as pass-through filtering or voxel downsampling) is performed to reduce noise. Let the acquired point cloud set be... Each point .

[0120] Using the standard point cloud model of refractory brick 61 already saved during the capture, let the set of brick point clouds be... First, coarse registration is performed to obtain the initial rotation matrix R0 and translation vector t0. Then, based on the initial alignment, the Iterative Closest Point (ICP) algorithm is used for fine registration to minimize the Euclidean distance between the two sets of points.

[0121] The ICP convergence condition is set as follows: maximum number of iterations. Convergence error threshold (Mean squared error).

[0122] The ICP error optimization formula is:

[0123] ;

[0124] Where R is the final rotation matrix; t is the final translation vector; The target masonry location; For brick points.

[0125] The least squares method and SVD decomposition method are used to solve the problem, and the pose transformation matrix T on which the brick should be placed on the masonry surface is finally obtained, with the expression:

[0126] ;

[0127] Where R is a 3×3 rotation matrix and t is a 3×1 translation vector.

[0128] The transformation relationship is:

[0129] ;

[0130] That is, after transformation, the brick point b reaches the target position p′.

[0131] The transformation matrix T is converted into the motion target of the end effector of the robotic arm; inverse kinematics is used to solve the joint space path of the gantry robotic arm; trapezoidal velocity planning is selected as the interpolation method to ensure smooth trajectory.

[0132] Before approaching the target location, make local fine adjustments based on the point cloud to ensure that the distance between the brick joints of the refractory bricks is controlled at approximately 5mm.

[0133] Actual control offset Δ:

[0134]

[0135] in, , , It is the desired position; , , This refers to the actual placement location.

[0136] like The fine-tuning compensation action is initiated, and the fine-tuning amount is adjusted slightly through reverse motion control.

[0137] When the truss robotic arm 6 moves to the placement position, it slowly descends along the Z-axis; the clamping mechanism 67 releases the brick and at the same time applies gentle pressure through a downward pressing action to ensure that the brick fully fits the masonry surface and eliminates tiny gaps.

[0138] In some embodiments, please refer to Figure 7 , Figure 7 This is a schematic diagram of a machine vision-based masonry path planning system provided in an embodiment of the present invention. The present invention provides a machine vision-based masonry path planning system 700, comprising: a data acquisition module 710, an analysis and conversion module 720, and a path determination module 730; wherein,

[0139] The data acquisition module 710 is configured to acquire the initial point cloud data of the surface to be coated of the target object;

[0140] The analysis and transformation module 720 is configured to analyze the initial point cloud data using the principal plane segmentation algorithm, determine the outer contour of the area to be coated, and determine the target spatial pose of the surface to be coated in the first spatial coordinate system through coordinate transformation; the target spatial pose includes the surface coordinates and surface normal vector of the surface to be coated.

[0141] The path determination module 730 is configured to obtain the spatial coordinates of the smearing part of the smearing robot arm in the first spatial coordinate system, determine the current normal vector of the smearing part based on the spatial coordinates, adjust the smearing angle and smearing distance of the smearing part relative to the surface to be smeared according to the target spatial posture of the surface to be smeared and the current normal vector of the smearing part, and determine the masonry path of the outer contour of the traversing area of ​​the smearing part of the smearing robot arm; wherein the smearing angle and smearing distance are located in the expected threshold range.

[0142] In some embodiments, the machine vision-based masonry path planning system further includes a preprocessing module; the preprocessing module is specifically configured as follows:

[0143] By using a pass-through filter to constrain the initial point cloud data, point cloud data whose depth coordinates are outside the preset coordinate range are filtered out to obtain intermediate point cloud data.

[0144] The intermediate point cloud data is divided into a voxel grid by using a voxel filter, and the center point of each voxel grid is retained as the preprocessed point cloud data.

[0145] In some embodiments, the analysis and conversion module 720 is specifically configured as follows:

[0146] The random sampling consensus algorithm is used to detect the principal plane and obtain the principal plane point cloud corresponding to the initial point cloud data.

[0147] Perform coordinate transformation on the point cloud in the main plane, project it onto a two-dimensional plane, and determine the convex hull of the point cloud;

[0148] The minimum bounding box is obtained based on the rotating rectangle corresponding to the convex hull of the point cloud, and the outer contour of the area to be coated is determined based on the minimum bounding box.

[0149] In some embodiments, the analysis and conversion module 720 is specifically configured as follows:

[0150] The initial point cloud data is analyzed using the principal plane segmentation algorithm to obtain the initial outer contour of the area to be painted.

[0151] Feature extraction is performed on the outer contour of the initial region to obtain the set of corner coordinates of the outer contour of the initial region;

[0152] Based on spatial positional relationships, the corner points in the set of corner point coordinates are sorted clockwise or counterclockwise to generate the final outer contour of the area to be painted.

[0153] In some embodiments, the analysis and conversion module 720 is specifically configured as follows:

[0154] The camera is calibrated to obtain its intrinsic parameters and distortion coefficients.

[0155] The rigid transformation matrix between the camera coordinate system and the robot arm base coordinate system is determined by using the nine-point calibration method combined with intrinsic parameters and distortion coefficients.

[0156] The target spatial pose of the surface to be coated in the first spatial coordinate system is determined based on the point cloud data captured by the camera and the rigid transformation matrix.

[0157] In some embodiments, the path determination module 730 is specifically configured as follows:

[0158] Determine the coordinates of each corner point from the surface coordinates of the surface to be coated;

[0159] The masonry path is determined based on the convex hull boundary, smearing angle, and smearing distance corresponding to the corner point coordinates.

[0160] In some embodiments, the machine vision-based masonry path planning system further includes a dynamic adjustment module; the dynamic adjustment module is specifically configured as follows:

[0161] The joint speed of the robotic arm is dynamically adjusted based on the trajectory curvature of the masonry path; among which, the speed of the straight joints is greater than that of the corner joints.

[0162] It should be noted that the machine vision-based masonry path planning system and the machine vision-based masonry path planning method provided in this application are based on the same application concept. Therefore, the specific implementation of this embodiment can refer to the implementation of the aforementioned machine vision-based masonry path planning method, and the repeated parts will not be described again.

[0163] In some embodiments, please refer to Figure 8 , Figure 8 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. The electronic device 800 provided in this application includes a processor 810 and a memory 820; the memory 820 stores a computer program, wherein the computer program, when executed by the processor, implements the aforementioned machine vision-based masonry path planning method.

[0164] Specifically, processor 810 may include, for example, a general-purpose microprocessor, an instruction set processor and / or an associated chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. Processor 810 may also include onboard memory for caching purposes. Processor 810 may be a single processing unit or multiple processing units for performing different actions of the method flow according to embodiments of this application.

[0165] Memory 820 may be any medium capable of containing, storing, transmitting, propagating, or transmitting instructions. For example, memory 820 may include, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, instruments, or propagation media. Specific examples of memory 820 include: magnetic storage devices such as magnetic tape or hard disk drives (HDDs); optical storage devices such as optical discs (CD-ROMs); and may also be random access memory (RAM) or flash memory; and / or wired / wireless communication links.

[0166] This application also provides a computer-readable medium storing a computer program thereon, which, when executed by a processor, implements the machine vision-based masonry path planning method described above. This computer-readable medium may be included in the device / apparatus / system described in the above embodiments; or it may exist independently and not assembled into that device / apparatus / system. The aforementioned computer-readable medium carries one or more programs, which, when executed, implement the method according to the embodiments of this application.

[0167] According to embodiments of this application, a computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this application, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this application, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media can also be any computer-readable medium other than computer-readable storage media, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wireless, wired, optical fiber, radio frequency signals, etc., or any suitable combination thereof.

[0168] Those skilled in the art will understand that the features described in the various embodiments and / or claims of this application can be combined and / or combined in various ways, even if such combinations or combinations are not explicitly described in this application. In particular, the features described in the various embodiments and / or claims of this application can be combined and / or combined in various ways without departing from the spirit and teachings of this application. All such combinations and / or combinations fall within the scope of this application. Therefore, the scope of this application should not be limited to the above embodiments, but should be defined not only by the appended claims, but also by their equivalents. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this invention should be included within the protection scope of this invention.

Claims

1. A machine vision-based masonry path planning method, characterized in that, include: Obtain the initial point cloud data of the surface to be coated on the target object; The initial point cloud data is analyzed using a principal plane segmentation algorithm to determine the outer contour of the area to be coated, and the target spatial pose of the area to be coated in the first spatial coordinate system is determined through coordinate transformation. The target spatial attitude includes the surface coordinates and surface normal vector of the surface to be coated; The spatial coordinates of the smearing part of the smearing robot arm in the first spatial coordinate system are obtained, and the current normal vector of the smearing part is determined based on the spatial coordinates. The smearing angle and smearing distance of the smearing part relative to the surface to be smeared are adjusted according to the target spatial posture of the surface to be smeared and the current normal vector of the smearing part, and the masonry path of the smearing part of the smearing robot arm traversing the outer contour of the region is determined. The smearing angle and the smearing distance are located within the expected threshold range.

2. The machine vision-based masonry path planning method as described in claim 1, characterized in that, After acquiring the initial point cloud data of the surface to be coated, the method further includes: preprocessing the initial point cloud data; the preprocessing includes: The initial point cloud data is constrained by a pass-through filter, and point cloud data whose depth coordinates are outside the preset coordinate range are filtered out to obtain intermediate point cloud data. The intermediate point cloud data is divided into a voxel grid using a voxel filter, and the center point of each voxel grid is retained as the preprocessed point cloud data.

3. The machine vision-based masonry path planning method as described in claim 1, characterized in that, The step of analyzing the initial point cloud data using a principal plane segmentation algorithm to determine the outer contour of the area to be painted includes: The main plane is detected using a random sampling consensus algorithm to obtain the main plane point cloud corresponding to the initial point cloud data; The point cloud on the main plane is transformed by coordinates and projected onto a two-dimensional plane to determine the convex hull of the point cloud; The minimum bounding box is obtained based on the rotating rectangle corresponding to the point cloud convex hull, and the outer contour of the area to be coated is determined according to the minimum bounding box.

4. The machine vision-based masonry path planning method as described in claim 1, characterized in that, The step of analyzing the initial point cloud data using a principal plane segmentation algorithm to determine the outer contour of the area to be painted includes: The initial point cloud data is analyzed using a principal plane segmentation algorithm to obtain the initial outer contour of the area to be coated. Feature extraction is performed on the outer contour of the initial region to obtain the set of corner coordinates of the outer contour of the initial region; Based on spatial positional relationships, the corner points in the set of corner point coordinates are sorted clockwise or counterclockwise to generate the final outer contour of the area to be coated.

5. The machine vision-based masonry path planning method as described in claim 1, characterized in that, Determining the target spatial pose of the surface to be coated in the first spatial coordinate system through coordinate transformation includes: The camera is calibrated to obtain its intrinsic parameters and distortion coefficients. The rigid transformation matrix between the camera coordinate system and the robot arm base coordinate system is determined by using the nine-point calibration method in combination with the intrinsic parameters and the distortion coefficients. The target spatial pose of the surface to be coated in the first spatial coordinate system is determined based on the point cloud data captured by the camera and the rigid transformation matrix.

6. The machine vision-based masonry path planning method as described in claim 1, characterized in that, The step of adjusting the application angle and application distance of the application part relative to the surface to be applied based on the target spatial orientation of the surface to be applied and the current normal vector of the application part, and determining the masonry path of the application part of the application robot arm traversing the outer contour of the region, includes: The coordinates of each corner point are determined from the surface coordinates of the surface to be coated; The masonry path is determined based on the convex hull boundary corresponding to the corner point position coordinates, the smearing angle, and the smearing distance.

7. The machine vision-based masonry path planning method as described in claim 1, characterized in that, After determining the masonry path of the coating section of the coating robot arm traversing the outer contour of the area, the method further includes: The joint speed of the robotic arm is dynamically adjusted based on the trajectory curvature of the masonry path; wherein, the speed of the straight joint is greater than that of the corner joint.

8. A machine vision-based masonry path planning system, characterized in that, include: The module consists of a data acquisition module, an analysis and transformation module, and a path determination module; among which, The data acquisition module is configured to acquire the initial point cloud data of the surface to be coated of the target. The analysis and transformation module is configured to analyze the initial point cloud data using a principal plane segmentation algorithm, determine the outer contour of the area to be coated, and determine the target spatial pose of the area to be coated in a first spatial coordinate system through coordinate transformation; the target spatial pose includes the surface coordinates and surface normal vector of the area to be coated. The path determination module is configured to obtain the spatial coordinates of the smearing part of the smearing robot arm in the first spatial coordinate system, determine the current normal vector of the smearing part based on the spatial coordinates, adjust the smearing angle and smearing distance of the smearing part relative to the surface to be smeared according to the target spatial posture of the surface to be smeared and the current normal vector of the smearing part, and determine the masonry path of the smearing part of the smearing robot arm traversing the outer contour of the region; wherein the smearing angle and the smearing distance are within the expected threshold range.

9. An electronic device comprising a processor and a memory; said memory having a storage for a computer program, wherein, When the computer program is executed by the processor, it implements the machine vision-based masonry path planning method according to any one of claims 1 to 7.

10. A computer storage medium, characterized in that, It stores a computer program, wherein the computer program, when executed by a processor, implements the machine vision-based masonry path planning method as described in any one of claims 1 to 7.

Images