A wall-building robot grasping and recognition method and system integrating BIM and visual observation information

Abstract

The present invention proposes a masonry robot grasping and identification method and system that integrates BIM and visual observation information, which belongs to the field of robot grasping. This application realizes the intelligent grasping and identification of the masonry robot by combining BIM model information and real-time visual observation data, effectively avoiding the problem of masonry interference, and ensuring the rationality and safety of the grasping position by constructing an undirected graph of brick layout and collision testing, while significantly improving the autonomy, flexibility and work efficiency of the masonry robot.

Description

Technical Field

[0001] The present invention relates to the field of robot grasping technology, and in particular to a wall-building robot grasping and identifying method and system integrating BIM and visual observation information. Background Art

[0002] As a pillar industry of the national economy and a labor-intensive industry, the construction industry is facing severe challenges of shortage of skilled construction workers, especially in bricklaying. Bricklaying not only requires high physical fitness and skills of workers, but also has high safety risks. With the increasing prominence of industry pain points such as labor shortage and rising labor costs, it has become an urgent need to realize the robotization of bricklaying operations.

[0003] In masonry robot technology, autonomous grasping is one of the key technologies to realize intelligent masonry, and accurate grasping and recognition is a prerequisite for successful grasping. At present, machine vision technology is widely used in grasping and recognition, just like the human eye, it can be used to identify brick types and detect the grasping position of bricks to be laid. However, the existing grasping and recognition technology only relies on visual information for grasping and recognition, fails to make full use of masonry task planning information, lacks consideration of the overall masonry task, and leads to unreasonable grasping strategies in complex masonry scenes, and then masonry interference, affecting construction efficiency and quality.

[0004] Therefore, finding a method that can not only consider masonry task planning, accurately identify the grasping position, but also improve the grasping stability is a technical problem that needs to be solved urgently by technical personnel in this field. Summary of the invention

[0005] In view of this, the present invention proposes a masonry robot grasping and recognition method and system that integrates BIM and visual observation information, which can realize the intelligent grasping and recognition of the masonry robot by combining BIM model information and real-time visual observation data, and ensure the rationality and safety of the grasping position by constructing an undirected graph of brick layout and collision testing, while significantly improving the autonomy, flexibility and work efficiency of the masonry robot.

[0006] The technical solution of the present invention is achieved in this way:

[0007] In a first aspect, the present invention provides a wall-building robot grasping and identifying method integrating BIM and visual observation information, comprising the following steps:

[0008] S1. The wall-laying robot obtains the masonry instruction information, determines the brick type to be laid and the position of the brick type to be laid in the overall masonry task according to the masonry instruction information, obtains the BIM model of the brick type to be laid from the BIM model library, and extracts the geometric and semantic information of the brick type to be laid from the BIM model of the brick type to be laid;

[0009] S2. Construct an undirected graph of brick layout based on the BIM model of the overall masonry task, perform a collision test, and determine candidate grab positions that meet the masonry requirements based on the results of the collision test;

[0010] S3, the wall-building robot collects three-dimensional point cloud data of the target scene, and analyzes the three-dimensional point cloud data to determine the main axis direction and size characteristics of the target brick;

[0011] S4, performing information fusion analysis on the geometric and semantic information of the brick type to be laid and the main axis direction and size characteristics of the target brick to obtain a grabbing rectangular frame of the target brick;

[0012] S5. Projecting the grabbing rectangular frame onto a two-dimensional image screen, calculating the grabbing posture parameters of the wall-building robot, and the wall-building robot performs brick grabbing according to the grabbing posture parameters.

[0013] On the basis of the above technical solution, preferably, the grasping posture parameters include the grasping center point position, the gripping claw orientation, the opening distance, the gripping finger width and the rotation angle.

[0014] On the basis of the above technical solution, preferably, step S2 specifically includes:

[0015] S21, extracting the spatial position information and geometric dimensions of all bricks to be laid according to the BIM model of the overall masonry task;

[0016] S22, constructing a brick layout undirected graph based on the spatial position information and geometric dimensions of all bricks to be laid, wherein each node in the brick layout undirected graph represents a brick, and each edge represents the spatial relationship between bricks;

[0017] S23, generating an initial candidate grabbing rectangle for each brick to be laid based on the spatial position information and geometric dimensions of the brick to be laid, and performing a collision test on each candidate grabbing rectangle;

[0018] S24. Screen out collision-free candidate grasping positions according to the collision test results.

[0019] Based on the above technical solution, preferably, constructing the brick layout undirected graph specifically includes:

[0020] Step 1: Take each brick to be laid as a node to obtain a node set of the brick layout undirected graph;

[0021] Step 2: Select any node and traverse all nodes in the brick layout undirected graph node set to check whether the bounding rectangles of any two nodes after offset intersect:

[0022] If so, add an edge between the two nodes;

[0023] Step 3: Repeat step 2 until all nodes in the brick layout undirected graph node set are traversed to obtain the brick layout undirected graph.

[0024] On the basis of the above technical solution, preferably, the collision test specifically includes:

[0025] Based on the brick layout undirected graph, obtain the minimum enclosing rectangle information of adjacent laid bricks;

[0026] Check whether the grab rectangle along the main axis intersects with adjacent laid bricks:

[0027] If so, the grab rectangle collides with the adjacent laid bricks;

[0028] If not, proceed to the next step;

[0029] Check whether the grab rectangle perpendicular to the main axis intersects with adjacent laid bricks:

[0030] If so, the grab rectangle collides with the adjacent laid bricks;

[0031] If not, the grab rectangle is a collision-free grab rectangle.

[0032] More preferably, step S4 specifically includes:

[0033] S41, obtaining semantic information of a grabbing rectangular frame of the brick type to be laid from the BIM model of the brick type to be laid, and determining the type of semantic information;

[0034] S42, constructing an oriented bounding box of the three-dimensional point cloud using the principal component analysis method, extracting the length and width of the brick to be grasped from the oriented bounding box, and calculating the principal axis direction vector according to the length and width of the brick to be grasped;

[0035] S43, determining the direction of the grabbing rectangular frame according to the semantic information type, and calculating the width and length of the grabbing rectangular frame to generate a final grabbing rectangular frame.

[0036] On the basis of the above technical solution, preferably, the semantic information type includes message1 and message2, and the direction of the grabbing rectangular frame is:

[0037] When the semantic information type is message1, the direction of the grabbing rectangle is perpendicular to the main axis direction, and the width of the grabbing rectangle is w=l1+2d; where l1 represents the short side length of the oriented bounding box; d represents the gap parameter between the gripper and the edge of the brick to be grabbed;

[0038] When the semantic information type is message2, the direction of the grabbing rectangle is parallel to the main axis direction, and the width of the grabbing rectangle is w=l2+2d; where l2 represents the length of the long side of the oriented bounding box.

[0039] In a second aspect, the present invention provides a masonry robot grasping and identifying system integrating BIM and visual observation information, which adopts the masonry robot grasping and identifying method as described above, and comprises:

[0040] An instruction acquisition module is configured to enable the wall-laying robot to acquire masonry instruction information, determine the brick type to be laid and the position of the brick type to be laid in the overall masonry task according to the masonry instruction information, acquire the BIM model of the brick type to be laid from the BIM model library, and extract geometric and semantic information of the brick type to be laid from the BIM model of the brick type to be laid;

[0041] A collision test module is configured to construct an undirected graph of brick layout based on the BIM model of the overall masonry task, and to perform a collision test, and to determine candidate grab positions that meet the masonry requirements according to the results of the collision test;

[0042] A data analysis module, which is configured for the wall-building robot to collect three-dimensional point cloud data of the target scene, and analyze the three-dimensional point cloud data to determine the main axis direction and size characteristics of the target brick;

[0043] An information fusion module is configured to perform information fusion analysis on the geometric and semantic information of the brick type to be laid and the main axis direction and size characteristics of the target brick to obtain a grabbing rectangular frame of the target brick;

[0044] The grabbing module is configured to project the grabbing rectangular frame onto a two-dimensional image screen, calculate the grabbing posture parameters of the wall-building robot, and the wall-building robot performs brick grabbing according to the grabbing posture parameters.

[0045] In a third aspect, the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer instructions, and the computer instructions enable the computer to implement the wall-building robot grasping and identification method as described above.

[0046] In a fourth aspect, the present invention provides an electronic device, comprising at least one processor, a memory, a communication interface and a bus; wherein:

[0047] The processor, memory, and communication interface communicate with each other via the bus;

[0048] The memory stores program instructions that can be executed by the processor, and the processor calls the program instructions to implement the wall-building robot grasping and identifying method as described above.

[0049] The wall-building robot grasping and identifying method of the present invention has the following beneficial effects compared with the prior art:

[0050] (1) By combining BIM model information and real-time visual observation data, the intelligent grasping and recognition of the masonry robot is realized, effectively avoiding the problem of masonry interference. By constructing an undirected graph of brick layout and collision testing, the rationality and safety of the grasping position are ensured, while the autonomy, flexibility and work efficiency of the masonry robot are significantly improved.

[0051] (2) By using the spatial position information and geometric dimensions extracted from the BIM model to construct an undirected graph, the spatial relationship between bricks is effectively represented, and the initial candidate grabbing rectangle is generated and a collision test is performed, which improves the safety and efficiency of the masonry process. By screening collision-free candidate grabbing positions, the masonry quality and efficiency are improved.

[0052] (3) By obtaining the semantic information of the grabbing rectangle of bricks from the BIM model, a more intelligent and flexible grabbing method selection is achieved. According to different types of semantic information, the system can adaptively adjust the direction and size of the grabbing rectangle to better meet the grabbing requirements of different types of bricks, efficiently handle various masonry scenes and brick types, and at the same time ensure the stability and reliability of grabbing. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0054] Figure 1 A schematic diagram of a masonry task of a masonry robot grasping and identifying method of the present invention;

[0055] Figure 2 A model diagram of a masonry robot grasping system of the masonry robot grasping and identifying method of the present invention;

[0056] Figure 3 Schematic diagram of the gripping posture representation of the clamp of the wall-building robot grasping and identifying method of the present invention; wherein (a) is the robot's spatial grasping posture representation, and (b) is the two-dimensional plane grasping posture representation;

[0057] Figure 4 It is a flow chart of the grasping and identifying method of the wall-building robot of the present invention;

[0058] Figure 5 A schematic diagram describing part of the contents of an IFC file derived from a BIM model of the wall-building robot grasping and identifying method of the present invention;

[0059] Figure 6A schematic diagram of the analytical path for extracting geometric information of brick components in the grasping and identifying method of the wall-building robot of the present invention;

[0060] Figure 7 A schematic diagram of brick component position information extraction of the wall-building robot grasping and identifying method of the present invention;

[0061] Figure 8 A schematic diagram of the coordinate system relationship of the component stretching process for extracting the geometric shape information of a brick component in the grasping and identifying method of the wall-building robot of the present invention;

[0062] Fig. 9 A schematic diagram of a technical route for constructing a BIM undirected graph of a masonry structure for a grasping and identifying method of a masonry robot according to the present invention;

[0063] Fig.10 It is a schematic diagram of the generation of a grabbing rectangular frame after information fusion analysis of the grabbing and recognition method of the wall-building robot of the present invention; wherein (a) is a schematic diagram of the grabbing rectangular frame perpendicular to the main axis, and (b) is a schematic diagram of the grabbing rectangular frame parallel to the main axis;

[0064] Fig.11 This is a schematic diagram of an example of a grabbing and detecting frame of bricks to be laid using BIM model information in the grabbing and identifying method of a wall-building robot of the present invention.

[0065] Main markings:

[0066] 1. Gripper; 2. Grasping surface; 3. Plastering surface; 4. Masonry robot; 5. 3D camera; 7. Brick; 8. Masonry unit. DETAILED DESCRIPTION

[0067] The following will be combined with the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0068] As can be understood by those skilled in the art, Figure 1 As shown, the masonry robot 4 can grab bricks 7 from the brick-taking platform and then lay them at the designated position. Since the connections between the bricks 7 need to be smeared with mud to enhance the stability and sealing of the connections, the grabbing position of the bricks 7 should avoid interference with masonry.

[0069] like Figure 2As shown, the masonry robot 4 adopts a planar grasping method. When the bricks are placed horizontally, the robotic arm performs a grasping operation vertically downward. The clamp 6 at the end of the robotic arm has four degrees of freedom, namely translational movement along the X, Y, and Z axes and rotational movement around the Z axis. The 3D camera 5 is fixedly connected to the clamp.

[0070] like Figure 3 As shown in (a), the gripping posture of the robot gripper in space can be represented by five parameters, namely, the gripping center position P (X, Y, Z), the gripper orientation distance D, the gripper opening distance W, the gripper finger width H, and the gripper rotation angle around the Z axis. like Figure 3 As shown in (b), the gripping posture of the gripper mapped to the two-dimensional plane can be represented by a rotating rectangular frame. The center of the rectangular frame can determine the position of the gripping center of the gripper, the width w of the rectangular frame can determine the opening distance W of the gripper, the height h of the rectangular frame can determine the width H of the gripper, and the angle θ between the width direction of the gripper and the X direction can determine the gripping angle

[0071] like Figure 4 As shown, the present invention provides a wall-building robot grasping and identifying method integrating BIM and visual observation information, comprising the following steps:

[0072] S1. The wall-building robot obtains masonry instruction information, determines the brick type to be laid and the position of the brick type to be laid in the overall masonry task according to the masonry instruction information, obtains the BIM model of the brick type to be laid from the BIM model library, and extracts the geometric and semantic information of the brick type to be laid from the BIM model of the brick type to be laid.

[0073] It can be understood that the IFC data format file is exported from the BIM model library, and the IfcOpenShell toolkit provided by Python is used to parse the IFC file. Due to the wide variety of brick types and different shapes, the IFC physical file uses the IfcBuildingElementProxy entity to represent the brick components. By querying all IfcBuildingElementProxy instances through by_type, all brick component information can be obtained.

[0074] like Figure 5 As shown, Figure 5The following is a schematic diagram of the contents of a partial IFC file. The IFC (Industry Foundation Classes) file is a plain text file format. The file begins with the "ISO-10303-21" field and ends with the "END-ISO-10303-21" field. The main content includes two parts: the header segment and the data segment. The header segment starts with the "HEADER" field and ends with the "ENDSEC" field. It contains basic information about the IFC file, such as the file type and creation time. The data segment is an important part of the IFC file. It starts with the "DATA" field and ends with the "ENDSEC" field. It describes the BIM model by using a large number of entity instances. Each entity instance starts with "#", followed by the numeric ID of the entity. The "=" symbol is the IFC entity type. The attributes of the entity are in brackets. Different attributes are separated by "," and arranged in the order of attribute definition. When referencing other entities, simply indicate "#" + "number". "$" is often used to indicate a placeholder for an empty value or an unprovided value. The instance number is unique throughout the IFC file and instances can reference each other.

[0075] This application can obtain the IfcBuildingElementProxy entity instance corresponding to the brick to be laid according to the laying instruction information, and use the node name of the brick to be laid in the undirected graph to quickly query and obtain the node name of the adjacent laid brick to be laid, and use by_guide to obtain the IfcBuildingElementProxy instance of the adjacent laid brick, so as to obtain the placement position and geometric shape information of the adjacent laid brick.

[0076] In one embodiment of the present application, since the relationship between entities in the IFC file is complex and the logical levels are numerous, in order to improve the efficiency of IFC file parsing, it is necessary to filter a large amount of useless entity information and extract the information required for the component geometry analysis model, such as Figure 6 As shown, Figure 6 The analytical path for extracting geometric information of brick components is illustrated.

[0077] Specifically, Figure 7As shown in the figure, the ObjectPlacement attribute of the brick IfcBuildingElementProxy entity is associated with an IfcLocalPlacement entity, which is used to describe the position and direction of an object relative to its parent object. It contains two attributes: PlacementRelTo and RelativePlacement. The attribute PlacementRelTo will refer to another IfcLocalPlacement entity to determine the reference coordinate system that provides the relative position for it. In the IFC standard, building objects are described in a hierarchically organized coordinate system. The hierarchical structure of the IFC spatial structure elements generally followed is IfcSite (site), IfcBuilding (building), IfcBuildingS torey (building floor), therefore, the reference coordinate system of the brick component is IfcBuildingStorey, and the reference coordinate system of IfcBuildingStorey is further IfcBuilding. IfcSite defines the global coordinate system, which is also the world coordinate system. By iterating the IfcLocalPlacement entity until the end of the entity reference, the absolute coordinates of the object in the world coordinate system WCS can be obtained by extracting the relative placement defined by IfcAxis2Placement3D; each building entity has its own local coordinate system, and the RelativePlacement attribute describes the position and direction of the local coordinate system relative to the reference coordinate system by referencing an IfcAxis2Placement3D entity.

[0078] In one embodiment of the present application, the Representation attribute of the brick IfcBuildingElementProxy entity is associated with the IfcProductDefinitionShape entity, the Items attribute of the IfcProductDefinitionShape entity is associated with the IfcShapeRepresentation entity, and the Items attribute of the IfcShapeRepresentation entity is associated with the IfcMappedItem entity. The IfcMappedItem entity can map the geometric shape expression described with the target instance, so that one or more target instances have the same geometric expression. It mainly has two attributes, MappingSource and MappingTarget. The MappingSource attribute is associated with the IfcRepresentationMap entity, and the content is the geometric shape expression of the mapping target; MappingTarget is the operation to be performed on the mapping target, and its associated IfcCartesianTransformationOperator3D entity can implement a series of geometric transformation operations such as translation, rotation, mirroring, and scaling on the mapping target. The MappingOrigin attribute of the IfcRepresentationMap entity is associated with the IfcAxis2Placement3D entity, and is associated with the IfcShapeRepresentation entity through the MappedRepresentation attribute. The Items property of the IfcShapeRepresentation entity is associated with the IfcExtrudedAreaSolid entity, which can describe the geometric shape expression. The IfcExtrudedAreaSolid entity has four properties, among which the SweptArea property inherited from its parent class Sweep entity defines the cross-section type of the extruded body, the Position property defines the position of the extruded entity relative to the object coordinate system, and the other two properties ExtrudedDirection and Depth define the extrusion direction and length respectively. The SweptArea property is associated with the IfcProfileDef entity, which is a specific expression of the extruded cross-section. The IfcProfileDef entity is the super type of all cross-section definitions (IfcArbitraryClosedProfileDef, IfcParameterizedProfileDef) in IFC. Therefore, when parsing the SweptArea property of the IfcExtrudedAreaSolid entity, conditional judgment is required to determine which cross-section type it is.If it is an IfcArbitraryClosedProfileDef entity type, it is necessary to further determine what type of two-dimensional bounded curve the IfcBoundedCurve entity associated with its OuterCurve attribute is. If the curve is defined by an IfcPolyLine entity, the curve is composed of a series of Cartesian coordinate points connected in sequence. If the curve is defined by an IfcCompositeCurve entity, the curve is composed of multiple straight segments and arc segments connected end to end. The straight segment can be represented by an IfcPolyLine entity, and the arc segment is represented by an IfcTrimmedCurve entity. The basic curve of the arc segment can be represented by an IfcCircle entity. Since the stretched cross-section of a brick is usually not a rectangle, the IfcArbitraryClosedProfileDef entity is generally used to describe the stretched cross-section.

[0079] like Figure 8 As shown, Figure 8 The repositioned extruded solid geometry is shown. The 3D coordinate system represented by the black axis is the object coordinate system, which is the coordinate system defined for the component instance through the IfcLocalPlacement entity; the 2D coordinate system represented by the green axis is the profile position coordinate system, which is placed based on the XY plane of the object coordinate system; the 3D coordinate system represented by the blue axis is the extruded profile repositioning coordinate system, and the starting position of the extrusion is positioned based on the object coordinate system.

[0080] In one embodiment of the present application, the type name of the brick entity is obtained by querying the ObjectType attribute value of the IfcBuildingElementProxy instance, the placement position and direction of the brick entity in the global coordinate system is obtained by querying the ObjectPlacement attribute value, and the geometric shape information of the brick entity is obtained by querying the Representation attribute value.

[0081] By querying the IfcMappedItem instance to obtain the brick entity mapping item information, the IfcMappedItem instance can enable brick entities with the same geometric shape to share a geometric representation, thereby compressing the model and achieving data reuse.

[0082] By querying the IfcExtrudedAreaSolid instance, we can obtain the SweptArea, Position, ExtrudedDirection and Depth attribute values, and obtain the extruded section information according to the SweptArea attribute value. According to the Position attribute value, we can obtain the relative coordinate system where the extruded entity is located. This coordinate system is established relative to the local coordinate system of the brick entity. According to the ExtrudedDirection attribute value, we can obtain the extruded direction of the section. According to the Depth attribute value, we can obtain the length of the extruded entity. We can obtain the brick profile information by querying the OuterCurve of the IFCARBITRARYCLOSEDPROFILEDEF instance.

[0083] It can be understood that in the IFC standard, IfcAxis2Placement3D is a class that represents the position of an axis in three-dimensional space and is used to locate objects in three-dimensional space. IfcAxis2Placement3D has three main properties, namely Location, Axis and RefDirection, where Location is used to define the origin coordinates of the local coordinate system in the global coordinate system, Axis is used to define the direction vector of the Z axis of the local coordinate system, and RefDirection is used to define the reference direction vector of the X axis of the local coordinate system. When Axis and RefDirection are defined as empty, the direction vectors of the X, Y, and Z axes are all default values, and are [1, 0, 0], [0, 1, 0], [0, 0, 1], respectively. When Axis and RefDirection are not defined as empty, the direction vector of the Y axis can be obtained by the cross product y=zx of the z-axis and x-axis direction vectors.

[0084] When the coordinate value of Location is [o x ,o y ,o z ], the direction vector of the X axis is [a x ,b x ,c x ], the direction vector of the Y axis is [a y ,b y ,c y ], the direction vector of the Z axis is [a z ,b z ,c z ], then the transformation matrix from the local coordinate system to the global coordinate system is:

[0085]

[0086] Since the IfcAxis2Placement3D of the building element proxy defines its position and direction in the global coordinate system, and the IfcAxis2Placement3D of the mapped item extrusion defines its position and direction in the local coordinate system of the building element, in order to obtain the coordinate value (x, y, z) of a point (x′, y′, z′) on the cross-sectional profile of the mapped item extrusion in the global coordinate system of the building element proxy, the relationship can be created as follows:

[0087]

[0088] In the above formula, T1 can be obtained by using formula (1) according to the corresponding attribute value of IfcAxis2Placement3D entity associated with the RelativePlacement attribute of IfcLocalPlacement entity, and the corresponding attribute value of IfcAxis2Placement3D entity associated with the Pisition attribute of IfcExtrudedAreaSolid entity can be obtained by using formula (1). Formula (2) can be used to transform a series of points defined on a two-dimensional plane (Z=0) of the extruded body contour into the global coordinate system where the building entity is located. Connecting the obtained point coordinates in sequence can ensure that the mapped item extruded body geometry can be correctly mapped to the building element and placed according to the position and direction of the building element.

[0089] In one embodiment of the present application, the origin coordinates of the local coordinate system O′-X′Y′Z′ in the three-dimensional space relative to the global coordinate system O-XYZ are (a, b, c), and the direction vectors of the three axes X, Y, and Z of the local coordinate system are [u 11 ,u 12 ,u 13 ]、[u 21 ,u 22 ,u 23 ]、[u 31 ,u 32 ,u 33 ], given that the coordinates of point p in the local coordinate system are (x′, y′, z′), find its coordinate position (x, y, z) in the global coordinate system:

[0090] The coordinate transformation from the local coordinate system to the global coordinate system is derived as follows:

[0091] OP=x*i+y*j+z*k

[0092] OO′=a*i+b*j+c*k

[0093] Among them, i represents the unit vector of the X axis in the global coordinate system O-XYZ, j represents the unit vector of the Y axis in the global coordinate system O-XYZ, and k represents the unit vector of the Z axis in the global coordinate system O-XYZ;

[0094] Coordinate axis correspondence:

[0095]

[0096] Wherein, i′ represents the unit vector of the X′ axis in the local coordinate system O′-X′Y′Z′, j′ represents the unit vector of the Y′ axis in the local coordinate system O′-X′Y′Z′, and k′ represents the unit vector of the Z′ axis in the local coordinate system O′-X′Y′Z′;

[0097] Joint formula:

[0098] O′P=x′*i′+y′*j′+z′*k′

[0099] = x′*(u 11 *i+u 12 *j+u 13 *k)+y′*(u 21 *i+u 22 *j+u 23 *k)+z′*(u 31 *i+u 32 *j+u 33 *k)

[0100] =(u 11 *x′+u 21 *y′+u 31 *z′)*i+(u 12 *x′+u 22 *y′+u 32 *z′)*j+(u 13 *x′+u 23 *y′+u 33 *z′)*k

[0101] OP=OO′+O′P

[0102] Where O′P represents the vector from the origin of the local coordinate system to point P, OO′ represents the vector from the origin of the global coordinate system to the origin of the local coordinate system, and O′P represents the vector from the origin of the global coordinate system to point P;

[0103] but

[0104]

[0105] The conversion relationship is:

[0106]

[0107] S2. Construct an undirected graph of brick layout based on the BIM model of the overall masonry task, perform a collision test, and determine the candidate grabbing positions that meet the masonry requirements based on the results of the collision test.

[0108] Specifically, step S2 specifically includes:

[0109] S21, extracting the spatial position information and geometric dimensions of all bricks to be laid according to the BIM model of the overall masonry task;

[0110] S22, constructing a brick layout undirected graph based on the spatial position information and geometric dimensions of all bricks to be laid, wherein each node in the brick layout undirected graph represents a brick, and each edge represents the spatial relationship between bricks;

[0111] S23, generating an initial candidate grabbing rectangle for each brick to be laid based on the spatial position information and geometric dimensions of the brick to be laid, and performing a collision test on each candidate grabbing rectangle;

[0112] S24. Screen out collision-free candidate grasping positions according to the collision test results.

[0113] In the embodiment of the present application, the node name of the undirected graph is the IfcBuildingElementProx instance number, the node position represents the placement position of each brick, and the edge represents the connection relationship between adjacent bricks; the connection relationship between building blocks means that the bricks need to be coated with mud for bonding.

[0114] By constructing an undirected graph of brick layout and performing collision tests, global planning and optimization of the overall masonry task are achieved. The spatial position information and geometric dimensions extracted from the BIM model are used to construct an undirected graph, which effectively represents the spatial relationship between bricks, generates an initial candidate grab rectangle and performs a collision test, ensuring that the selected grab position will not conflict with other bricks or structures, thereby improving the safety and efficiency of the masonry process. By screening non-collision candidate grab rectangle positions, possible interference and errors in the masonry process are reduced, and the quality and efficiency of masonry are improved. At the same time, it can be applied to different types of bricks and complex masonry scenes.

[0115] like Fig. 9 As shown, further, constructing the brick layout undirected graph specifically includes:

[0116] Step 1: Take each brick to be laid as a node to obtain a node set of the brick layout undirected graph;

[0117] Step 2: Select any node and traverse all nodes in the brick layout undirected graph node set to check whether the bounding rectangles of any two nodes after offset intersect:

[0118] If so, add an edge between the two nodes;

[0119] Step 3: Repeat step 2 until all nodes in the brick layout undirected graph node set are traversed to obtain the brick layout undirected graph.

[0120] In an embodiment of the present application, the minimum circumscribed rectangle is obtained using the vertex coordinates of the cross-section contour of the brick entity, each brick entity is represented by an instance number, and the boundary points of the convex hull are used to approximate the minimum circumscribed rectangle.

[0121] Specifically, the convex hull-based method is used to calculate the minimum circumscribed rectangle SmallRect1 of the polygonal cross section of the brick component, and the left small corner coordinate (x min ,y min ) and the upper right corner coordinate (x max ,y max );

[0122] Give the generated minimum bounding rectangle an offset d of the thickness of the brick joint to regenerate the minimum bounding rectangle SmallRect2, and get the left small corner coordinate (x′ min , y′ min ) and the upper right corner coordinate (x′ max , y′ max ), where x′ min =x min -d, y′ min =y min -d, x′ max =x max +d,y′ max =y max +d;

[0123] By creating a Rectangle class and an empty list Rectangles to store Rectangle class instances, the instance name id of the brick component and the horizontal coordinate c of the brick component placement position are set. x , the horizontal coordinate c of the brick component placement position y , x′ min , y′ min , x′ max , y′ max Assign values ​​to the 7 attribute values ​​id, c x 、c y 、x min ,y min 、x max ,y max Then get a Rectangle instance and add it to the empty list Rectangles to generate an undirected graph.

[0124] As you can understand, in Python, you can create a Rectangle class and define its seven instance attributes, including id, c x 、c y 、x min ,y min 、x max ,y max These attributes are initialized by the __init__ constructor. The __init__ method is a special method that Python automatically calls when a new instance of a class is created. When you need to find the location of a rectangle, you can quickly access it through its id.

[0125] Specifically, initialize the brick component connection graph G and dictionary Pos;

[0126] Traverse the Rectangles list and store the center point coordinates in the Pos dictionary for each rectangle Rect id;

[0127] Use the key of the Pos dictionary (i.e. the id of the rectangle) as a node and add it to the graph;

[0128] Use two nested loops to traverse each pair of rectangles in the Rectangles list (excluding comparisons with themselves); if two rectangles intersect, add an edge connecting the nodes in graph G.

[0129] In one embodiment of the present application, when the minimum and maximum x and y coordinates of the convex hull boundary points are respectively min x 、min y 、max x 、max y ;

[0130] The coordinates of the lower left corner are (min x , min y ), the coordinates of the upper right corner are (max x , max y );

[0131] When there is mud bonding between bricks, the minimum bounding box matrix is ​​offset. The coordinates of the lower left corner of the minimum bounding box after the offset are (min x -d,min y -d), the upper right corner coordinate is (max x +d,max y +d), d represents the thickness of the mortar joints between bricks;

[0132] Check if a rectangle is to the left, right, above, or below another rectangle:

[0133] If any of the conditions is true, the two rectangles do not intersect;

[0134] If all conditions are not true, the two rectangles intersect, and each of the extracted block solid section profile vertices is added to the graph as a node, and the relationship between them is added to the graph as an edge.

[0135] Furthermore, the collision test specifically includes:

[0136] Based on the brick layout undirected graph, obtain the minimum enclosing rectangle information of adjacent laid bricks;

[0137] Check whether the grab rectangle along the main axis intersects with adjacent laid bricks:

[0138] If so, the grab rectangle collides with the adjacent laid bricks;

[0139] If not, proceed to the next step;

[0140] Check whether the grab rectangle perpendicular to the main axis intersects with adjacent laid bricks:

[0141] If so, the grab rectangle collides with the adjacent laid bricks;

[0142] If not, the grab rectangle is a collision-free grab rectangle.

[0143] The collision test is described with a specific embodiment:

[0144] The coordinates of the lower left corner and upper right corner of rectangle Rect1 are

[0145] The coordinates of the lower left corner and upper right corner of rectangle Rect2 are

[0146] like or or or If any of the conditions is true, rectangles Rect1 and Rect2 do not intersect, otherwise they intersect.

[0147] S3, the wall-building robot collects three-dimensional point cloud data of the target scene, and analyzes the three-dimensional point cloud data to determine the main axis direction and size characteristics of the target brick;

[0148] S4. The geometric and semantic information of the brick type to be laid is fused and analyzed with the main axis direction and size characteristics of the target brick to obtain a grabbing rectangle of the target brick.

[0149] Specifically, step S4 specifically includes:

[0150] S41, obtaining semantic information of a grabbing rectangular frame of the brick type to be laid from the BIM model of the brick type to be laid, and determining the type of semantic information;

[0151] S42, constructing an oriented bounding box of the three-dimensional point cloud using the principal component analysis method, extracting the length and width of the brick to be grasped from the oriented bounding box, and calculating the principal axis direction vector according to the length and width of the brick to be grasped;

[0152] S43, determining the direction of the grabbing rectangular frame according to the semantic information type, and calculating the width and length of the grabbing rectangular frame to generate a final grabbing rectangular frame.

[0153] like Fig.10 As shown, the dotted line represents a grabbing rectangle, which represents a directional packaging box; further, the semantic information type includes message1 and message2, and the direction of the grabbing rectangle is:

[0154] When the semantic information type is message1, the direction of the grabbing rectangle is perpendicular to the main axis direction, and the width of the grabbing rectangle is w=l1+2d; where l1 represents the short side length of the oriented bounding box; d represents the gap parameter between the gripper and the edge of the brick to be grabbed;

[0155] When the semantic information type is message2, the direction of the grabbing rectangle is parallel to the main axis direction, and the width of the grabbing rectangle is w=l2+2d; where l2 represents the length of the long side of the oriented bounding box.

[0156] By obtaining the semantic information of the grabbing rectangle of bricks from the BIM model, a more intelligent and flexible grabbing method selection is achieved. According to different types of semantic information, the system can adaptively adjust the direction and size of the grabbing rectangle to better meet the grabbing needs of different types of bricks, efficiently handle various masonry scenes and brick types, and at the same time ensure the stability and reliability of grabbing.

[0157] S5. Project the grasping rectangular frame onto a two-dimensional image screen, calculate the grasping posture parameters of the wall-building robot, and the wall-building robot performs brick grasping according to the grasping posture parameters; wherein the grasping posture parameters include the grasping center point position, gripper orientation, opening distance, gripper finger width and rotation angle.

[0158] Specifically, a specific embodiment is used to illustrate:

[0159] Obtain the coordinates of the lower left corner (x1, y1) and the upper right corner (x2, y2) of the minimum circumscribed rectangle of the surface contour of the BIM model of the brick to be laid;

[0160] The width of the minimum enclosing rectangle is width=x2-x1, the height is height=y2-y1, and the long side direction of the minimum enclosing rectangle is the main axis direction;

[0161] If width>height, the main axis direction is parallel to the X axis;

[0162] If width≤height, the main axis direction is perpendicular to the X axis;

[0163] If the coordinates of the lower left corner and upper right corner of the grab rectangle perpendicular to the main axis are (x′1, y′1) and (x′2, y′2) respectively, the coordinates of the lower left corner and upper right corner of the grab rectangle along the main axis are (x″1, y″1) and (x″2, y″2) respectively;

[0164] If the main axis direction is parallel to the X-axis, then x′1=x1+width / 2-h / 2, y′1=y1+height / 2-w / 2, x′2=x′1+h, y′2=y′1+w, x″1=x1+width / 2-w / 2, y″1=y1+height / 2-h / 2, x″2=x″1+w, y″2=y″1+h;

[0165] If the main axis is perpendicular to the X axis, then x′1=x1+width / 2-w / 2, y′1=y1+height / 2-h / 2, x′2=x′1+w, y′2=y′1+h, x1 ″ =x1+width / 2-h / 2,y1 ″ =y1+height / 2-w / 2, x″2=x″1+h, y″2=y″1+w.

[0166] Among them, w is the opening distance of the clamping jaws, h is the width of the clamping fingers, and w can be determined according to the clamping width of the brick.

[0167] The semantic information corresponding to the grasping frames generated in the above four situations can be summarized into two items, namely: message1 = {"The opening direction of the jaws is perpendicular to the main axis direction of the brick to be laid, and the opening distance of the jaws is the length of the short side of the brick to be laid l+2d", message2 = {"The opening direction of the jaws is parallel to the main axis direction of the brick to be laid, and the opening distance of the jaws is the length of the long side of the brick to be laid l+2d"}, d is the clamping width l, which is the distance symmetrically extended to both sides along the opening direction of the jaws from the grasping center.

[0168] like Fig.11As shown, the present application first performs BIM modeling on the overall masonry task and obtains the BIM model of the brick type to be laid, constructs a brick layout design drawing according to the BIM model of the overall masonry task, and constructs a brick layout undirected graph according to the brick design drawing, and uses the brick layout undirected graph to obtain the adjacent brick information of the bricks to be laid, and obtains the BIM model grabbing rectangular frame of the brick type to be laid. By combining the BIM model information and real-time visual observation data, the intelligent grabbing and recognition of the masonry robot is realized, and the masonry interference problem is effectively avoided. By constructing the brick layout undirected graph and collision test, the rationality and safety of the grabbing position are ensured, while the autonomy, flexibility and work efficiency of the masonry robot are significantly improved.

[0169] The present invention provides a masonry robot grasping and identifying system integrating BIM and visual observation information, which adopts the masonry robot grasping and identifying method as described above, and comprises:

[0170] An instruction acquisition module is configured to enable the wall-laying robot to acquire masonry instruction information, determine the brick type to be laid and the position of the brick type to be laid in the overall masonry task according to the masonry instruction information, acquire the BIM model of the brick type to be laid from the BIM model library, and extract geometric and semantic information of the brick type to be laid from the BIM model of the brick type to be laid;

[0171] A collision test module is configured to construct an undirected graph of brick layout based on the BIM model of the overall masonry task, and to perform a collision test, and to determine candidate grab positions that meet the masonry requirements according to the results of the collision test;

[0172] A data analysis module, which is configured for the wall-building robot to collect three-dimensional point cloud data of the target scene, and analyze the three-dimensional point cloud data to determine the main axis direction and size characteristics of the target brick;

[0173] An information fusion module is configured to perform information fusion analysis on the geometric and semantic information of the brick type to be laid and the main axis direction and size characteristics of the target brick to obtain a grabbing rectangular frame of the target brick;

[0174] The grasping module is configured to project the grasping rectangular frame onto a two-dimensional image screen, calculate the grasping posture parameters of the wall-building robot, and the wall-building robot performs brick grasping according to the grasping posture parameters; wherein the grasping posture parameters include the grasping center point position, the gripper orientation, the opening distance, the gripper finger width and the rotation angle.

[0175] The present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer instructions, and the computer instructions enable the computer to implement the wall-building robot grasping and identifying method as described above.

[0176] The present invention provides an electronic device, comprising: at least one processor, a memory, a communication interface and a bus; wherein the processor, the memory and the communication interface communicate with each other through the bus; the memory stores program instructions that can be executed by the processor, and the processor calls the program instructions to implement the wall-building robot grasping and identification method as described above.

Claims

1. A wall-building robot grasping and recognition method integrating BIM and visual observation information, characterized in that: The following steps are involved: S1. The wall-laying robot obtains the masonry instruction information, determines the brick type to be laid and the position of the brick type to be laid in the overall masonry task according to the masonry instruction information, obtains the BIM model of the brick type to be laid from the BIM model library, and extracts the geometric and semantic information of the brick type to be laid from the BIM model of the brick type to be laid; S2. Construct an undirected graph of brick layout based on the BIM model of the overall masonry task, perform a collision test, and determine candidate grab positions that meet the masonry requirements based on the results of the collision test; Step S2 specifically includes: S21, extracting the spatial position information and geometric dimensions of all bricks to be laid according to the BIM model of the overall masonry task; S22, constructing a brick layout undirected graph based on the spatial position information and geometric dimensions of all bricks to be laid, wherein each node in the brick layout undirected graph represents a brick, and each edge represents the spatial relationship between bricks; S23, generating an initial candidate grabbing rectangle for each brick to be laid based on the spatial position information and geometric dimensions of the brick to be laid, and performing a collision test on each candidate grabbing rectangle; S24, screening out collision-free candidate grasping positions according to the collision test results; S3, the wall-building robot collects three-dimensional point cloud data of the target scene, and analyzes the three-dimensional point cloud data to determine the main axis direction and size characteristics of the target brick; S4, performing information fusion analysis on the geometric and semantic information of the brick type to be laid and the main axis direction and size characteristics of the target brick to obtain a grabbing rectangular frame of the target brick; S5. Projecting the grabbing rectangular frame onto a two-dimensional image screen, calculating the grabbing posture parameters of the wall-building robot, and the wall-building robot performs brick grabbing according to the grabbing posture parameters.

2. A wall-building robot grasping and identifying method integrating BIM and visual observation information as claimed in claim 1, characterized in that: The grasping posture parameters include the grasping center point position, gripper orientation, opening distance, gripper finger width and rotation angle.

3. The wall-building robot grasping and identification method integrating BIM and visual observation information as claimed in claim 1, characterized in that: Constructing a brick layout undirected graph specifically includes: Step 1: Take each brick to be laid as a node to obtain a node set of the brick layout undirected graph; Step 2: Select any node and traverse all nodes in the brick layout undirected graph node set to check whether the bounding rectangles of any two nodes after offset intersect: If so, add an edge between the two nodes; Step 3: Repeat step 2 until all nodes in the brick layout undirected graph node set are traversed to obtain the brick layout undirected graph.

4. A wall-building robot grasping and identifying method integrating BIM and visual observation information as claimed in claim 3, characterized in that: The collision test specifically includes: Based on the brick layout undirected graph, obtain the minimum enclosing rectangle information of adjacent laid bricks; Check whether the grab rectangle along the main axis intersects with adjacent laid bricks: If so, the grab rectangle collides with the adjacent laid bricks; If not, proceed to the next step; Check whether the grab rectangle perpendicular to the main axis intersects with adjacent laid bricks: If so, the grab rectangle collides with the adjacent laid bricks; If not, the grab rectangle is a collision-free grab rectangle.

5. The wall-building robot grasping and identification method integrating BIM and visual observation information as claimed in claim 1, characterized in that: Step S4 specifically includes: S41, obtaining semantic information of a grabbing rectangular frame of the brick type to be laid from the BIM model of the brick type to be laid, and determining the type of semantic information; S42, constructing an oriented bounding box of the three-dimensional point cloud using the principal component analysis method, extracting the length and width of the brick to be grasped from the oriented bounding box, and calculating the principal axis direction vector according to the length and width of the brick to be grasped; S43, determining the direction of the grabbing rectangular frame according to the semantic information type, and calculating the width and length of the grabbing rectangular frame to generate a final grabbing rectangular frame.

6. The wall-building robot grasping and identification method integrating BIM and visual observation information as claimed in claim 5, characterized in that: The semantic information types include and , the grabbing rectangular frame direction is: When the semantic information type is When the grabbing rectangle is perpendicular to the main axis, the width of the grabbing rectangle is ;in Indicates the length of the short side of the oriented bounding box; Indicates the clearance parameter between the gripper and the edge of the brick to be grasped; When the semantic information type is When the grabbing rectangle is parallel to the main axis, the width of the grabbing rectangle is ;in Indicates the length of the long side of the oriented bounding box.

7. A wall-building robot grasping and recognition system integrating BIM and visual observation information, characterized in that: The wall-building robot grasping and identifying method according to any one of claims 1 to 6 comprises: An instruction acquisition module is configured to enable the wall-laying robot to acquire masonry instruction information, determine the brick type to be laid and the position of the brick type to be laid in the overall masonry task according to the masonry instruction information, acquire the BIM model of the brick type to be laid from the BIM model library, and extract geometric and semantic information of the brick type to be laid from the BIM model of the brick type to be laid; A collision test module is configured to construct an undirected graph of brick layout based on the BIM model of the overall masonry task, and to perform a collision test, and to determine candidate grab positions that meet the masonry requirements according to the results of the collision test; A data analysis module, which is configured for the wall-building robot to collect three-dimensional point cloud data of the target scene, and analyze the three-dimensional point cloud data to determine the main axis direction and size characteristics of the target brick; An information fusion module is configured to perform information fusion analysis on the geometric and semantic information of the brick type to be laid and the main axis direction and size characteristics of the target brick to obtain a grabbing rectangular frame of the target brick; The grabbing module is configured to project the grabbing rectangular frame onto a two-dimensional image screen, calculate the grabbing posture parameters of the wall-building robot, and the wall-building robot performs brick grabbing according to the grabbing posture parameters.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions enable the computer to implement the wall-building robot grasping and identification method according to any one of claims 1 to 6.

9. An electronic device, characterized in that: At least one processor, a memory, a communication interface and a bus; wherein, The processor, memory, and communication interface communicate with each other via the bus; The memory stores program instructions that can be executed by the processor, and the processor calls the program instructions to implement the wall-building robot grasping and identification method according to any one of claims 1-6.

Images